Notes on Taiga API

The first step is to generate a AUTH Token for API Access, you can generate one for your taiga account using below script:
We would also need to know our openswitch project id to use in our rest calls. We can know the project id with this rest call:
https://api.taiga.io/api/v1/projects/by_slug?slug=<project name, for example, openswitch>
The Taiga ID we have are actually called ref(s) and they can be linked to multiple things like projects, issues etc. for example, if we need to access too fields of this issue(https://api.taiga.io/api/v1/issues/914) , these are the api calls to make:
Grab issue id from
Use the issue id to make another call
The above call will give return us regular fields, but, we also have, custom fields inside each issue like Feature, Target Release etc which can be accessed with this call:
There are two endpoints to search issues:

https://api.taiga.io/api/v1/search?get_all=true&project=1&text=texttosearch -> To search text in issues subject + ref + description + tags.

https://api.taiga.io/api/v1/issues?project=1&q=text -> To search text in issues subject + ref
Interesting, there is also a Taiga Ansible Module:

Basic DNS Concepts in Linux

Tags

, ,

DNS

This post is just a collection of information around understanding the basics of DNS Concepts in Linux in a fashion that I can understand. I give all the credit to original articles that I got inspiration and knowledge from.

Basic Terminology

Domain Name System – The domain name system, more commonly known as “DNS” is the networking system in place that allows us to resolve human-friendly names to unique addresses.

Domain Name – A domain name is the human-friendly name that we are used to associating with an internet resource.

For instance, “google.com” is a domain name. The URL “google.com” is associated with the servers owned by Google Inc. The domain name system allows us to reach the Google servers when we type “google.com” into our browsers.

Top-Level Domain – A top-level domain, or TLD, is the most general part of the domain. The top-level domain is the furthest portion to the right (as separated by a dot). Common top-level domains are “com”, “net”, “org”, “gov”, “edu”, and “io”.

Top-level domains are at the top of the hierarchy in terms of domain names.

Hosts – Within a domain, the domain owner can define individual hosts, which refer to separate computers or services accessible through a domain.

For instance, most domain owners make their web servers accessible through the bare domain (example.com) and also through the “host” definition “www” (www.example.com).

You can have other host definitions under the general domain. You could have API access through an “api” host (api.example.com) or you could have ftp access by defining a host called “ftp” or “files” (ftp.example.com or files.example.com). The host names can be arbitrary as long as they are unique for the domain.

SubDomain – DNS works in a hierarchy. TLDs can have many domains under them. For instance, the “com” TLD has both “google.com” and “ubuntu.com” underneath it. A “subdomain” refers to any domain that is part of a larger domain. In this case, “ubuntu.com” can be said to be a subdomain of “com”. This is typically just called the domain or the “ubuntu” portion is called a SLD, which means second level domain.

Likewise, each domain can control “subdomains” that are located under it. This is usually what we mean by subdomains. For instance you could have a subdomain for the history department of your school at “www.history.school.edu”. The “history” portion is a subdomain.

The difference between a host name and a subdomain is that a host defines a computer or resource, while a subdomain extends the parent domain. It is a method of subdividing the domain itself.

Whether talking about subdomains or hosts, you can begin to see that the left-most portions of a domain are the most specific. This is how DNS works: from most to least specific as you read from left-to-right.

Fully Qualified Domain Name – A fully qualified domain name, often called FQDN, is what we call an absolute domain name. Domains in the DNS system can be given relative to one another, and as such, can be somewhat ambiguous. A FQDN is an absolute name that specifies its location in relation to the absolute root of the domain name system.

This means that it specifies each parent domain including the TLD. A proper FQDN ends with a dot, indicating the root of the DNS hierarchy. An example of a FQDN is “mail.google.com.”.

Name Server – A name server is a computer designated to translate domain names into IP addresses. These servers do most of the work in the DNS system. Since the total number of domain translations is too much for any one server, each server may redirect request to other name servers or delegate responsibility for a subset of subdomains they are responsible for.

Name servers can be “authoritative”, meaning that they give answers to queries about domains under their control. Otherwise, they may point to other servers, or serve cached copies of other name servers’ data.

Zone File – A zone file is a simple text file that contains the mappings between domain names and IP addresses. This is how the DNS system finally finds out which IP address should be contacted when a user requests a certain domain name.

Zone files reside in name servers and generally define the resources available under a specific domain, or the place that one can go to get that information.

Records – Within a zone file, records are kept. In its simplest form, a record is basically a single mapping between a resource and a name. These can map a domain name to an IP address, define the name servers for the domain, define the mail servers for the domain, etc.

How DNS Works

Root Servers

As we said above, DNS is, at its core, a hierarchical system. At the top of this system is what are known as “root servers”. These servers are controlled by various organizations and are delegated authority by ICANN (Internet Corporation for Assigned Names and Numbers).

There are currently 13 root servers in operation. However, as there are an incredible number of names to resolve every minute, each of these servers is actually mirrored. The interesting thing about this set up is that each of the mirrors for a single root server share the same IP address. When requests are made for a certain root server, the request will be routed to the nearest mirror of that root server.

What do these root servers do? Root servers handle requests for information about Top-level domains. So if a request comes in for something a lower-level name server cannot resolve, a query is made to the root server for the domain.

The root servers won’t actually know where the domain is hosted. They will, however, be able to direct the requester to the name servers that handle the specifically requested top-level domain.

So if a request for “www.wikipedia.org” is made to the root server, the root server will tell not find the result in its records. It will check its zone files for a listing that matches “www.wikipedia.org”. It will not find one.

It will instead find a record for the “org” TLD and give the requesting entity the address of the name server responsible for “org” addresses.

TLD Servers – The requester then sends a new request to the IP address (given to it by the root server) that is responsible for the top-level domain of the request.

So, to continue our example, it would send a request to the name server responsible for knowing about “org” domains to see if it knows where “www.wikipedia.org” is located.

Once again, the requester will look for “www.wikipdia.org” in its zone files. It will not find this record in its files.

However, it will find a record listing the IP address of the name server responsible for “wikipedia.org”. This is getting much closer to the answer we want.

Domain-Level Name Servers – At this point, the requester has the IP address of the name server that is responsible for knowing the actual IP address of the resource. It sends a new request to the name server asking, once again, if it can resolve “www.wikipedia.org”.

The name server checks its zone files and it finds that it has a zone file associated with “wikipedia.org”. Inside of this file, there is a record for the “www” host. This record tells the IP address where this host is located. The name server returns the final answer to the requester.

What is a Resolving Name Server?

In the above scenario, we referred to a “requester”. What is the requester in this situation?

In almost all cases, the requester will be what we call a “resolving name server” A resolving name server is one configured to ask other servers questions. It is basically an intermediary for a user which caches previous query results to improve speed and knows the addresses of the root servers to be able to “resolve” requests made for things it doesn’t already know about.

Basically, a user will usually have a few resolving name servers configured on their computer system. The resolving name servers are usually provided by an ISP or other organizations. For instance Google provides resolving DNS servers that you can query. These can be either configured in your computer automatically or manually.

When you type a URL in the address bar of your browser, your computer first looks to see if it can find out locally where the resource is located. It checks the “hosts” file on the computer and a few other locations. It then sends the request to the resolving name server and waits back to receive the IP address of the resource.

The resolving name server then checks its cache for the answer. If it doesn’t find it, it goes through the steps outlined above.

Resolving name servers basically compress the requesting process for the end user. The clients simply have to know to ask the resolving name servers where a resource is located and be confident that they will investigate and return the final answer.

Zone Files

Zone files are the way that name servers store information about the domains they know about. Every domain that a name server knows about is stored in a zone file. Most requests coming to the average name server are not something that the server will have zone files for.

If it is configured to handle recursive queries, like a resolving name server, it will find out the answer and return it. Otherwise, it will tell the requesting party where to look next.

The more zone files that a name server has, the more requests it will be able to answer authoritatively.

A zone file describes a DNS “zone”, which is basically a subset of the entire DNS naming system. It generally is used to configure just a single domain. It can contain a number of records which define where resources are for the domain in question.

The zone’s $ORIGIN is a parameter equal to the zone’s highest level of authority by default.

So if a zone file is used to configure the “example.com.” domain, the $ORIGIN would be set to example.com..

This is either configured at the top of the zone file or it can be defined in the DNS server’s configuration file that references the zone file. Either way, this parameter describes what the zone is going to be authoritative for.

Similarly, the $TTL configures the “time to live” of the information it provides. It is basically a timer. A caching name server can use previously queried results to answer questions until the TTL value runs out.

Record Types

Within the zone file, we can have many different record types. We will go over some of the more common (or mandatory types) here

SOA Records – The Start of Authority, or SOA, record is a mandatory record in all zone files. It must be the first real record in a file (although $ORIGIN or $TTL specifications may appear above). It is also one of the most complex to understand.

The start of authority record looks something like this:

domain.com. IN SOA ns1.domain.com. admin.domain.com. (

12083   ; serial number

3h     ; refresh interval

30m     ; retry interval

3w     ; exiry period

1h     ; negative TTL

)

Let’s explain what each part is for:

  • com.: This is the root of the zone. This specifies that the zone file is for the domain.com. domain. Often, you’ll see this replaced with @, which is just a placeholder that substitutes the contents of the $ORIGIN variable we learned about above.
  • IN SOA: The “IN” portion means internet (and will be present in many records). The SOA is the indicator that this is a Start of Authority record.
  • domain.com.: This defines the primary master name server for this domain. Name servers can either be master or slaves, and if dynamic DNS is configured one server needs to be a “primary master”, which goes here. If you haven’t configured dynamic DNS, then this is just one of your master name servers.
  • domain.com.: This is the email address of the administrator for this zone. The “@” is replaced with a dot in the email address. If the name portion of the email address normally has a dot in it, this is replace with a “\” in this part (your.name@domain.com becomes your\name.domain.com).
  • 12083: This is the serial number for the zone file. Every time you edit a zone file, you must increment this number for the zone file to propagate correctly. Slave servers will check if the master server’s serial number for a zone is larger than the one they have on their system. If it is, it requests the new zone file, if not, it continues serving the original file.
  • 3h: This is the refresh interval for the zone. This is the amount of time that the slave will wait before polling the master for zone file changes.
  • 30m: This is the retry interval for this zone. If the slave cannot connect to the master when the refresh period is up, it will wait this amount of time and retry to poll the master.
  • 3w: This is the expiry period. If a slave name server has not been able to contact the master for this amount of time, it no longer returns responses as an authoritative source for this zone.
  • 1h: This is the amount of time that the name server will cache a name error if it cannot find the requested name in this file.

A and AAAA Records

Both of these records map a host to an IP address. The “A” record is used to map a host to an IPv4 IP address, while “AAAA” records are used to map a host to an IPv6 address.

The general format of these records is this:

host     IN     A       IPv4_address

host     IN     AAAA   IPv6_address

So since our SOA record called out a primary master server at “ns1.domain.com”, we would have to map this to an address to an IP address since “ns1.domain.com” is within the “domain.com” zone that this file is defining.

The record could look something like this:

ns1     IN A       111.222.111.222

Notice that we don’t have to give the full name. We can just give the host, without the FQDN and the DNS server will fill in the rest with the $ORIGIN value. However, we could just as easily use the entire FQDN if we feel like being semantic:

ns1.domain.com.   IN A       111.222.111.222

In most cases, this is where you’ll define your web server as “www”:

www     IN A       222.222.222.222

We should also tell where the base domain resolves to. We can do this like this:

domain.com.     IN A       222.222.222.222

We could have used the “@” to refer to the base domain instead:

@       IN A       222.222.222.222

We also have the option of resolving anything that under this domain that is not defined explicitly to this server too. We can do this with the “*” wild card:

*       IN A       222.222.222.222

All of these work just as well with AAAA records for IPv6 addresses.

CNAME Records

CNAME records define an alias for canonical name for your server (one defined by an A or AAAA record).

For instance, we could have an A name record defining the “server1” host and then use the “www” as an alias for this host:

server1     IN A       111.111.111.111

www         IN CNAME   server1

Be aware that these aliases come with some performance losses because they require an additional query to the server. Most of the time, the same result could be achieved by using additional A or AAAA records.

One case when a CNAME is recommended is to provide an alias for a resource outside of the current zone.

MX Records

MX records are used to define the mail exchanges that are used for the domain. This helps email messages arrive at your mail server correctly.

Unlike many other record types, mail records generally don’t map a host to something, because they apply to the entire zone. As such, they usually look like this:

IN MX 10   mail.domain.com.

Note that there is no host name at the beginning.

Also note that there is an extra number in there. This is the preference number that helps computers decide which server to send mail to if there are multiple mail servers defined. Lower numbers have a higher priority.

The MX record should generally point to a host defined by an A or AAAA record, and not one defined by a CNAME.

So, let’s say that we have two mail servers. There would have to be records that look something like this:

IN MX 10 mail1.domain.com.

IN MX 50 mail2.domain.com.

mail1   IN A       111.111.111.111

mail2   IN A       222.222.222.222

In this example, the “mail1” host is the preferred email exchange server.

We could also write that like this:

IN MX 10 mail1

IN MX 50 mail2

mail1   IN A       111.111.111.111

mail2   IN A       222.222.222.222

NS Records

This record type defines the name servers that are used for this zone.

You may be wondering, “if the zone file resides on the name server, why does it need to reference itself?”. Part of what makes DNS so successful is its multiple levels of caching. One reason for defining name servers within the zone file is that the zone file may be actually being served from a cached copy on another name server. There are other reasons for needing the name servers defined on the name server itself, but we won’t go into that here.

Like the MX records, these are zone-wide parameters, so they do not take hosts either. In general, they look like this:

IN NS     ns1.domain.com.

IN NS     ns2.domain.com.

You should have at least two name servers defined in each zone file in order to operate correctly if there is a problem with one server. Most DNS server software considers a zone file to be invalid if there is only a single name server.

As always, include the mapping for the hosts with A or AAAA records:

IN NS     ns1.domain.com.

IN NS     ns2.domain.com.

ns1     IN A     111.222.111.111

ns2     IN A     123.211.111.233

There are quite a few other record types you can use, but these are probably the most common types that you will come across.

Load Balancing

Load Balancing refers to efficiently distributing incoming network traffic across a group of back-end servers, also known as a “server farm” or “server pool”.

Modern high-traffic websites must serve hundreds of thousands, if not millions, of concurrent requests from users or clients and return the correct text, images, video, or application data, all in a fast and reliable manner. To cost effectively scale to meet these high volumes, modern computing best practice generally requires adding more servers.

A load balancer acts as the “traffic cop” sitting in front of your servers and routing client requests across all servers capable of fulfilling those requests in a manner that maximizes speed and capacity utilization and ensures that no one server is overworked, which could degrade performance. If a single server goes down, the load balancer redirects traffic to the remaining online servers. When a new server is added to the server group, the load balancer automatically starts to send requests to it.

In this manner, a load balancer performs the following functions:

  • Distributes client requests or network load efficiently across multiple servers
  • Ensures high availability and reliability by sending requests only to servers that are online
  • Provides the flexibility to add or subtract servers as demand dictates

Load Balancing Algorithms

Different load balancing algorithms provide different benefits; the choice depends on your needs:

  • Round-robin –Requests are distributed across the group of servers sequentially.
  • Least connections – A new request is sent to the server with the fewest current connections to clients. The relative computing capacity of each server is factored into determining which one has the least connections.
  • IP hash – The IP address of the client is used to determine which server receives the request.

Session Persistence

Information about a user’s session is often stored locally in the browser. For example, in a shopping cart application the items in a user’s cart might be stored at the browser level until the user is ready to purchase them. Changing which server receives requests from that client in the middle of the shopping session can cause performance issues or outright transaction failure. In such cases, it is essential that all requests from a client are sent to the same server for the duration of the session. This is known as session persistence.

The best load balancers can handle session persistence as needed. Another use case for session persistence is when an upstream server stores information requested by a user in its cache to boost performance. Switching servers would cause that information to be fetched for the second time, creating performance inefficiencies.

Dynamic Configuration of Server Groups

Many fast-changing applications require new servers to be added or taken down on a constant basis. This is common in environments such as the Amazon Elastic Compute Cloud (EC2), which enables users to pay only for the computing capacity they actually use, while at the same time ensuring that capacity scales up in response traffic spikes. In such environments it greatly helps if the load balancer can dynamically add or remove servers from the group without interrupting existing connections.

Hardware vs. Software Load Balancing

Load balancers typically come in two flavors: hardware-based and software-based. Vendors of hardware-based solutions load proprietary software onto the machine they provide, which often uses specialized processors. To cope with increasing traffic at your website, you have to buy more or bigger machines from the vendor. Software solutions generally run on commodity hardware, making them less expensive and more flexible. You can install the software on the hardware of your choice or in cloud environments like AWS EC2.

http://www.slideshare.net/AmazonWebServices/availability-scalability-with-elastic-load-balancing-route-53-cpn204-aws-reinvent-2013?qid=2d3e1261-4b90-41cf-b3cb-a8eeb804e11e&v=default&b=&from_search=2/

Proxy Server

A proxy server, also known as a “proxy” or “application-level gateway”, is a computer that acts as a gateway between a local network (e.g., all the computers at one company or in one building) and a larger-scale network such as the Internet. Proxy servers provide increased performance and security. In some cases, they monitor employees’ use of outside resources.

A proxy server works by intercepting connections between sender and receiver. All incoming data enters through one port and is forwarded to the rest of the network via another port. By blocking direct access between two networks, proxy servers make it much more difficult for hackers to get internal addresses and details of a private network.

Some proxy servers are a group of applications or servers that block common Internet services. For example, an HTTP proxy intercepts web access, and an SMTP proxy intercepts email. A proxy server uses a network addressing scheme to present one organization-wide IP address to the Internet. The server funnels all user requests to the Internet and returns responses to the appropriate users. In addition to restricting access from outside, this mechanism can prevent inside users from reaching specific Internet resources (e.g., certain websites). A proxy server can also be one of the components of a firewall.

Proxies may also cache web pages. Each time an internal user requests a URL from outside, a temporary copy is stored locally. The next time an internal user requests the same URL, the proxy can serve the local copy instead of retrieving the original across the network, improving performance.

Note: Do not confuse a proxy server with a NAT (Network Address Translation) device. A proxy server connects to, responds to, and receives traffic from the Internet, acting on behalf of the client computer, while a NAT device transparently changes the origination address of traffic coming through it before passing it to the Internet.

The Forward Proxy

When people talk about a proxy server (often simply known as a “proxy”), more often than not they are referring to a forward proxy. Let me explain what this particular server does.

A forward proxy provides proxy services to a client or a group of clients. Oftentimes, these clients belong to a common internal network like the one shown below.

fig7

When one of these clients makes a connection attempt to that file transfer server on the Internet, its requests have to pass through the forward proxy first.

Depending on the forward proxy’s settings, a request can be allowed or denied. If allowed, then the request is forwarded to the firewall and then to the file transfer server. From the point of view of the file transfer server, it is the proxy server that issued the request, not the client. So when the server responds, it addresses its response to the proxy.

But then when the forward proxy receives the response, it recognizes it as a response to the request that went through earlier. And so it in turn sends that response to the client that made the request.

Because proxy servers can keep track of requests, responses, their sources and their destinations, different clients can send out various requests to different servers through the forward proxy and the proxy will intermediate for all of them. Again, some requests will be allowed, while some will be denied.

As you can see, the proxy can serve as a single point of access and control, making it easier for you to enforce security policies. A forward proxy is typically used in tandem with a firewall to enhance an internal network’s security by controlling traffic originating from clients in the internal network that are directed at hosts on the Internet. Thus, from a security standpoint, a forward proxy is primarily aimed at enforcing security on client computers in your internal network.

But then client computers aren’t always the only ones you find in your internal network. Sometimes, you also have servers. And when those servers have to provide services to external clients (e.g. field staff who need to access files from your FTP server), a more appropriate solution would be a reverse proxy.

The Reverse Proxy

As its name implies, a reverse proxy does the exact opposite of what a forward proxy does. While a forward proxy proxies in behalf of clients (or requesting hosts), a reverse proxy proxies in behalf of servers. A reverse proxy accepts requests from external clients on behalf of servers stationed behind it just like what the figure below illustrates.

fig8

To the client in our example, it is the reverse proxy that is providing file transfer services. The client is oblivious to the file transfer servers behind the proxy, which are actually providing those services. In effect, whereas a forward proxy hides the identities of clients, a reverse proxy hides the identities of servers.

An Internet-based attacker would therefore find it considerably more difficult to acquire data found in those file transfer servers than if he wouldn’t have had to deal with a reverse proxy. No wonder reverse proxy servers like

Just like forward proxy servers, reverse proxies also provide a single point of access and control. You typically set it up to work alongside one or two firewalls to control traffic and requests directed to your internal servers.

In most cases, reverse proxy servers also act as load balancers for the servers behind it. Load balancers play a crucial role in providing high availability to network services that receive large volumes of requests. When a reverse proxy performs load balancing, it distributes incoming requests to a cluster of servers, all providing the same kind of service. So, for instance, a reverse proxy load balancing FTP services will have a cluster of FTP servers behind it.

Both types of proxy servers relay requests and responses between source and destination machines. But in the case of reverse proxy servers, client requests that go through them normally originate from the Internet, while, in the case of forward proxies, client requests normally come from the internal network behind them.

Install & Configure DNS Server

1.Network Information

We’ll use “mydns.net” domain as an example for this DNS installation. “mail”, “web”, “ns” are the hosts that resides within this domain.

It is possible to configure a single system to act as a caching name server, primary/master and secondary/slave. We will configure this DNS as a Primay/Master as well as Caching DNS server.

We’ll be installing DNS server on “10.16.0.24”.

  1. Install Bind

Install the bind9 package using the appropriate package management utilities for your Linux distributions.

On Debian/Ubuntu flavors, do the following:

$ sudo apt-get install bind9

On Redhat/CentOS/Fedora flavors, do the following:

# yum install bind9

All the DNS configurations are stored under /etc/bind directory. The primary configuration is /etc/bind/named.conf which will include other needed files. The file named /etc/bind/db.root describes the root nameservers in the world.

  1. Configure Cache NameServer

The job of a DNS caching server is to query other DNS servers and cache the response. Next time when the same query is given, it will provide the response from the cache. The cache will be updated periodically.

Please note that even though you can configure bind to work as a Primary and as a Caching server, it is not advised to do so for security reasons. Having a separate caching server is advisable.

All we have to do to configure a Cache NameServer is to add your ISP (Internet Service Provider)’s DNS server or any OpenDNS server to the file /etc/bind/named.conf.options.

For Example, we will use google’s public DNS servers, 8.8.8.8 and 8.8.4.4.

Uncomment and edit the following line as shown below in /etc/bind/named.conf.options file.

forwarders {

8.8.8.8;

8.8.4.4;

};

After the above change, restart the DNS server.

$ sudo service bind9 restart

  1. Test the Cache NameServer

You can use the dig command to test DNS services.

$ dig ubuntu.com

;; Query time: 1323 msec

Now when the second time you execute the dig, there should be an improvement in the Query time. As you see below, it took only 3 msec the second time, as it is getting the info from our caching DNS server.

$ dig ubuntu.com

;; Query time: 3 msec

  1. Configure Primary/Master Nameserver

Next, we will configure bind9 to be the Primary/Master for the domain/zone “mydns.net”.

As a first step in configuring our Primary/Master Nameserver, we should add Forward and Reverse resolution to bind9.

To add a DNS Forward and Reverse resolution to bind9, edit /etc/bind9/named.conf.local.

zone “mydns.net” {

type master;

file “/etc/bind/db.mydns.net”;

};

zone “0.16.10.in-addr.arpa” {

type master;

notify no;

file “/etc/bind/db.10”;

};

Now the file /etc/bind/db.mydns.net will have the details for resolving hostname to IP address for this domain/zone, and the file /etc/bind/db.10 will have the details for resolving IP address to hostname.

  1. Build the Forward Resolution for Primary/Master NameServer

Now we will add the details which is necessary for forward resolution into /etc/bind/db.mydns.net.

First, copy /etc/bind/db.local to /etc/bind/db.mydns.net

$ sudo cp /etc/bind/db.local /etc/bind/db.mydns.net

Next, edit the /etc/bind/db.mydns.net and replace the following.

  1. In the line which has SOA: localhost. – This is the FQDN of the server in charge for this domain. I’ve installed bind9 in 10.16.0.24, whose hostname is “ns”. So replace the “localhost.” with “ns.mydns.net.”. Make sure it end’s with a dot(.).
  2. In the line which has SOA: root.localhost. – This is the E-Mail address of the person who is responsible for this server. Use dot(.) instead of @.
  3. In the line which has NS: localhost. – This is defining the Name server for the domain (NS). We have to change this to the fully qualified domain name of the name server. Change it to “ns.mydns.net.”. Make sure you have a “.” at the end.

Next, define the A record and MX record for the domain. A record is the one which maps hostname to IP address, and MX record will tell the mailserver to use for this domain.

Once the changes are done, the /etc/bind/db.mydns.net file will look like the following:

$TTL   604800

@   IN SOA ns.mydns.net. user.localhost. (

1024       ; Serial

604800     ; Refresh

86400     ; Retry

2419200     ; Expire

604800 )   ; Negative Cache TTL

;

@   IN NS ns.mydns.net.

mydns.net.   IN     MX     10     mail.mydns.net.

ns IN A   10.16.0.24

web IN A   10.16.0.28

mail IN A   10.16.0.30

  1. Build the Reverse Resolution for Primary/Master NameServer

We will add the details which are necessary for reverse resolution to the file /etc/bind/db.10. Copy the file /etc/bind/db.127 to /etc/bind/db.10

$ sudo cp /etc/bind/db.127 /etc/bind/db.10

Next, edit the /etc/bind/db.10 file, and basically changing the same options as /etc/bind/db.mydns.net

$TTL   604800

@   IN SOA ns.mydns.net. root.localhost. (

20         ; Serial

604800     ; Refresh

86400     ; Retry

2419200     ; Expire

604800 )   ; Negative Cache TTL

;

@   IN NS ns.

Next, for each A record in /etc/bind/db.mydns.net, add a PTR record.

$TTL   604800

@   IN SOA ns.mydns.net. root.mydns.net. (

20     ; Serial

604800     ; Refresh

86400     ; Retry

2419200     ; Expire

604800 )   ; Negative Cache TTL

;

@   IN NS ns.

24   IN PTR ns.mydns.net.

30   IN PTR mail.mydns.net.

28   IN PTR web.mydns.net.

Whenever you are modifying the file db.mydns.net and db.10, you need to increment the “Serial” number as well. Typically admin uses DDMMYYSS for serial numbers and when they modify, the change the serial number appropriately.

Finally, restart the bind9 service:

$ sudo service bind9 restart

  1. Test the DNS server

Now we have configured the DNS server for our domain. We will test our DNS server by pinging mail.mydns.net from web.mydns.net.

If the ping is success, then we have configured the DNS successfully.

You can also use nslookup and dig to test DNS servers.

On server, add the following to /etc/resolv.conf

nameserver 10.16.0.24

Now ping, mail.mydns.net, which should resolve the address appropriately from the DNS server that we just configured.

$ ping mail.mydns.net

PING mail.mydns.net (10.16.0.30) 56(84) bytes of data.

64 bytes from mail.mydns.net (10.42.0.30): icmp_req=1 ttl=64 time=0.482 ms

64 bytes from mail.mydns.net (10.42.0.30): icmp_req=2 ttl=64 time=0.532 ms

Basic Linux Networking Concepts

Tags

, , , , , ,

There are so many resources in the Internet that helped me understand the different concepts related to Linux Networking, I am creating this post to serve as a Aggregation of all those concepts in a format that I found it easy to understand. I give actual credit to all those internet articles that I gathered this info from.

This post can probably help newbies who want to learn Linux Networking or people preparing for a interview and need a one stop shop to refresh all concepts 😉

IP Adddressing

Basic Terminology

  • IP Address—The unique number ID assigned to one host or interface in a network.
  • Subnet—A portion of a network sharing a particular subnet address.
  • Subnet mask—A 32-bit combination used to describe which portion of an address refers to the subnet and which part refers to the host.
  • Interface—A network connection.

An IP address is an address used in order to uniquely identify a device on an IP network. The address is made up of 32 binary bits, which can be divisible into a network portion and host portion with the help of a subnet mask. The 32 binary bits are broken into four octets (1 octet = 8 bits). Each octet is converted to decimal and separated by a period (dot). For this reason, an IP address is said to be expressed in dotted decimal format (for example, 172.16.81.100). The value in each octet ranges from 0 to 255 decimal, or 00000000 – 11111111 binary.

Here is how binary octets convert to decimal: The right most bit, or least significant bit, of an octet holds a value of 20. The bit just to the left of that holds a value of 21. This continues until the left-most bit, or most significant bit, which holds a value of 27. So if all binary bits are a one, the decimal equivalent would be 255 as shown here:

1 1 1 1 1 1 1 1

128 64 32 16 8 4 2 1 (128+64+32+16+8+4+2+1=255)

Here is a sample octet conversion when not all of the bits are set to 1.

0 1 0 0 0 0 0 1

0 64 0 0 0 0 0 1 (0+64+0+0+0+0+0+1=65)

And this is sample shows an IP address represented in both binary and decimal.

  1. 1.     23.     19 (decimal)

00001010.00000001.00010111.00010011 (binary)

These octets are broken down to provide an addressing scheme that can accommodate large and small networks. There are five different classes of networks, A to E. This document focuses on addressing classes A to C, since classes D and E are reserved and discussion of them is beyond the scope of this document.

Given an IP address, its class can be determined from the three high-order bits. Figure 1 shows the significance in the three high order bits and the range of addresses that fall into each class. For informational purposes, Class D and Class E addresses are also shown.

Figure 1

figure1

In a Class A address, the first octet is the network portion, so the Class A example in Figure 1 has a major network address of 1.0.0.0 – 127.255.255.255. Octets 2, 3, and 4 (the next 24 bits) are for the network manager to divide into subnets and hosts as he/she sees fit. Class A addresses are used for networks that have more than 65,536 hosts (actually, up to 16777214 hosts!).

In a Class B address, the first two octets are the network portion, so the Class B example in Figure 1 has a major network address of 128.0.0.0 – 191.255.255.255. Octets 3 and 4 (16 bits) are for local subnets and hosts. Class B addresses are used for networks that have between 256 and 65534 hosts.

In a Class C address, the first three octets are the network portion. The Class C example in Figure 1 has a major network address of 192.0.0.0 – 223.255.255.255. Octet 4 (8 bits) is for local subnets and hosts – perfect for networks with less than 254 hosts.

Network Masks

A network mask helps you know which portion of the address identifies the network and which portion of the address identifies the node. Class A, B, and C networks have default masks, also known as natural masks, as shown here:

Class A: 255.0.0.0

Class B: 255.255.0.0

Class C: 255.255.255.0

An IP address on a Class A network that has not been subnetted would have an address/mask pair similar to: 8.20.15.1 255.0.0.0. To see how the mask helps you identify the network and node parts of the address, convert the address and mask to binary numbers.

8.20.15.1 = 00001000.00010100.00001111.00000001

255.0.0.0 = 11111111.00000000.00000000.00000000

Once you have the address and the mask represented in binary, then identifying the network and host ID is easier. Any address bits which have corresponding mask bits set to 1 represent the network ID. Any address bits that have corresponding mask bits set to 0 represent the node ID.

8.20.15.1 = 00001000.00010100.00001111.00000001

255.0.0.0 = 11111111.00000000.00000000.00000000

———————————–

net id |     host id

netid = 00001000 = 8

hostid = 00010100.00001111.00000001 = 20.15.1

Understanding Subnetting

Subnetting allows you to create multiple logical networks that exist within a single Class A, B, or C network. If you do not subnet, you are only able to use one network from your Class A, B, or C network, which is unrealistic.

Each data link on a network must have a unique network ID, with every node on that link being a member of the same network. If you break a major network (Class A, B, or C) into smaller subnetworks, it allows you to create a network of interconnecting subnetworks. Each data link on this network would then have a unique network/subnetwork ID. Any device, or gateway, connecting n networks/subnetworks has n distinct IP addresses, one for each network / subnetwork that it interconnects.

In order to subnet a network, extend the natural mask using some of the bits from the host ID portion of the address to create a subnetwork ID. For example, given a Class C network of 204.17.5.0 which has a natural mask of 255.255.255.0, you can create subnets in this manner:

204.17.5.0 –     11001100.00010001.00000101.00000000

255.255.255.224 – 11111111.11111111.11111111.11100000

————————–|sub|—-

By extending the mask to be 255.255.255.224, you have taken three bits (indicated by “sub”) from the original host portion of the address and used them to make subnets. With these three bits, it is possible to create eight subnets. With the remaining five host ID bits, each subnet can have up to 32 host addresses, 30 of which can actually be assigned to a device since host ids of all zeros or all ones are not allowed (it is very important to remember this). So, with this in mind, these subnets have been created.

204.17.5.0 255.255.255.224     host address range 1 to 30

204.17.5.32 255.255.255.224   host address range 33 to 62

204.17.5.64 255.255.255.224   host address range 65 to 94

204.17.5.96 255.255.255.224   host address range 97 to 126

204.17.5.128 255.255.255.224   host address range 129 to 158

204.17.5.160 255.255.255.224   host address range 161 to 190

204.17.5.192 255.255.255.224   host address range 193 to 222

204.17.5.224 255.255.255.224   host address range 225 to 254

Note:There are two ways to denote these masks. First, since you are using three bits more than the “natural” Class C mask, you can denote these addresses as having a 3-bit subnet mask. Or, secondly, the mask of 255.255.255.224 can also be denoted as /27 as there are 27 bits that are set in the mask. This second method is used with CIDR. With this method, one of these networks can be described with the notation prefix/length. For example, 204.17.5.32/27 denotes the network 204.17.5.32 255.255.255.224. When appropriate the prefix/length notation is used to denote the mask throughout the rest of this document.

The network subnetting scheme in this section allows for eight subnets, and the network might appear as:

Figure 2

figure2

Notice that each of the routers in Figure 2 is attached to four subnetworks, one subnetwork is common to both routers. Also, each router has an IP address for each subnetwork to which it is attached. Each subnetwork could potentially support up to 30 host addresses.

This brings up an interesting point. The more host bits you use for a subnet mask, the more subnets you have available. However, the more subnets available, the less host addresses available per subnet. For example, a Class C network of 204.17.5.0 and a mask of 255.255.255.224 (/27) allows you to have eight subnets, each with 32 host addresses (30 of which could be assigned to devices). If you use a mask of 255.255.255.240 (/28), the break down is:

204.17.5.0 –     11001100.00010001.00000101.00000000

255.255.255.240 – 11111111.11111111.11111111.11110000

————————–|sub |—

Since you now have four bits to make subnets with, you only have four bits left for host addresses. So in this case you can have up to 16 subnets, each of which can have up to 16 host addresses (14 of which can be assigned to devices).

Take a look at how a Class B network might be subnetted. If you have network 172.16.0.0 ,then you know that its natural mask is 255.255.0.0 or 172.16.0.0/16. Extending the mask to anything beyond 255.255.0.0 means you are subnetting. You can quickly see that you have the ability to create a lot more subnets than with the Class C network. If you use a mask of 255.255.248.0 (/21), how many subnets and hosts per subnet does this allow for?

172.16.0.0 –   10101100.00010000.00000000.00000000

255.255.248.0 – 11111111.11111111.11111000.00000000

—————–| sub |———–

You are using five bits from the original host bits for subnets. This allows you to have 32 subnets (25). After using the five bits for subnetting, you are left with 11 bits for host addresses. This allows each subnet so have 2048 host addresses (211), 2046 of which could be assigned to devices.

Note:In the past, there were limitations to the use of a subnet 0 (all subnet bits are set to zero) and all ones subnet (all subnet bits set to one). Some devices would not allow the use of these subnets. Cisco Systems devices allow the use of these subnets when the ip subnet zero command is configured.

CIDR

Classless Interdomain Routing (CIDR) was introduced to improve both address space utilization and routing scalability in the Internet. It was needed because of the rapid growth of the Internet and growth of the IP routing tables held in the Internet routers.

CIDR moves way from the traditional IP classes (Class A, Class B, Class C, and so on). In CIDR , an IP network is represented by a prefix, which is an IP address and some indication of the length of the mask. Length means the number of left-most contiguous mask bits that are set to one. So network 172.16.0.0 255.255.0.0 can be represented as 172.16.0.0/16. CIDR also depicts a more hierarchical Internet architecture, where each domain takes its IP addresses from a higher level. This allows for the summarization of the domains to be done at the higher level. For example, if an ISP owns network 172.16.0.0/16, then the ISP can offer 172.16.1.0/24, 172.16.2.0/24, and so on to customers. Yet, when advertising to other providers, the ISP only needs to advertise 172.16.0.0/16.

NAT (Network Address Translation)

Primarily NAT was introduced to the world of IT and networking due to the lack of IP addresses, or looking at it from another view, due to the vast amount of growing IT technology relying on IP addresses. To add to this, NAT adds a layer of security, by hiding computers, servers and other IT equipment from the outside world.

How NAT works

When computers and servers within a network communicate, they need to be identified to each other by a unique address, in which resulted in the creation of a 32 bit number, and the combinations of these 32 bits would accommodate for over 4 billion unique addresses, known as IP address. This was named IPv4, and although over 4 billion addresses sounds a lot, it really is not considering how fast the world of computers and the internet has grown.

To circumvent this problem, a temporary solution was produced known as NAT. NAT resulted in two types of IP addresses, public and private. A range of private addresses were introduced, which anyone could use, as long as these were kept private within the network and not routed on the internet. The range of private addresses known as RFC 1918 are;

Class A 10.0.0.0 – 10.255.255.255

Class B 172.16.0.0 – 172.31.255.255

Class C 192.168.0.0 – 192.168.255.255

NAT allows you to use these private IP address on the internal network. So within your private network you would assign a unique IP address to all your computers, servers and other IP driven resources, usually done via DHCP. Another company can use the same private IP addresses as well, as long as they are kept internal to their network. So two companies maybe using the same range of IP addresses but because they are private to their network, they are not conflicting with each other.

However when internal hosts do need to communicate to the public network (Internet) then this is where a public address comes into the equation. This address usually purchased from an ISP is a routable public address everyone can see, which would represent your network gateway. This public address would be unique, no one else would use this address.

Now getting to the point; When a host on the internal network with an internal IP address does need to communicate outside it’s private network, it would use the public IP address on the network’s gateway to identify itself to the rest of the world, and this translation of converting a private IP address to public is done by NAT. For example a computer on an internal address of 192.168.1.10 wanted to communicate with a web server somewhere on the internet, NAT would translate the address 192.168.1.10 to the company’s public address, lets call this 1.1.1.1 for example. so that the internal address is identified as the public address when communicating with the outside world. This has to be done because when the web server somewhere on the internet was to reply to this internal computer, it needs to send this to a unique and routable address on the internet, the public address. It can not use the original address of 192.168.1.10, as this is private, none routable and hidden from the outside world. This address, of 1.1.1.1 would be the address of the public address for that company and can be seen by everyone. Now the web server would reply to that public address, 1.1.1.1. NAT would then use its records to translate the packets received from the web server that was destined to 1.1.1.1 back to the internal network address of 192.168.1.10, and though the computer who requested the original info, will receive the requested packets.

Now you can obviously see the two benefits of NAT. Firstly it would save on the IP addresses we use, as every single computer does not need a public address, and also it would hide these private computers from the outside world. Everyone can only see the public address, the rest is hidden behind this public address. So from the internet only the public address on the external interface of the firewall or router can be seen, and nothing beyond it.

Types of NAT

Three main types of NAT rules are used today depending on what needs to be accomplished;

Static NAT

A pool of public IP addresses are assigned to the NAT device. A private IP address can then be statically mapped to anyone of these public addresses. This type of NATTING scheme is usually used for servers requiring the same IP address always, hence the name “static”, so server 1 will always have the same IP address assigned to it, server 2 will have a different public IP address assigned to it and so on.

Dynamic NAT

Again the NAT device will consist of a pool of IP addresses. This time though the pool of IP addresses will be used when needed and then given back to the pool. So if computer A needed a public address, it would take one from the pool, then hand it back when done. The next time the same computer wanted an IP address it may be assigned a different public address from the pool, because the one used previously may be in use by another computer, hence the name “dynamic”. So users who want to communicate on the internet at any one time will be limited by how many public IP addresses are available in the NAT pool. A company would purchase a number of public IP’s depending on their need.

Port Address Translation (PAT)

In this type of setup, a company would only have one public IP address assigned to their network, and so everyone would share this one public address when using the internet, browsing the web for example. Yes, you may be asking how can everyone share one address, well the clue lies within the name, Port address translation. When a computer wants to use the internet, the NAT device, using the PAT method will remember the IP address and source port of the internal host. For example 192.168.1.10 with a source port of 55331 wanted to browse Amazon.com. The NAT device will keep a note of this, and when Amazon replies to the public address and the port number of 55331, the NAT device will use the PAT method and look up the port information which maps to the internal computer requesting it. So it would be saying, this information Amazon has sent back to the public address and port number 55331, maps to the IP address 192.168.1.10 who originally requested it, though the information is for that computer. So the connections are uniquely identified by a source port, all using the same public IP but with unique source ports to identify who requested what information.

A company would save a reasonable amount of money and IP addresses using this method because it is only using one IP address. This has been a major factor to why IPv6 has been mentioned for some years now but still not required in most countries.

NAT is also implemented in home based routers and hardware firewalls such as the Netgear’s and the Linksys of this world as well as the high end hardware firewalls such as the likes of Cisco and Juniper.

This has proved a valuable feature on hardware firewalls for saving public IP addresses and also a countermeasure for some types of attacks such as a reconnaissance attack.

Disadvantages of NAT

As with everything, NAT does have it’s drawbacks. Some applications and services such as VPN and video conferencing struggle to process via NAT (Not entirely true as you can most of the time get them configured to work with NAT, but can get a little messy when setting rules up in applications,, routers and firewalls).

Check out http://www.9tut.com/network-address-translation-nat-tutorial for more details on NAT.

What is DHCP?

Dynamic Host Configuration Protocol (DHCP) is a network protocol that enables a server to automatically assign an IP address to a computer from a defined range of numbers (i.e., a scope) configured for a given network.

DHCP assigns an IP address when a system is started, for example:

  1. A user turns on a computer with a DHCP client.
  2. The client computer sends a broadcast request (called a DISCOVER or DHCPDISCOVER), looking for a DHCP server to answer.
  3. The router directs the DISCOVER packet to the correct DHCP server.
  4. The server receives the DISCOVER packet. Based on availability and usage policies set on the server, the server determines an appropriate address (if any) to give to the client. The server then temporarily reserves that address for the client and sends back to the client an OFFER (or DHCPOFFER) packet, with that address information. The server also configures the client’s DNS servers, WINS servers, NTP servers, and sometimes other services as well.
  5. The client sends a REQUEST (or DHCPREQUEST) packet, letting the server know that it intends to use the address.
  6. The server sends an ACK (or DHCPACK) packet, confirming that the client has a been given a lease on the address for a server-specified period of time.

Ethernet

The most common type of LAN hardware is known as Ethernet. In its simplest form, it consists of a single cable with hosts attached to it through connectors, taps, or transceivers. Simple Ethernets are relatively inexpensive to install, which together with a net transfer rate of 10, 100, or even 1,000 Megabits per second, accounts for much of its popularity.

Ethernets come in three flavors: thick, thin, and twisted pair.

Linux Network Devices :

The Linux kernel supports a number of hardware drivers for various types of equipment. This section gives a short overview of the driver families available and the interface names they use.

There is a number of standard names for interfaces in Linux, which are listed here. Most drivers support more than one interface, in which case the interfaces are numbered, as in eth0 and eth1:

lo

This is the local loopback interface. It is used for testing purposes, as well as a couple of network applications. It works like a closed circuit in that any datagram written to it will immediately be returned to the host’s networking layer. There’s always one loopback device present in the kernel, and there’s little sense in having more.

eth0, eth1, …

These are the Ethernet card interfaces. They are used for most Ethernet cards, including many of the parallel port Ethernet cards.

tr0, tr1, …

These are the Token Ring card interfaces. They are used for most Token Ring cards, including non-IBM manufactured cards.

sl0, sl1, …

These are the SLIP interfaces. SLIP interfaces are associated with serial lines in the order in which they are allocated for SLIP.

ppp0, ppp1, …

These are the PPP interfaces. Just like SLIP interfaces, a PPP interface is associated with a serial line once it is converted to PPP mode.

plip0, plip1, …

These are the PLIP interfaces. PLIP transports IP datagrams over parallel lines. The interfaces are allocated by the PLIP driver at system boot time and are mapped onto parallel ports. In the 2.0.x kernels there is a direct relationship between the device name and the I/O port of the parallel port, but in later kernels the device names are allocated sequentially, just as for SLIP and PPP devices.

ax0, ax1, …

These are the AX.25 interfaces. AX.25 is the primary protocol used by amateur radio operators. AX.25 interfaces are allocated and mapped in a similar fashion to SLIP devices.

There are many other types of interfaces available for other network drivers. We’ve listed only the most common ones.

Network Configuration Files

/etc/resolve.conf       →         List servers for internet domain name resolution.

/etc/hosts                                           →         Lists hosts to be resolved locally

Red Hat/Fedora/CentOS

/etc/sysconfig/network        →         Specify network configuration. eg. Static , , , etc.

/etc/sysconfig/network-scripts/ifcfg-device              →       Specify TCP network information.

/etc/network/interfaces                 →       Specify network configuration and devices. eg. Static IP and info, DHCP, etc.

Configuring Hostname

This is a three step process:

  1. Issue the command: hostname new-host-name
  2. Change network configuration file: /etc/sysconfig/network
    Edit entry: HOSTNAME=new-host-name
  3. Restart systems which relied on the hostname (or reboot):
    • Restart network services: service network restart
      (or: /etc/init.d/network restart)
    • Restart desktop:
      • Bring down system to console mode: init 3
      • Bring up X-Windows: init 5

One may also want to check the file /etc/hosts for an entry using the system name which allows the system to be self aware.

Activating & De-activating your NIC

Commands for starting and stopping TCP/IP network services on a Network Interface Card (NIC):

  • Activate: /sbin/ifup eth0
    (Also: ifconfig eth0 up – Note: Even if no IP address is assigned you can listen.)
  • De-Activate: /sbin/ifdown eth0
    (Also: ifconfig eth0 down)

These scripts use the scripts and NIC config
files in /etc/sysconfig/network-scripts/

Configuring Static IP on Redhat Machine

You will need the following information in order to complete this configuration:

  • IP Address
  • Gateway Address
  • Broadcast Address
  • Netmask
  • Network Address
  • DNS Server Addresses
  • System Hostname

The first thing is to turn off NetworkManager.This service is responsible for requesting DHCP addresses and configuring the network interfaces. Since we are setting them statically we do not need it.

First let’s stop NetworkManager:
service NetworkManager stop
or /etc/init.d/NetworkManager stop

Now let’s make sure it does not start at boot:
chkconfig NetworkManager off

Let’s also make sure the network service is set to start at boot:
chkconfig network on

Now that NM (NetworkManager) is out of the way we need to edit three files.

/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/resolv.conf

First let’s edit /etc/sysconfig/network. Here we will need to tell the system to turn on networking, the hostname of the machine and the gateway. Open the file in your favorite text editor and add or change the following lines. Of course you will need to make sure the configuration matches your system, this is just an example.
NETWORKING=yes
HOSTNAME=server.domain.com
GATEWAY=192.168.1.1

Now save and close that file. Let’s move on to editing the /etc/sysconfig/network-scripts/ifcfg-eth0 file. Open the file in your favorite text editor and add or change the following lines.
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.1.255
IPADDR=192.168.1.10
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes

Now that the network interface is taken care of we need the final important piece of the puzzle, DNS.Without DNS you will not be able to go to a website by name, on connect to anything else on the network unless you know the IP address. To tell the system what DNS servers to use we edit the /etc/resolv.conf file. Open the file in your favorite editor and add or change the following lines (search line is optional).
search domain.com
nameserver 192.168.1.2
nameserver 192.168.1.3

Now that we have given the system the necessary information all we need to do is restart the network service.
service network restart

Configuring dynamic via DHCP

Edit the following two files.

/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-eth0

First let’s edit /etc/sysconfig/network. Here we will need to tell the system to turn on networking, the hostname of the machine and the gateway. Open the file in your favorite text editor and add or change the following lines. Of course you will need to make sure the configuration matches your system, this is just an example.
NETWORKING=yes
HOSTNAME=server.domain.com

Now save and close that file. Let’s move on to editing the /etc/sysconfig/network-scripts/ifcfg-eth0 file. Open the file in your favorite text editor and add or change the following lines.
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes

Now that we have given the system the necessary information all we need to do is restart the network service.
service network restart

Configuring Network Card

Red hat Enterprise Linux version 5.x and 4.x provides the following tools to make changes to network configuration such as add new card, assign/change IP address/subnet/gateway, and change DNS server and more.

[a] GUI tool – system-config-network

[b] Command line text based GUI tool (No X Windows/Gnome/KDE required) – system-config-network

[c] Edit configuration files stored in /etc/sysconfig/network-scripts/ directory. This method works with remote server over the ssh based session.

1. GUI tool system-config-network

Open the X terminal or login using ssh over X based session command (ssh -X user@server-name-here). Type the following command at shell prompt:

Warning: It is important that you configure network interface cards correctly over ssh -X based session; otherwise, you will be locked out due to wrong network configuration.

$ system-config-network &
Sample outputs:

figure3

Fig.01: Configuring the RHEL/CentOS server network card using GUI tool

You will see a Window as above. Next, select your Ethernet card (such as eth0 or eth1) and click on the Edit button. You can now setup/modify IP address, netmask, default gateway and other properties. Here is an example from my personal RHEL 5.x server:

fig4

Fig.02: Setting or modifying IPv4 and IPv6 properties on RHEL/CentOS based server

You can obtain IP address using DHCP or setup manually. Once IP address assigned, click on Ok button to save the changes. You can activate card by clicking on Activate button.

2. Command line tool system-config-network-tui

If you don’t have X windows GUI (gnome/kde desktop) installed on RHEL/CentOS/Fedora based system, than type the following command at shell prompt (this method also works on remote server using ssh based session):

Warning: It is important that you configure network interface cards correctly over ssh based session; otherwise, you will be locked out due to wrong network configuration.

# system-config-network-tui &
Sample outputs:

fig5

Fig.03: RHEL/CentOS command line network config tool (click to large)

Select your Ethernet card such as eth0 or eth1 and hit [Enter] or [F12] special key to configure IP properties for selected NIC:

fig6

Fig.04: Setting up IPv4 properties on RHEL/CentOS (click to large)

You can obtain an IP address using DHCP or setup IP address manually. Once an IP address assigned, click on the Ok button to save the changes.

Edit configuration files stored in /etc/sysconfig/network-scripts/ directory

You can configure network card by editing text files stored in /etc/sysconfig/network-scripts/ directory. Open the terminal or login using ssh. Next, change directory to /etc/sysconfig/network-scripts/:
# cd /etc/sysconfig/network-scripts/
You need to edit / create files as follows using a text editor such as vi:

  • /etc/sysconfig/network-scripts/ifcfg-eth0 : First Ethernet card configuration file.
  • /etc/sysconfig/network-scripts/ifcfg-eth1 : Second Ethernet card configuration file.

Examples: Edit eth0 configuration file

To edit/create first NIC file, type the following command in /etc/sysconfig/network-scripts/ directory:

# vi ifcfg-eth0
OR
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Edit or modify as follows for static ip configuration:

# eth0 – Intel Corporation 82573E Gigabit Ethernet Controller (Copper) on server1.cyberciti.biz by nixCraft on 10/4/2007

DEVICE=eth0

BOOTPROTO=static

DHCPCLASS=

HWADDR=00:30:48:56:A6:2E

IPADDR=10.10.29.66

NETMASK=255.255.255.192

ONBOOT=yes

Save and close the file.

Setting up default gateway and server (host) name

You need to define a default gateway (router IP) and hostname in /etc/sysconfig/network file. Edit /etc/sysconfig/network, enter:
# vi /etc/sysconfig/network
Append or modify configuration as follows:

NETWORKING=yes

HOSTNAME=www1.nixcraft.in

GATEWAY=10.10.29.65

Save and close the file. Finally, you need to restart the network service, run:
# /etc/init.d/network restart
OR
# /sbin/service network restart

Setting up dns server

Make sure you have correct DNS server defined in /etc/resolv.conf file:
# vi /etc/resolv.conf
Setup DNS server IP address as follows (feel free to replace 10.0.80.11, 10.0.80.12 as per your setup):
nameserver 10.0.80.11
nameserver 10.0.80.12
nameserver 202.67.222.222

Save and close the file. Now you can ping the gateway/other hosts using the ping command:
$ ping 10.0.80.12

Configuring DHCP Server

The DHCP protocol lets a DHCP client to lease network configuration parameters such as an IP address. In fact parameters to lease are not limited to IP address only and they also include:

  • IP addresses and network masks
  • Domain Names servers ( DNS )
  • Default Gateways
  • WINS servers
  • Syslog hosts
  • Proxy servers
  • NTP servers
  • X Font servers
  • Syslog hosts

Each host set to obtain an IP address dynamically will upon boot send a DHCP request over the network ( by definition this is a broadcast of all 1’s ) to discover whether there is a DHCP server available on the network and consequently ask for an network configuration. DHCP client is then obligated to maintain a communication with DHCP server and renew its IP address regularly as dictated by IP address’s lease time expiry. In case that DHCP client fails to renew its IP address ( disconnection, client is turned off and etc. ) its IP address expires and DHCP server is free to lease this IP address to another DHCP client.

DHCP server keeps a record of all leased IP addresses and stores them into a file called dhcpd.leases which can be found in /var/lib/dhcp directory ( location of this file may vary depending on Linux system in use ). Having such a file allows DHCP server to keep track of all IP address leases even after its reboot or power failure.

Here are some advantages of having a DHCP server connected to network:

  • No IP address conflicts. DHCP can guarantee that all hosts on the network will have unique IP address. DHCP server keeps a record of all IP addresses assigned and cross reference them with host’s MAC addresses.
  • Based on the MAC address DHCP allows for a fixed parameter configuration for a specific host
  • Efficiency with minimum local client configuration

1.DHCP Server Installation

Standard DHCP server implementation available in various Linux distributions is an Open source version maintained by ISC ( Internet System Consortium ). There are currently 3 major versions 2, 3, 4 where version 3 supports backup servers, and version 4 supports IPv6. This article deals only with ISC DHCP v3.

Use the following commands to install DHCP on your Linux server:

Debian and Ubuntu:

# apt-get install dhcp3-server

Redhat & Fedora:

# yum install dhcp

2.Basic DHCP Configuration

By default DHCP server configuration does not include any subnets on which DHCP server should lease IP addresses. Therefore, depends on your Linux system you may get a following error message when you attempt to start DHCP with default dhcpd.conf configuration file.

Starting ISC DHCP server: dhcpdcheck syslog for diagnostics. … failed!

Checking a log files and particularly /var/log/syslog reveals this problem in more detail:

No subnet declaration for eth0 (some IP address).

As it happens very often your server may be connected to multiple network subnets. In order to start DHCP server at least one subnet must be defined in DHCP configuration file /etc/dhcp/dhcpd.conf.

NOTE: if your server has access to more than one subnet, DHCP requires all subnets to be defined even though there isn’t immediate intention to enable DHCP service on that subnet.

Below is the simplest example of DHCP configuration file:

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

}

subnet 192.168.0.0 netmask 255.255.0.0 {

}

This configuration file instructs DHCP server to listen for DHCP client requests on subnet 10.1.1.0 with netmask 255.255.255.0. Furthermore, it will assign IP addresses in range 10.1.1.3 – 10.1.1.254. It also defines an empty definition of subnet with network ID 192.168.0.0.

Alter above code with your subnet and insert it into /etc/dhcp/dhcpd.conf. When ready restart your DHCP server with ( restart command may vary ) :

# /etc/init.d/isc-dhcp-server restart

3.DHCP default and max lease time

At this point we can add to our DHCP configuration another setting and that is to set default and max lease time expiry.

  • default-lease-time is a value in seconds in which a leased IP address expiry will be set to if DHCP client does not ask for any other specific expiry lease time
  • max-lease-time is a value in seconds which defines a maximum expiry time for an IP address leased by DHCP server

default-lease-time 600;

max-lease-time 7200;

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

}

subnet 192.168.0.0 netmask 255.255.0.0 {

}

4.Define DNS server

Another configuration parameter possible to be set by DHCP server to its client is a definition of DNS server. If you want your clients to use DNS server with an IP address 8.8.8.8 and 10.1.1.1 you can do it by including an option “domain-name-servers” to DHCP’s configuration file.

default-lease-time 600;

max-lease-time 7200;

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

option domain-name-servers 10.1.1.1, 8.8.8.8;

}

subnet 192.168.0.0 netmask 255.255.0.0 {

}

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

option routers 10.1.1.1;

}

5.Set default gateway

DHCP also allows for client’s gateway configuration.To set any client on the local network to use default gateway 10.1.1.1, add line “option routers 10.1.1.1” into dhcpd.conf file as demonstrated below:

default-lease-time 600;

max-lease-time 7200;

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

option domain-name-servers 10.1.1.1, 8.8.8.8;

option routers 10.1.1.1;

}

subnet 192.168.0.0 netmask 255.255.0.0 {

}

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

option routers 10.1.1.1;

}

DHCP will now set DHCP client with gateway 10.1.1.1.

6.Host specific configuration

There maybe a need to set static IP address to a particular host on the network such as printer, web server and etc. In this case it is posible to amend DHCP server configuration to lease a choosen IP address to a specific host defined by its MAC address.

default-lease-time 600;

max-lease-time 7200;

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254;

option domain-name-servers 10.1.1.1, 8.8.8.8;

option routers 10.1.1.1;

}

subnet 192.168.0.0 netmask 255.255.0.0 {

}

host printer {

hardware ethernet 00:16:d3:b7:8f:86;

fixed-address 10.1.1.100;

}

host web-server {

hardware ethernet 00:17:a4:c2:44:22;

fixed-address 10.1.1.200;

}

The above DHCP configuration file will permanently assign the IP address 10.1.1.100 to a host “printer” with a MAC address 00:16:d3:b7:8f:86 and IP address 10.1.1.200 to host “web-server” with MAC address 00:17:a4:c2:44:22.

Basics of IPtables in Linux

Tags

, , , , , , ,

A firewall is a program running on a Gateway, Bridge or PC/Laptop/Smartphone that is capable of filtering incoming, outgoing, and forwarded network packets. A firewall is essentially a tool that lets you restrict you or your network’s access to the Internet, and someone else’s access from the Internet to your network.

Linux comes with a host based firewall called Netfilter. According to the official project site:

netfilter is a set of hooks inside the Linux kernel that allows kernel modules to register callback functions with the network stack. A registered callback function is then called back for every packet that traverses the respective hook within the network stack.

This Linux based firewall is controlled by the program called iptables to handles filtering for IPv4, and ip6tables handles filtering for IPv6.

IP Tables is a Lower Level Firewall Implementation Solution available in Linux Systems.

Most Distro’s have other Firewalll options as well:

For examples,

RHEL/ CentOS has Firewalld. Firewalld is a easier to use/implement IP Tables but some programs are not compatible, and would direct interaction with IP Tables.

SELinux ( Security Enhanced Linux) : Here is more info of you are curious about this topic:

https://www.digitalocean.com/community/tutorials/an-introduction-to-selinux-on-centos-7-part-1-basic-concepts

IP Tables provides basic low level routing system for how Linux deals with IP Traffic/ Packets.

Contains Several Important Tables:

1) NAT – NW Address Translation

Network address translation (NAT) is a methodology of remapping one IP address space into another by modifying network address information in Internet Protocol (IP) packet headers while they are in transit across a traffic routing device

This is the basic method through which several VMs running on same Hardware Server (Hypervisor) can get unique IP/MAC Addresses etc. The Virtualization programs like VMWare ESXi/ KVM would modify the IP Packets that are received by Hypervisors NIC and map them to appropriate IP Addresses of VM’s the program is managing.

In Cloud Computing world, we also use NAT to provide Outbound Internet Access to VMs running in a private subnet.

2) MANGLE – for modifying packets

3) Filter – Decide which packets gets processed

IP Tables deals with Tables

Tables have Chains

A Chain is a sequential list of rules

A Rule defines particular kinds of packets and where they go.

Filter Table: Default Chains

INPUT

OUTPUT

FORWARD

Each Rule has 2 things:

1) Specification of type of packet – criteria for matching

2) Routing policy – ACCEPT, DROP, REJECT etc

Iptables Config File

The default config files for RHEL / CentOS / Fedora Linux are:

  • /etc/sysconfig/iptables – The system scripts that activate the firewall by reading this file.

Turn On Firewall

Type the following two commands to turn on firewall:

chkconfig iptables on       #starts the service on boot

service iptables start

service iptables restart     # restart the firewall

service iptables stop                 # stop the firewall

Understanding Firewall

There are total 3 chains:

  1. INPUT – The default chain is used for packets addressed to the system. Use this to open or close incoming ports (such as 80,25, and 110 etc) and ip addresses / subnet (such as 202.54.1.20/29).
  2. OUTPUT – The default chain is used when packets are generating from the system. Use this open or close outgoing ports and ip addresses / subnets.
  3. FORWARD – The default chains is used when packets send through another interface. Usually used when you setup Linux as router. For example, eth0 connected to ADSL/Cable modem and eth1 is connected to local LAN. Use FORWARD chain to send and receive traffic from LAN to the Internet.

Packet Matching Rules

  1. Each packet starts at the first rule in the chain .
  2. A packet proceeds until it matches a rule.
  3. If a match found, then control will jump to the specified target (such as REJECT, ACCEPT, DROP).

Target Meanings

  1. The target ACCEPT means allow packet.
  2. The target REJECT means to drop the packet and send an error message to remote host.
  3. The target DROP means drop the packet and do not send an error message to remote host or sending host.

Iptables Match Criteria

-t <-table-> If you don’t specify a table, then the filter table is assumed. The possible built-in tables include: filter, nat, mangle
-j <target> Jump to the specified target chain when the packet matches the current rule.
-A Append rule to end of a chain
-F Flush. Deletes all the rules in the selected table
-p <protocol-type> Match protocol. Types include, icmp, tcp, udp, and all
-s <ip-address> Match source IP address
-d <ip-address> Match destination IP address
-i <interface-name> Match “input” interface on which the packet enters.
-o <interface-name> Match “output” interface on which the packet exits

Examples to Understand How To Write a Firewall Rule

  1. Rule to accept tcp packets coming on interface eth0 from any address to my machine

         iptables -A INPUT -s 0/0 -i eth0 -d 192.168.1.1 -p TCP -j ACCEPT

0/0 – Means any IP address

  1. Rule to allow ssh connection from a network

         iptables -A INPUT -i eth0 -p tcp -s 192.168.1.0/25 –dport 22 -m state –state NEW,ESTABLISHED     -j ACCEPT

         iptables -A OUTPUT -o eth0 -p tcp –sport 22 -m state –state ESTABLISHED -j ACCEPT

  1. Rule to restrict ICMP echo requests no more than one per second. This will allow the system to filter traffice that characterize DOS.

         iptables -A INPUT -p tcp –syn -m limit –limit 5/s -i eth0 -j ACCEPT

–syn – used to identify a new tcp connection

A distributed denial-of-service (DDoS) attack occurs when multiple systems flood the bandwidth or resources of a targeted system, usually one or more web servers. Such an attack is often the result of multiple compromised systems (for example a botnet) flooding the targeted system with traffic.

  1. Rule to Forward packets.

iptables -A FORWARD -s 0/0 -i eth0 -d 192.168.1.2 -o eth1 -p TCP –sport 1024:65535 -m multiport    –dports 80,443 -j ACCEPT

         iptables -A FORWARD -d 0/0 -o eth0 -s 192.168.1.2 -i eth1 -p TCP -m state –state ESTABLISHED -j ACCEPT

  1. Rule to block an IP address range

iptables -A INPUT -s 192.168.1.0/24 -j DROP

There are many blogs that are available online about iptable rules. Following are some of thee references:

http://www.linuxhowtos.org/Security/iptables.htm

http://www.tecmint.com/basic-guide-on-iptables-linux-firewall-tips-commands/

http://www.thegeekstuff.com/2011/06/iptables-rules-examples/

Understanding the OpenStack Platform relative to AWS

Tags

, , , , , , , , , , , , , , , , , , , ,

Foreword:

Notes on Choosing a Cloud Provider – There are currently many Big players in the market:

  • AWS is the clear Dominator
  • Google Cloud
  • Microsoft Azure
  • Open Stack
  • Cloud Foundry etc.

In my experience, AWS seems to be a popular choice. This trend would probably continue for the coming years as they continue to improve their existing services and also introduce new services(for example, AWS IoT). At this point in the market, enough people in the industry already know about AWS – their service offerings and their API’s etc.

Google, with the GCE project, is an interesting player in the market. Google’s Services are being delivered SaaS way from a long time. One of the glaring difference when it comes to GCE is the fact that the cloud traffic flows through Google’s private distributed internet,unlike AWS whose traffic flows in the public internet. Google doesn’t have a elaborate list of services like AWS, but, they are very competitive in pricing and promise better performance. For examples, Google does Load Balancing/ Auto scaling right off the shelf, you would not need to create a ELB, worry about pre-warming it, Auto-scaling rules etc. Some cool things.

While many new Start-ups, Small/ Mid-size companies are embracing the Public Cloud, there are many traditional Big Enterprise companies that have their own data-centers etc that are embracing Public Clouds for their new software projects, they are also taking a Hybrid Approach where-in their additional Infra resides in the cloud and is seamlessly integrated with their existing Infra behind their Data centers.

OpenStack and Cloud Foundry seem to be the clear fore-runners in the Hybrid Cloud Space. Their Services not just help Enterprises transform their existing Infra into a Private Cloud, but, also help them supplement additional Infra in the Public Clouds.

While AWS is currently supporting Hybrid Customer Implementations in different ways, for example –

  • AWS Storage Gateway
  • VPC Peering, allowing setup of Customer Gateways and VPG(Virtual Private Gateways) for connecting your data-center resources to that on AWS cloud.
  • Setting up a bastion host on-premise with technologies like OpenVPN, OpenSwan etc
  • Active Directory Domain Controllers running in the cloud that become part of the AD on-premise.
  • AWS Directory Services etc.

One glaring lack is a unified solution with greater and deeper integration to all the private resources inside the data center, for example, OpenStack services (Nova) can be used for provisioning and management of servers using Foreman.

Foreman’s core is tightly integrated with Puppet, while providing a unified dashboard for all the compute resources available inside the data center, Foreman relies on Puppet for their configuration/ monitoring. Foreman also uses PackStack for configuration of nodes.

Not to mention that AWS is also following suit and taking steps in this direction, the recent announcement of its deeper vcenter integration is an example.

Open Stack Platform at High Level consists of:

  1. RHEV (Red Hat Enterprise Virtualization) – Important projects include KVM, libvirt etc. Many small companies with small private data centers who want to do away with VMWare licenses still adopt and use these technologies.

The trade-off here would be the fact that there are High IT Hardware Budgets, More     Effort and Time goes into Capacity Planning, Infra/ Inventory Management and           Provisioning/ Availabity of new Infra takes time.

2. Satellite for Software Lifecycle Management

3. Red Hat Open Stack Platform

Of course, there are other important and popular projects like JBoss/Wildfly etc.

A little while ago, I have a HBR article that discussed the concept of Holistic Learning, i.e, Learning/Remembering new things by associating things them to things you already know. I thought this blog post could serve as interesting place to anyone who wants to know about Open Stack Platform if they already know AWS Platform like me.

Red Hat OpenStack Platform Services as they relate to some other service offered by AWS:

Horizon -> AWS Console

Nova -> EC2

Nuetron -> Collective equivalent of VPC, Route 53, Security Groups, ACL’s etc that AWS supports.

Elastic IP’s/ Private IP’s -> Public/Floating IP’s

Nuetron Load Balancer as Services (Not so popular among customers yet. .. ) -> ELB (Elastic Load Balancer)

Cloud Forms -> Cloud Formation, Ops Works etc

Cielometer -> Cloud Watch

Heat -> Auto Scaling

Key Stone -> AWS IAM

Swift -> S3

Cinder (LVM based Storage Service) -> EBS

Glance (Images are considered to be nothing but bootable partitions) -> AMI

Open Stack Projects currently being Incubated:

Sahara -> EMR (Elastic Map Reduce)

Trove -> RDS (Relational DB Service)

Marconi -> SQA (Simple Queue Service)

Designate -> Route 53 (DNS Service)

Barbican -> AWS Key Management Service

Ironic -> Not Applicable to AWS as AWS doesn’t have Services that allow bare-metal provisioning. The closest you can get is requesting for tenancy at time of provisioning and Placement Groups

More details on OpenStack Platform and the Architecture of its components here.

Block Developers from creating branches with wrong/funky names in Gerrit

Gerrit does support Regex matching, hence, we can setup control on the type of Branch names the Developers choose to create branch names. For Example, if we are to allow only Branch names that start with dev_ :

^refs/heads/dev_*  regular expression match

Applies to all references that match the expression.

Gerrit relies on the ^ prefix to activate regex matching.

The ^ prefix is also part of the pattern, and therefore always anchors to the start.

Few more things to keep in mind regarding Gerrit Permissions:

For refs/heads/QA/master Gerrit tries rules in the following order, until it has a decision:

1) refs/heads/QA/master  exact match, wins

2) refs/heads/QA/*  glob wildcard, longest common prefix

3) refs/heads/*  glob wildcard, common prefix

4) refs/*  glob wildcard, common prefix

For regular expressions “most specific” is more complicated to evaluate.

^refs/heads/QA/.*

^refs/heads/QA/stable-[0-9.]+

The 2nd rule is more specific, as it matches fewer possible strings.

Most Specific First

For example in Global Configuration, if we have:

perm

Result for refs/heads/master

-2..+2 Administrators

+0..+2 CI Server  refs/heads/* rule also matches!

Result for refs/heads/next

-1..+1 Administrators

+0..+2 CI Server

Handling Huge Git Repositories, Cleaning up Meta-data in Git etc

Handling Huge Git Repositories:

Common Workflows where All Developers Push to a Central Git Repository that resides on a server with modest specs( for eg 2-4 GB RAM, with 80% of space on the repositories mount used up etc ). If for some reason, the Repositories Size bloats to a few gigs .. the git operations like time to clone, push etc takes a hit. The Easier solution is to increase RAM and keep the disc’s usge to about 40% for Optimum Performance.

We can fine tune a few things on the Server in cases where the git clone command either hangs or takes really long time to complete etc …

The central problem is that Git doesn’t know how much memory it can
allocate on the machine, and so if there are really big files in the
repository, it assumes that it can handle them entirely in memory.
The solution is to set various configuraiton parameters so that Git
doesn’t try to handle very large chunks of data in memory.

In .git/config file (client and/or server) I added the following:

[core]
packedGitLimit = 128m
packedGitWindowSize = 128m
windowMemory = 128m
[pack]
deltaCacheSize = 128m
packSizeLimit = 128m

Just find out more info on settings related to pack etc.

Cleaning up Meta-data in Git:

Sometimes, People by mistake or lacking knowledge of how Bad Git is with Binary files, check-in Huge Zip or Tar files etc. This causes the size of Git Repo to increase and the size of Repo wont decrease even if the files were removed from the Repo in the subsequent Commits, the Huge Files still reside inside Git as Meta-data. I discuss a few commands and couple of tools to clean up this meta-data:

git fsck —full (also —unreachable or —no-reflogs ) will show you dangling commits

try to know which branches contain those commits:
git branch –contains 793db7f272ba4bbdd1e32f14410a52a412667042

to know the size of files that are committed part of git commit:

git ls-tree -lr e3ee14d5f4aed1977ef3c4fb81056c9bd0c1dd3e | cut -c54- | grep -v ‘^ ‘ | sort -u

To remove all dangling commits and those reachable from the reflogs do this:

git reflog expire –expire-unreachable=now –all
git gc –prune=now
Unreachable objects that are younger than two weeks are not removed so we use –prune=now which means “remove unreachable objects that were created before now”.

We can delete the Huge files in Git Meta data using 2 ways : Git filter branch or bfg tool.

Git filter branch method:

$ git count-objects -v
$ git verify-pack -v .git/objects/pack/pack-976ab381417423059392182c374dd3813becdb4a.pack | sort -k 3 -n | tail -5
966ec9f788f862e0820c2132378128383768724b blob 833 453 27788
aea647e1263f4a710246d088ff6de822d99710be blob 976 435 30843
7835428d81cf42578560c709817b285e006075b5 tree 1237 1031 17267
255480a8b635da33e973a63e5081f696c4eeb1b3 blob 34622 33244 71675
598c38faf8d7317e0570c489003c5565cf61d4bc blob 38324 37390 34285

$ git rev-list –objects –all | grep 598c38faf8d7317e0570c489003c5565cf61d4bc
598c38faf8d7317e0570c489003c5565cf61d4bc screenshot.png
git filter-branch –index-filter ‘git rm –cached –ignore-unmatch screenshot.png’ — —all
Repack everything:
$ git pack-refs –all –prune
$ git reflog expire –expire-unreachable=now –all
$ git repack -A -d
Garbage collect:
$ git gc –aggressive –prune=now
Recount:
$ git count-objects -v
Using BFG Tool:
Copy the bfg.jar fie from here to a folder above the git repo location
http://rtyley.github.io/bfg-repo-cleaner
java -jar bfg-1.11.8.jar –strip-blobs-bigger-than 50M <Git Repo Name>

Note: We run the above commands in a cloned copy of Repo, clean up the meta-data and push it back to central server.

Since we are running this on a cloned copy and not the central repo, we need to make sure other Developers clone their workspaces freshly, so, someone would not Push a old copy with un-cleaned meta-data yet again.

As a Best Practice, we need to advocate using properly updated .gitattribute or .gitignore files.

Code Merging Best Practices

These days, Merging is Cheap, Thanks to Awesome DVCS like Git, Perforce etc. But, This Activity needs to be done Carefully and under Collective Collaboration with an eye for Future Issues. Else, we end up losing valuable and increase Stress during your Release Time or Run into Issues that take longer time to resolve after your Code is in Production. Either Way, It lowers Confidence in Team and creates Doubts about Processes and Tools Involved, But, No one knows How to Fix the System.

As an Engineer who used to create a lot of Service Patch Binaries, by cherry-picking a range of commits for a old Release Branch/ and sometimes Multiple old Release Branches, onto a new service branch, compile/package the code and provide the customer who requested for it, Here are a Few recommendations that Dev/ Dev Leads can keep in their mind when they merging their Release Branch UP to the master:

First Challenge: Renamed classes

– Description: Classes were renamed to better identify them, or to fit into a pattern. [Examples: SubscriberStatus –>
SubscriberStatusConstants; Feature –> FeatureVO]
– Impact: Renamed classes caused many other classes to be changed due to the import statements needing to change. This caused many classes to be modified in the Release branch, which meant that if the class had changed at all in master, the class was highlighted as having a conflict between master and Release Branch. This, in turn, meant that many classes had to be manually looked at in order to resolve the conflicts. Furthermore, any new code written in master that referenced the old class name now had to be “fixed” when it was merged up to Release Branch.

– Suggestion: Do not rename classes in a branch. Rename them in master, and merge up to the branches. Since the class is simply being renamed, the impact to master would be minimal. This is an example of a low-risk refactoring that has a high code-footprint.

Second Challenge: Removing code-behind

– Description: Some classes were deleted, and the code inside them is put into another file, say some mxml file.
– Impact: When merging from master to Release Branch, git merge would see no changes in the mxml in master, and happily accept the mxml from Release Branch. For the Deleted file, git merge would see that it was deleted in Release Branch, and therefore any changes in master would be considered irrelevant. This means that the merge tool happily reported no conflicts. This meant that you would have to take the modified deleted file, re-paste it into the mxml in release branch, and compare the code — often accounting for renamed classes (as in the first challenge).
– Suggestion: Remove code-behind in master, not in the branches. Since the files are being combined, we’d rather they were combined into a file that will result in conflicts if there are changes between master and a later branch. After combining in master, merge up to the branch.

Third Challenge: Copy/Pasted classes
– Description: Code from a Command class or other “old architecture” class is copied into a new class that fits the new architecture.
[Examples: UsageSummaryCommand -> UsageSummaryResponseHandler; AddRemoveWindowShadeRenderer –> WindowShadeRenderer;
SSSDAccountModel -> SSSDAccountPM]
– Impact: There are two distinct situations here:
– The “old architecture” class still exists: When you get a conflict, you happily merge the code into the “old architecture” class. Unfortunately, If you are not aware that this code has been copied into another class. It is quite possible we could entirely miss these, and this poses not just a risk to “merging,” but a risk to already-delivered functionality.

– The “old architecture” class was deleted: There are no conflicts when merging, and it just looks like the file has been deleted. You generally search for the re-factored class (usually from code segments inside the file). However, if a SECOND merge is done, the deleted file will already have been considered merged, and there is no way to see that you need to update another class.
– Suggestion: Still do this re-factor in the release branch, as these types of impacts would be detrimental in master. In the old class in master, add a comment at the very top of the file that says “to be phased out, use XXX class in xxx release branch ” or something along those lines.

Fourth Challenge: Replacing renamed classes
– Description: I only had one occurrence of this. A class was renamed in the release branch (Feature -> FeatureVO) and then a NEW Feature class was created in the same package as the old one.
– Impact: This causes confusion, because the imports to the old Feature class were not failing, but the contents of the class were not
equivalent.
– Suggestion: Don’t create a new class with the same name as an old, renamed class.

Fifth Challenge: Splitting projects

– Description: CoreAPI was split into CoreAPI and CommonLib.
– Impact: An git merge from master to release branch was not sufficient. You have to use a external tool (BeyondCompare for eg) to compare CoreAPI with CommonLib.
– Suggestion: Although painful to deal with, I believe that this was the correct way to do this in release branch. Splitting the project in the master would have been too disruptful to the “next release.”

Sixth Challenge: Code Quality Tools Related Rules
– Description: A CQ Tool, For Example, FlexPMD enforces this specific rule — for example, “No throwing hard-coded Strings in an Event — you must use a constant.” The rules are less stringent for master and early branches, and get more strict for later branches.
– Impact: Merging code was not the “last step.” After merging, code had to be altered in order to meet the more strict FlexPMD rules. Hence, my “merged code” contains edits done by me that were necessary to get the code building. There were less of these than I expected to encounter but it will still be troublesome to deal with.
– Suggestion: Instead of always introducing FlexPMD rules in the last branch, let each rule be individually placed. For example, if a strict rule is introduced saying “no unused private methods,” it might be better to put it in the master, and merge it up. However, if a rule enforcing the error-handling framework is put in, and that framework doesn’t exist until the Release branch, then obviously, the rule must go in Release Branch and later.

Seventh Challenge: Moving packages
– Description: A class is moved into a different package, and the old package is deleted.
– Impact: This *can* result in a tree conflict, in general, but *definitely* will result in a conflict if the new package is independently created in both the master and the release branch. Tree conflicts require me to manually merge the files. Sometimes this results in me seeing a deleted file in one location, or an added file in another without them being in conflict.
– Suggestion: Don’t create the same package, independently, in branch and master — merge it up. Otherwise, there isn’t much we can do about this one.

Eighth Challenge: Disjoint code
– Description: Code changed in master in a method or file that no longer exists in release branch, and there is no identical method in another file, nor another file with the same basic code.
– Impact: Changes can’t be merged — there is no where to put them.
– Suggestion: Not much we can do here… Sometimes the changes might be irrelevant…

Ninth Challenge: Duplicate fixes
– Description: Someone fixes a Defect in master, and someone else fixes the same defect in release branch, but they do it a different way.
– Impact: The impact is worse if there were multiple files touched in the fix. This requires the merger to attempt to understand the two different fixes, evaluate them for correctness, simplicity, performance, readability (or any other “superior” metric) and choose one. Often times, you can’t take half of one solution and half of the other, and you don’t want to take both, either, since they may not interact well together, or reduce performance.
– Suggestion: A defect should only be fixed in the “closest to release” master/branch where it needs to go, and then merged up to higher releases.

Tenth Challenge: Lack of Merging
– Description: In the worst case, virtually no one merges code until release time and its already too late with too little time.
– Impact: All conflicts become worse and worse to deal with over time.
– Suggestion: Merging needs to happen more frequently.

Eleventh Challenge: Split Classes
– Description: Parts of a class were moved into several different classes, and the class, itself, was fired upon with phasers set to kill.
– Impact: It was difficult to trace down where merged code needed to be placed.
– Suggestion: Add comments to the old class in master to explain where the code can be found in the later branches.

Twelfth Challenge: Large files
– Description: Files that are a little bit on the large side.
– Impact: Comparing them and making changes – in Eclipse – takes a ridiculously long time.

There will be Multiple Features/Bugs that modify these Huge Classes in slightly different ways, sometimes, it’s a challenge when we try to cherry-pick Multiple Features/Bugs and both modify same Lines in 2 different ways.

– Suggestion: Break it up a little.

Thirteenth Challenge: Tree Conflicted Directories
– Description: Directories are moved, or are created independently between branches
– Impact: The comparison must be performed manually.
– Suggestion: Take note of the directory and do a follow up merge. The conflicts should be handled using a comparison tool such as
BeyondCompare or Eclipse’s built in tool and commit with the merge.

 

Having Said and done all this, Code never ceases to throw Bananas at us 😉 It can become even more Interesting if you have to support multiple customers on multiple master(s) or released products.

Suggestion: Suggest your customers to migrate/upgrade to latest/later Releases.

If not, You can maintain Customer Specific Baselines that Co-Exist with your Release Branches. It is on the Onus of Dev Team to write Custom Code Re-using Code from Earlier Feautures/Bugs.

 

How to perform a branch landing in git

A branch landing is the process of cherry-picking a range of commits from a source branch to a destination branch.  This may be required to merge changes from a development branch onto a production branch, or to merge changes from one production branch to another.  Here are the steps required.

  1. Check out the source branch
    git checkout <Branch1>
  2. Determine the starting and ending commit SHA that you would like to land
    git log –pretty=oneline
  3. Check out the destination branch
    git checkout <Branch2>
  4. Cherry pick the range of commits
    git cherry-pick <StartSHA>..<EndSHA>
  5. Resolve any merge conflicts and commit the change
    View unmerged files:  git ls-files -u
  6. Resume the cherry pick process
    git cherry-pick –continue

Linux Screen Program – An Interesting Application

I came across this program recently and used it to solve an interesting problem, thought of sharing it just as an Idea

Linux screen, needs a yum command to install, open-source

We have a particular jenkins build that needs user inputs that cant be provided as parameters at the beginning, and we have no control over the processes that jenkins will spawn to perform the build, If jenkins created a screen with the command:  screen -m -S aa -t aa -d before it starts the build, we can then later login the server

[root@server ~]# screen -ls
There are screens on:
        29076.pts-10.server       (Detached)
        28433.pts-5.server        (Attached)
        28544.aa        (Detached)
        28241.pts-1.server        (Attached)
4 Sockets in /var/run/screen/S-root.
we can login to the screen where our jenkins build is running to provide the inputs for jenkins to continue the build
 screen -r 28544.aa
Looks like, it s a pretty popular tool that supports many other interesting features like remote collaboration with screen sharing/ sessions etc, the following links are useful to get me started: