Thomas' site on Thomas' site

2019 Blog Update

Tue, 01 Jan 2019 00:00:00 +0000

Once in a while, this site gets some TLC. Spring cleaning came early this year.

Since I registered this domain I used:

“home made” html pages
templeet (the engine which was powering linuxfr.org)
tiddlywiki (the all in one page javascript wiki)
pelican another static site generator in Python using restructured text

This year, I decided to spent a few hours to convert this site to Hugo. Hugo has better support and template than pelican, also it uses markdown which is more widely used.

This time, I removed some outdated posts about networking, software, out-of-date patches, etc. that nobody uses anymore.

Perhaps next time I will decide to simply use the “cloud” for my occasional blogging ..

BGP weirdness

Fri, 13 Oct 2017 00:00:00 +0100

Having implemented tens of BGP RFC, I am sharing some of the ‘pecularities’ that BGP accumulated over time.

Using DNS Anycasting to help CDN

Mon, 20 Feb 2017 00:00:00 +0000

CDNs (Akamai, Netflix, Facebook, Google, Apple, Micrsoft, Amazon, …) do not rely on BGP for routing decisions. In this talk, I will explain how DNS can be used to help them find the “optimal” path to your end users using DNS Anyacst.

HTTP RFC vs real world

Tue, 07 May 2013 00:00:00 +0100

After quite some work, exaproxy is now serving our producting traffic nicely, however it does not mean that once in a while we do not get some surprises !

I found the following request in our logs recently:

GET /a-page.php?many=argument&passed=here\r\n
HTTP/1.0\r\n
Host: changed.to.protect.the.innocent.com\r\n
Connection: close\r\n
\r\n

Yes, you read properly, there is a new line between the URI and the procotol version, here HTTP/1.0, when RFC2616 (the document which explains how a web server and your browser should speak) clearly does not allow it.

So what was my surprise to find out that apache is replying correctly to this request if no virtual hosts are configured (without HTTP headers) !

»› telnet 127.0.0.1 80
Trying 127.0.0.1...
Won't send login name and/or authentication information.
Connected to localhost.
Escape character is '^]'.
GET /
HTTP/1.0
Host: changed.to.protect.the.innocent.com
Connection: close

<html>
some more html
</html>

Connection closed by foreign host.

compared to ..

»› telnet 127.0.0.1 80
Trying 127.0.0.1...
Won't send login name and/or authentication information.
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0
Host: changed.to.protect.the.innocent.com
Connection: close

HTTP/1.1 200 OK
Date: Tue, 07 May 2013 21:59:15 GMT
Server: Apache/2.2.22 (Unix) DAV/2 PHP/5.3.15 with Suhosin-Patch mod_ssl/2.2.22 OpenSSL/0.9.8r
X-Powered-By: PHP/5.3.15
Connection: close
Content-Type: text/html


<html>
some more html
</html>

Connection closed by foreign host.

The internet is full of surprises :)

Now time to fix our request parsing code to allow this broken client to call back home !

ASN or Community ?

Fri, 16 Nov 2012 00:00:00 +0000

Someone found a way to cause ExaBGP to mis-behave back in September (2012-09-05) .. The bug was “funny” enough to justify a blog entry, so let’s look at the route which caused it.

If you are in an hurry, the answer to the mystery was that a route contained an ASN which could not fit in a Python integer. The patch to fix the issue was a one liner to change the base class of an ASN from an integer to a long …

Integer size depends on your architecture, on my Macbook Pro it is 64 bits (63 bits signed integer), so the bug did not exist on my development platform ! On the machine affected, however the size of an integer was 32 bit, so the biggest integer possible was 2147483647

>>> sys.maxint
9223372036854775807

>>> pow(2,63)-1
9223372036854775807L

>>> pow(2,31)-1
2147483647

Still, no internet registry has yet allocated any ASN with a value near 2147483647, so how can that ASN be found “in the wild” ?

The exact route as parsed by ExaBGP was :

route ipv4 unicast 178.216.216.0/21 
  next-hop 193.5.69.5 
  origin igp 
  as-path [ 196621 8758 3356 20485 21127 28884 41771 2737504257 2737504258 41771 ]
  community [ 3356:2 3356:22 3356:100 3356:123 3356:503 3356:2067 20485:11754 8758:13030 ]

So the route was leaving the autonomous system of 41771 and going back to it, this would indicate a routing loop and can not really happen unless the route was “tweaked”. However looking back at the RFC, this is however still a valid route.

But how can 2737504257 and 2737504258 be in the AS-PATH ! ? ! Clearly a filter is missing, or something went very wrong somewhere !

Looking at the ASN, you can notice something suspicious :

2737504257 is (41771<<16) + 1 known as 41771:1
2737504258 is (41771<<16) + 2 known as 41771:2

I can only conclude the originator must have been using something even more bizarre than ExaBGP and inserted a community in the AS-PATH attribute by error !

So, let’s have a look at who is that network :

aut-num:        AS41771
as-name:        MKC-OMSK-AS
descr:          MultiCable Networks LLC

Ha, ha, we are surely right, some crazy russians :D some crazy polite russians to be exact as they replied to a mail I sent and have fixed the issue.

Thank you to Marek for noticing that the ASN looked like a community

Can you not see we are competing ?

Wed, 24 Oct 2012 00:00:00 +0100

Why no legislation from the “Creative” Industry should be passed - ever. This “rant” has been on my Desktop folder for over over two years waiting for my site to change, it finally found its way here …

Who are the ‘Cretive Industries’ ?

If one look at the relation between ISPs and the ‘Music Industry’ or more precisely the Music DISTRIBUTION industry, which control the music production, the reason for the tension should be obvious.

Both Industries are in the business of DISTRIBUTION. The ISPs, unlike the ‘Music Industry’ are not content producer just a cheap conduits for anyone to communicate any kind of information.

With the now wide deployment of broadband and high speed internet in homes, it means the anyone can now easily distribute any information, music included, at low cost, to a large audience. So in theory, we should see new channel for music appearing and artists embracing the internet, more competition and therefore lower prices.

However, comparing the price of a movie, music included, with the price of a album still makes me think the price practiced are still very high and that someone is getting a golden deal, and that it is not the consumer.

Is it all about monopoly ?

The only business which succeeded at it is Apple but the “low” price they practice have been, and I am sure still are, a point of tension between them and the label. Is apple becoming the new monopoly ? One can wonder if the settlement of Apple Computer/Inc. with the Beetles’ Apple Corps is an indication that Apple has long term ambitions extending behind the creation of technological products.

The ‘Music Industry’ have been used for years to hold a quasi-monopoly on distribution, and like with every monopoly the price of music has been high. This monopoly is the reason with this industry is slow to move toward new distribution mechanism. The most popular format on the internet are DRM free and customer are not interested in buying the same music over and over (once in tape format, once in CD form, etc.). I am surprised to not have to option to buy DVD with video clips at a premium yet.

The problem is that the lack of legal download solution have created a generation of people who had no other option to listen their music in digital form than to break copyright law. A problem that the ‘Music Industry’ blame on the ISPs, putting enormous pressure and lobbying for ISPs to ‘take action’ against their customers performing such copyright infringement (which is not copyright thief like they so often like to say).

Resisting changes

The ‘Music Industry’ is resisting changes, and seems to not have yet understood Apple idea of how to sell on itunes: make is so cheap to help impulse purchase. I hope that the success of the iPhone application store will open some eyes and that within my lifetime I will see the ‘Music Industry’ working in providing easy ways to purchase music instead of trying to sue music lovers.

The ‘Music Industry’ want to force regulations (through strong handed lobbying) on ISPs to fix their business model, is ill advised and unproductive and may damage irrevocably the internet infrastructure and end-to-end principle it is based upon. They now want the ISP to censure the internet and block websites.

Nothing about this resistance is new, tapes, VHS. Every new technology changing the distribution game has seen lobbying and legal attacks from the distribution.

Tomorrow

Hopefully, like radio, the internet will become a successful distribution medium for the ‘Music Industry’ to use. However, even radios are under pressure even after helping labels to reach the success they enjoy now-days.

But, at the end, Listeners want choice. The internet is about bringing the world diversity to one’s living room, from new unknown indie album to collectors’ vintage.

Some business like Spotify and Last.fm, and more recently Microsoft, have understood it and seems to have done some progress. This struggle between the media creation industry and the technology distribution providers is not new, let hope that time once more the consumer interest will win over the corporate will to restrict media distribution.

Conclusion

Thank you for reading, time for me to see a © Mickey movie with my son.

VOIP QOS

Thu, 11 Oct 2012 00:00:00 +0100

I was invited to present at this year Leeds ITSPA’s meeting with my IXLeeds’s hat on.

For once I did not speak about BGP/FlowSpec but VoIP QOS and how Internet Exchange can help to achieve end-to-end call quality.

The meeting was interesting and the drink at the Adelphi gave me an opportunity to meet people I normal do not cross. I had to miss the evening curry but as I am living only 20 mns from Leeds, I am sure I will have another opportunity to try the restaurant.

BGP for High Availability

Thu, 24 Mar 2011 00:00:00 +0000

I presented at UKUUG in order to explain how BGP can be used to achieve High-Availibility.

ExaBGP

Thu, 03 Sep 2009 00:00:00 +0100

ExaBGP is an application designed to provide an easy way for programmers and system admistrators to interact with BGP networks. The program allows the injection of arbitrary routes into a network, including IPv6 and FlowSpec, and the relaying of received routes to business logic backend applications.

Introduction

Many security professionals are currently using NetFlow to monitor their networks and react to DDOS attacks. By centralising their traffic information, they are able to corrolate the information and detect more and more advanced attacks.

BGP is then often used to blackhole the destination IP of the attack at the network edge, protecting the core but still allowing the attacker to succeed.

RFC 5575, better known as FlowSpec, was designed to help security professionals react to such attacks in a more fine grained manner by deploying precise filtering rules, and by taking advantage of recent routers advanced ASICS/Traffic filtering features.

ExaBGP can be used for conditional announcements, for example only anycasting a service IP when the service is established as running correctly.

Getting Started

ExaBGP is available on github and will run on most Unix flavours with any recent version of python 2 (2.4 to 3.7).

Simply use pip install it, or untar the code into any directory of your choice, then edit one of the template files to reflect what you want to do, and finally test your configuration calling “bin/bgpd” with your configuration as the first parameter.

What does it look like …

The configuration contains the usual BGP information fields; Cisco, Juniper, Quagga and BIRD users should find their mark very quickly.

The program wiki has more information on how to install and configure the application.

As usual, comments and feedback are welcome. Feel free to use the site’s bug tracker to contact us.

The real cost of P2P

Mon, 30 Jun 2008 00:00:00 +0100

Should you read slashdot, you must have already seen its readers complaining about their ISP traffic shaping policies. When working in the ISP industry it is painful to see the lack of understanding those ‘techies’ are displaying.

In the UK, if anything ISPs are guilty of bad advertising misleading customers with ‘up to’ speeds and obscure fair usage policies and trying to market their product on price instead of quality (but Internet is a commodity market nowdays, so it is to be expected)

Customers should be clearly told that DSL product sold are contended. Previously dialup products were as well, but the impact with dialup was much more noticeable with the inability to get online.

The recent increase in content (video even more than P2P) has recently caused many of them to realise that they had oversubscribed their infrastructure to the point they could not deliver to their customers what they came to expect.

Once down to the wall, ISPs had only a few options :

raise price to reflect the cost of running the service at a low contention (and we all know that it is impossible)
apply traffic policing globally (everyone is slowed down the same way to modem speed).
apply targeted traffic policy (P2P users here you are)

As it is hard to tell a customer, who may cancel its contract returning a then useless free router, that he can no longer have fast email and web surfing, the path of least resistance is to throttle P2P traffic which is an important part (but not all) of an ISP traffic, freeing capacity for other services and allowing to delay infrastructure upgrade. (The cost of implementing traffic shaping is recovered if it allows to delay a network upgrade if only for a month!)

For information, an ISP for a DSL service can be simplified as:

the ‘last mile’ cost from the home to the exchange
the cost of the space used, power consumed and hardware located at the exchange
the cost of moving the traffic within the country (fiber, etc.)
the cost of the space used, power consumed and hardware located at national point of presence
the cost of moving the traffic to other ISPs
the cost of supporting the customer (ie: taking unrelated calls about their virus or other issues)
the cost of collection the client payment
all other generic business

For quite few small/medium ISPs, the transit cost (the cost an ISP will pay for another bigger ISP to take its traffic somewhere worldwide) is more than the income that the customer provides. Most ISPs are making a loss trying to become big enough to be acquired.

P2P being notably known to not really care about locality, one can see why it is the target of shaping (with the fact that the biggest torrent are often providing copyrighted material for which end users may or may not have a license to see/use).

In that context it is not surprising that the industry is facing issues and trying to find more income streams (see my rant on Phorm).

Phorm, the logical conclusion to legal pressure on ISP

Fri, 29 Feb 2008 00:00:00 +0000

Up to recently, ISP felt that they had the same status as traditional telecommunication provider and were protected from prosecution for the traffic going through their network. It was then none of their business to police the information flowing through their network.

The situation became hazier when BT decided to deploy [cleanfeed](http://en.wikipedia.org/wiki/Cleanfeed_(content_blocking_system). Up to that point ISP had been transproxying web traffic in order to cache the web page requested and save on bandwidth cost but had never actively interfered with the data passing through their network.

More recently threat of legislation pushed by the IFPI, children protection lobby, and government (all ignoring that transproxying can be easily evaded) seems to be changing the landscape for ISP, which are now under increased pressure to police their traffic for the benefit of who can afford to lobby them.

Deploying large scale filtering/transproxying solution is expensive, and with little chance of seeing the cost paid the either the end user or the legislator, It is only natural for ISP to seek some kind of form or remuneration of the cost of deploying such possibly soon legally required solutions.

In that context it is not that strange to see the UK largest ISP sell their customer web traffic (not protect by any data protection law) to an organisation selling targeted advertising.

Up to now, advertiser had to rely on cookies to track surfing habit, making it possible for customers to protect their privacy (refusing them or using anomymisers).

With this new system (described here) our average UK broadband users can only hope that the ISP marketing firm will honour its promise to not monitor their traffic.

The most interesting part seems to be that even once ‘unsubscribed’ the traffic may still go through the advertiser ‘anomyser proxies’. One can only wonder if those proxies role will not block cookies from competitors giving Phorm a quasi monopoly for advertising in the UK.

Posts

Fri, 16 Nov 2007 00:00:00 +0000

Ancien patch for Postfix checking multiple recipients for Postfix’s Access Policy Delegation

WARNING

Use the patch provided here at your own risk : do not use if you are not able to understand the code provided

Before using this patch, you may want to read this thread on the postfix-user mailing list where I was told: * that I am ill advised to want such a patch in postfix as its “//approach is fundamentally flawed//” * that this patch is too resource intensive

In order to address the last point, I made sure that : * the feature is turned off by default * the maximum among of memory available to the feature can be set.

With the default values :

smtpd_client_connection_count_limit (default: 50)
smtpd_recipient_limit (default: 1000)
line_length_limit (default: 2048)

The worse case memory utilisation for the feature is around 2Mb per smtpd instance which is 40Mb with the default settings - which are exceptionally large for the recipient limit. Limiting mails to 50 recipients makes the worse case overhead per smtpd 100kb.

40 Mb is indeed a lot for an old machine but on recent hardware it will not even be noticed (and this memory will only be allocated if the mails received have lots of recipients).

The other way to get all the recipients of a mail would be to track the “recipient” sent to the policy server at each RCPT using the “instance” attribute and use the result at the DATA state.

With this approach the policy server need : * to be called at each RCPT (and not only at DATA) * keep track of the recipients for each mail * to perform some cleaning should the connection close between the RCPT and DATA state

The patch provides two new configuration options: * a boolean : access_delegation_recipients, which need to be turned on to use the feature * an integer : smtpd_recipients_length_limit, which limit the among of memory the list of recipients can take, it is set to zero by default meaning that no limitation will be performed. Should its value be under “line_length_limit”, the value will be changed at run time to this default.

It changes the SMTPD POLICY Protocol adding a line starting with “recipients=”. The key contains a “\r” separated list of the mail recipients (or the single recipient, exactly as the recipient key). The list is only set during the DATA and END_OF_DATA state and only if the lenght of the string is under the value set in smtpd_recipients_lenght_limit.

This patch/feature is useful for : * boucing spam to a list of forged inexistent email addresses (especially when your MX and storage servers are not on the same machines). * to allow per domain policies, ie per domain white-listing, etc. * you tell me

You can download the “fourth” version (released the 26th of November 2007) of this `patch for postfix 2.6 20071111

I have updated the patch to apply cleanly on a more recent version of the patch for postfix 2.6 20080201 (which applies cleanly on postfix-2.5.1-rc1)

Should you have downloaded any previous version, please update as the third contain a memory leak which cause the memory utilisation to be up to two times what it should and any version before should simply not be used.

BGP firewall

Sat, 15 Sep 2007 00:00:00 +0100

If you are protecting your network from packet with spoofed source IP, it is likely that you have to update your routers ACL each time the route you learn from your customers are changing. This can be automated, but could this be done without having to generated ?

BGP firewall

WARNING

This post is kept for nostalgic reason. Please do not use this solution for anything in production as it is more than likely that it will cause issues with any serious traffic.

Always up-to-date ACL

ACL can be auto-generated from the content of the Registry Database, which is likely to be out of date, but it could also be possible to use the content of the router’s RIB to auto-generate those filters.

Juniper has a feature called SCU/DCU (which from what I can read on their side seems to be mainly used for traffic accounting) which can be (ab)used to create some kind of dynamic prefix-list based the the communities taged on your BGP route.

Configuration

The example below uses SCU to create a firewall blocking packet entering your network with invalid source IPs.

In order to do so we :

tag all routes with communities depending on their source (transit,customer,peer)
create a policy statement based on those communities
apply this policy statement to our RIB to create the classes
use those classes as filtering term of firewalls

[edit policy-options]
policy-statement community-to-class {
  term is-peering { ... }
  term is-transit { ... }
  term is-customer {
  from community [ route-customer originate-customer ];
    then {
      destination-class customer;
      source-class customer;
    }
  }
  term is-orginated-here {
    from community originate;
    then {
      destination-class originate;
      source-class originate;
    }
  }
}

Then we tell the Juniper to build the SCU from our routing table.

[edit routing-options]
forwarding-table {
  export [ community-to-class load-balancing ];
  unicast-reverse-path feasible-paths;
}

We then create a firewall saying that we intended to use this SCU as a match close.

[edit firewall]
filter external-incoming-transit {
  ...
  term originate-deny {
    from { 
      source-class originate;
    }
    then { 
      count deny-spoof-originate;
      discard;
    } 
  }
}

And finally apply it to the interface ..

[edit interface .... }
unit 123 {
  description "a peer/transit interface";
  vlan-id 123;
  family inet {
    rpf-check {
      mode loose;
    }
    no-redirects;
    filter {
      input external-incoming-transit;
    }
    address 1.2.3.4/30;
  }
}

The route will have been tagged with an import statement on your bgp peers or sourced within your network

[edit policy-options]
community originate members 30740:30740

[edit routing-options]
aggregate {
  route 82.219.0.0/16 {
    community 30740:30740;
    as-path {
      origin igp;
    }
  }
}

Finally do not forget to remove those communities from the routes you are receiving from ebgp.

BGP route leak

Thu, 13 Sep 2007 00:00:00 +0100

Leaking BGP routes is a common sport among the ISP community. I done a presentation on my personal experience at LINX 57 over ten years ago but it is still valid today.

Background

If you are joining an exchange you should assume that other member will leak, and be prepared. Please consider those method as non-exclusive, the more you filter the less likely you are to leak.

Things no-one should announce or accept

Small Prefixes

Many ISP carry their customer routes (DSL, etc.) in iBGP as the IGP should remain stable and small to converge quickly.

Should an ISP leask those route, you could see thousands of /32-/2x routes, as the smaller prefix routable over the internet is a /24, make sure to not accept very small prefixes

/* match and refuse any route smaller/longer than a /24 */

[edit policy-options]
policy-statement no-small-prefixes {
    from {
        route-filter 0.0.0.0/0 prefix-length-range /25-/32 reject;
    }
    then reject;
}

BOGONS

As well, make sure you do not accept (or announce) reserved ranges and non-routable ones.

/* bogon, rfc1918, etc. */

[edit policy-options]
policy-statement no-bogons {
    from {
        route-filter 224.0.0.0/4 orlonger reject;
        ......
    }
}

Things obviously wrong ..

Only you can know what you can not learn from your peers but the transfer lan of an IX may look like something you would only learn from a mis-configuration

/* Linx LAN */

[edit policy-options]
policy-statement no-ix {
    from {
        route-filter 195.66.224.0/22 orlonger reject;
    }
    then reject;
}

And then make sure you never see it .. or announce it

/* should never get in or out */

[edit protocols bgp group linx]
export [ no-small-prefixes no-ix no-bogons ];
import [ no-small-prefixes no-ix no-bogons ];

Protect yourself

Max Prefix

The quickest and simplest way to get some form of protection is a max-prefix limit, ie to put an upper bound to the number of routes you will accept from your peers

The router will prefix then will shutdown a session should the ebgp speaker send you more than a predefined number of routes (was it necessary to say it ?)

neighbor 195.66.224.xxx {
    description "AS-ACCEPTED | Peer name | noc@peer.co.uk | AS-SENT";
    family inet {
        unicast {
            prefix-limit {
                maximum 150;
                teardown 80 idle-timeout 5;
            }
        }
    }
    peer-as 1234;
}

Max Prefix Limitations

On cisco this works great as the count is performed on prefix accepted. On juniper not as good the counting is done on prefix received (before any kind of filtering) which is much less useful. For the clueful

Go and thanks RAS for his excelent max-prefix auto-tuning work at http://juniper.cluepon.net/index.php/OS_Auto_Tuning_Prefix_Limits

Please push for this feature to your SE.

Peers are not your transit providers

As an ISP you know who your transit providers are and their ASN. You should filter from your announcement any route with an AS-PATH which contain them

Here is an example for Juniper (assuming your transit is from Level3 and Sprint)

/* define the routes we have learned from transit (example) */
as-path routes-level3 3356.*;
as-path routes-sprint 1239.*;

/* create a policy blocking their distribution */
[edit policy-options]
policy-statement no-transit {
    term remove-path {
        from {
            protocol bgp;
            as-path [ routes-level3 route-sprint ];
        }
        then reject;
    }
}

/* make sure that no linx peer will ever get them again */
[edit protocols bgp group linx]
export [ no-transit ];

Peers are not your customers

You should never see your customers routes from your peers neither. Peers should know better

Your peers should not neither leak routes with reserved ASN, mainly when they can be filtered with one line.

[edit protocol bgp group linx]
  remove-private;

Protect your reputation

Filtering routes using communities

First you must tag your route to know what is what

It is in every book, your tag your route inbound and filter them outbound.

/* define a communtiy to identify routes learned from transit */
community route-transit members 1234:1239;

/* create a policy to apply this community to a route */
policy-statement tag-transit {
    then {
        community add route-transit;
    }               
}

/* make sure all routes from transit have that community */
[edit protocols bgp group transit]
import [ tag-transit tag-transit-provider-specific ];

(repeat with peers)

Then your use this to stop the annoucement to your peers

/* define a policy rejecting routes identified as transit */
[edit policy-options]
policy-statement export-transit {
    term remove-peering {
        from {      
            protocol bgp;
            community route-transit;
        }           
        then reject;
    }               
    term remove-peering ...
    term remove-community ...
    term prepend-one-time ...
}

/* and make sure no linx peer sees it */
[edit protocols bgp group linx]
export [ export-peering export-linx ];

Don’t make a typo with your community definition without filtering on as-path as it hurts.

Filter using AS-PATH

Most large networks have very “private” peering policies and it is unlikely that you should ever learn any of their route via peering (otherwise it would be called transit).

/* define the routes you will never see through peers */
as-path leaked-sprint ".{1,}1239.*";
as-path leaked-telia ".{1,}1299.*";

/* create a policy blocking their distribution */
[edit policy-options]
policy-statement no-leak {
    term remove-path {
        from {
            protocol bgp;
            as-path [ leaked-telia leaked-sprint ];
        }
        then reject;
    }
}

/* make sure that no linx peer will ever get them again */
[edit protocols bgp group linx]
import [ no-leak ];

Filtering using the registry DB

Some tools, such as irrpt_, exist to help with the generation of filter based on the content of the IRR DB (RIPE, ARIN, etc.) gather and track prefix within AS-Macro.

DNS Good Practice

Wed, 08 Mar 2006 00:00:00 +0000

I wrote this post ages ago, and therefore the information will sound a bit outdated, even if still accurate.

Prelude

DNS is the service on which Internet is based, however, quite strangely it is often overlooked.

In order to provide the best possible reliability, a lot of energy is placed on the hosting. All major e-commerce sites are load-balanced with redundant database back-ends, etc.

Without a resilient and reliable DNS server, no one can hope to smoothly run any Internet services. However, lots of highly redundant web servers are based on weak DNS foundations.

DNS is often misunderstood and it is assumed to be resistant to failure “by design”. Those who think like this will probably suffer a DNS outage sooner or later, however this could have easily been avoided by just taking a little care.

DNS resilience should be the second concern after routing resilience. According to experience (at least mine) lots of ISP and big accounts do not have reliable DNS Getting more Information

This document is not intended to explain DNS basics but to provide good practical advise. If you want to learn more about DNS and understand which DNS is good for you please consult the very good DJBDNS FAQ. located at http://cr.yp.to/djbdns/faq.html

You will probably find theses pages very interesting as well:

Please, do not contact me to fix your DNS. Even so I am litterate with both DJBDNS and BIND, I do not wish to spend my time supporting it. Please refer to your ISP support department, read the FAQs, read newsgroups, Most BIND questions have already been answered numerous times.

However, please free to report any fault or inexactitude about this document

Good DNS record

Having a reliable and resilient DNS server is only the first step to secure DNS informtation. Hosting valid and well formed DNS information is crutial as well.

Lots of good books such as “DNS and BIND” will provide your with all the information you need to configure BIND. However, a well formed BIND file is only the start for good DNS management. Used DNS example

The following domains will be used as examples within this document: /var/named/domain.net on ns0.domain.net domain.net, the firm main domain name ie: bbc.co.uk, cnn.com, isp.net

$ORIGIN domain.net.
domain.net.     86400   IN      SOA
				ns0.domain.net. hostmaster.domain.net. (
				2002020819 28800 7200 604800 86400 )
				NS      ns0.domain.net.
				NS      ns1.domain.net.
				NS      ns2.domain.net.
				MX      10 mx
				MX      20 secondary

mx              A       10.0.0.25
				A       10.0.0.26
				A       10.0.0.27
				A       10.0.0.27

secondary       A       169.254.0.25
				A       169.254.0.26
				A       169.254.0.27
				A       169.254.0.28

ns0             A       10.0.0.1
ns1             A       10.0.0.2
ns2             A       10.0.0.3
ns-secondary0   A       10.0.0.4
ns-cache0       A       10.0.0.5
ns-cache1       A       10.0.0.6

ns-staff0       A       192.168.0.254

smtp            A       10.0.0.25
pop             CNAME   pop3
pop3            A       10.0.0.110
imap            CNAME   imap4
imap4           A       10.0.0.143
webmail         A       10.0.0.443

www             A       10.0.0.80

staff           NS      ns-staff0
				NS      ns-secondary0

/var/named/staff.domain.net on ns0-staff.domain.net staff.domain.net, a delegated domain used by the employee for their own site.

$ORIGIN staff.domain.net.
staff.domain.net.
				86400   IN      SOA
				ns-staff0.domain.net. hostmaster.domain.net. (
				2002021312 28800 7200 604800 86400 )
				NS      ns-staff0.domain.net.
				A       192.168.0.80
				MX      10 mx.domain.net.
				MX      20 secondary.domain.net.
firewall        A       192.168.0.254
smtp            A       192.168.0.25
www             CNAME   firewall
*               CNAME   www
www.*           CNAME   www

/var/named/customer.com on ns0.domain.net customer.com, a domain managed by domain.net owned by one of its customer.

$ORIGIN customer.com.
customer.com.   86400   IN      SOA
				ns0.domain.net. hostmaster.domain.net. (
				2002020819 28800 7200 604800 86400 )
				NS      ns0.domain.net.
				NS      ns1.domain.net.
				NS      ns2.domain.net.
				MX      10 mx.domain.net.
				MX      20 secondary.domain.net.

smtp            CNAME   smtp.domain.net.
pop             CNAME   pop.domain.net
imap            CNAME   imap.domain.net.
www             CNAME   www.domain.net.

/var/named/0.168.192.in-addr.arpa on ns0.domain.net 0.168.192.in-addr.arpais the domain which allow IP to Name DNS

$ORIGIN 0.168.192.in-addr.arpa.
0.168.192.in-addr.arpa.
				86400   IN      SOA
				ns0.domain.net. hostmaster.domain.net. (
				2002021415 28800 7200 604800 86400 )
				NS      ns0.domain.net.
				NS      ns1.domain.net.
				NS      ns2.domain.net.
0               A       255.255.255.0
				PTR     domain.net.
1               PTR     server-at-ip-1.domain.net.
				PTR     another-server-at-ip-1.domain.net.
25              PTR     smtp.staff.domain.net.
254             PTR     firewall.staff.domain.net.

DNS records context

domain.net is an isp or firm domain. The DNS server ns0, ns1 and ns2 for domain.net. are known from the DNS root server (ie: have glue records).

This domain contains all the services that the user of the domain needs to access, such as:

pop
smtp
www

The domain staff.domain.net. is a delegation controlled by the employee and customer.com is the customer domain. it is used for the staff web server

The staff.domain.net only has one DNS server as it is a non-important service. In this example, the DNS is provided by the same server which provides mail and web. This is the only case where you should allow a zone (and with reticence) to only have one DNS server as SOA.

DNS code of conduct

As a genaral rule, all services which are going to be used by an end-user (understand everyone outside the firm IT department) should always be on different IP address, even if all services are provided by single computers.

Whenever possible try to use reserved class C to preserve the pool of real world address.

This is important to make sure that you can migrate any service from the server at any time without disturbing end-users. Using FQDN is not sufficient as you can not be sure that end-users have not misconfigured their computers.

As a golden rule, it is important to not use any mail records for SMTP, POP, IMAP service as this limits your scalability options. There is no such thing such as a mail service.

Also, keep the SMTP and MX record separated. It allows to use simple round-robin for the MX service. Ultimately you could have to have all customer accessible services, such as SMTP, POP, IMAP and HTTP behind load balancers to provide the highest availibility possible.

In the case of SMTP, you can probably use the same server as for MX. However, the secondary MX server will most probably be situated outside your network to avoid mail bouncing in case of network outage.

Whenever possible try to use reserved class C to to preserve the pool of real world addresses. Reserved class C are IP addresses you can not find in the internet reserved for office and private network use. The most frequently used range are:

10.0.0.0/8
192.168.0.0/16
172.16.0.0/16
169.254.0.0/16 (for transfer networks)

Within the domain.net network, customers will use the IP of ns-cache0 and ns-cache1 as their resolving DNS. ns0 and ns1.domain.net should only be queried by other DNS servers for authoritative answers.

The staff.domain.net domain make use of wildcards (star) to catch all DNS name not already present in the list.

As a consequence, the employee will be able to use surname-name.staff.domain.net and www.surname-name.staff.domain.net as names for their web site. No DNS change will be necessary when new staff start or leave the firm.

Be reminded that the use of CNAME record with MX information is not allowed.

As well, if a customer is using your own mail servers, you should never redefine the MX service. Just use your own mx record in their zone file.

Finally, do not redefine customer services pointing to your server per IP but always alias them with CNAME records.

As DNS is a caching system, changes that need quick propagation change must be prepared. To do so you can change the TTL (Time To Live) of a record which represents how long a DNS will keep DNS information. The TTL is expressed in second and is placed just after the name.

Please bear in mind that the first changes performed on the DNS zone file will take up to the previous TTL to be known by all the internet. Restarting your own cache DNS server can speed-up local updates.

For example, if you are planning to move your web server of room and IP.

Initial record

www             A       10.0.0.80

Make sure you have low TTL on your www record. Then wait for the information to propagate. TTL change to 5 minutes

# www           A       10.0.0.80
www     300     A       10.0.0.80

You can move the web server to the new IP as if there are problems you can change the IP address to the previous one in less than 5 minutes.

New information

# www           A       10.0.0.80
www             A       10.0.0.60

Do not forget to restore the default TTL once everything is fine. Built in failover limitation

Unlike the web, DNS was designed with service failure in mind. As it is a crucial service, it is possible to have more than one DNS server answering authoritatively for a domain. However, a common mistake is to think that having two DNS means you are safe. You should make sure that your DNS are on different networks.

In order to achieve the best possible reliability, ISP often have peering agreements to host each others DNS servers.

For example, serious ISP often have one of their Authoritative DNS servers located on another backbone. It provides them protection against BGP problems and Telco faults.

This is very important for mail servers which are performing reverse DNS looking, whithout this precaution any serious outage would cause mail bouncing.

As well, additional protection against malicious plannified Deny Of Service can be deployed to insure the highest DNS uptime possible. Number of DNS

For example, a small/medium ISP will have:

Two authoritative DNS servers in its network
One authoritative DNS servers located remotely
One secondary DNS server for its customers wanting control of their zone
Four caching DNS servers for customers

DNS servers should be presented to the customer classed by proximity depending on their location (for obvious performance reason, DNS is mostly UDP).

Hopefully, DNS can be allocated dynamicly per customer at connection time for most DSL, ISDN or Modem like connection making it easy to change and scale. Local DNS

Every service relies heavily on DNS such as SMTP servers should use its own DNS server and have local resolv.conf like: /etc/resolv.conf

search mydomain.com
domain mydomain.com
nameserver 127.0.0.1
nameserver 10.0.0.1
nameserver 10.0.1.1

Where 10.0.0.1 and 10.0.1.1 are trusted DNS for the server to use should the local DNS server fail. Delegation

Delegation can typically be used when you feel the need to register a new domain name such as : domain-forum.com, domain-resellers.com, domain-users.com, domain-staff.com, etc.

As well, it allows content filter application (such as N2H2, WebSense or SurfControl) to block sub-site without affecting the main site. Ie: webnews.firm.com is better than www.site.com/webnews (Some NewsGroups can provide adult material which may be unsuitable for young surfers).

Delegation allows you to create new domains, independant of your master domain name. These domains are real domains and as such can have different DNS servers as well as different mail or web servers.

The previous example names can be changed as follows: delegated name

forum.domain.com	is better than	domain-forum.com
resellers.domain.com	instead of	domain-resellers.com
users.domain.com	instead of	domain-users.com
staff.domain.com	instead of	domain-staff.com

One obvious advantage is that you do not have to pay for a new domain name.

In addition, it is nearly impossible for a firm to market and advertise more than one domain name and network identity. By using delegation, end users feel secure as they recognise a known domain name.

Delegation can also be used to manage your DNS record. For example, if you provide DSL or a similar kind of connectivity, you may have in your DNS something like:

dsl-10-0-0-1.domain.com
dsl-10-0-0-2.domain.com
…
dsl-10-0-0-253.domain.com

This will make the DNS zone file to fill quickly, which is both bad for management and performance. This can be avoided with the creation of a dsl.domain.com zone:

10-0-0-1.dsl.domain.com
10-0-0-2.dsl.domain.com
…
10-0-0-253.dsl.domain.com

This is only possible if you have DNS management tools with easy front-end. and remember to add these delegations to your /etc/resolv.conf to not have to tape the FQDN (Fully Qualified Domain Name)

Zone delegation works quite well with split horizon, you can have a delegated domain for each office like london.domain.com and paris.domain.com, these domains are invisible outside the offices’ firewalls.

Used in conjonction with the web, it is very handy to manage localisation: www.uk.domain.com can be situated within the uk firm isp when www.fr.domain.com can be hosted in france. Whois and Zone Transfer

Whois is a tool to find information for a domain. It will return the authoritative DNS servers as well as well as some information regarding the registar.

For example the output of “whois bind.com” is:

Whois Server Version 1.3

Domain names in the .com, .net, and .org domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

	Domain Name: BIND.COM
	Registrar: NETWORK SOLUTIONS, INC.
	Whois Server: whois.networksolutions.com
	Referral URL: http://www.networksolutions.com
	Name Server: NS1.DNS.WEBACT.COM
	Name Server: NS2.DNS.WEBACT.COM
	Name Server: NS3.DNS.WEBACT.COM
	Name Server: NS4.DNS.WEBACT.COM
	Updated Date: 07-jan-2002


>>> Last update of whois database: Tue, 5 Mar 2002 05:19:23 EST <<<

The Registry database contains ONLY .COM, .NET, .ORG, .EDU domains and
Registrars.


Found InterNIC referral to whois.networksolutions.com.

The Data in the VeriSign Registrar WHOIS database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information about
or related to a domain name registration record.  VeriSign does not guarantee
its accuracy.  Additionally, the data may not reflect updates to billing contact
information.  By submitting a WHOIS query, you agree to use this Data only
for lawful purposes and that under no circumstances will you use this Data to:
(1) allow, enable, or otherwise support the transmission of mass unsolicited,
commercial advertising or solicitations via e-mail, telephone, or facsimile; or
(2) enable high volume, automated, electronic processes that apply to VeriSign
(or its computer systems).  The compilation, repackaging, dissemination or
other use of this Data is expressly prohibited without the prior written
consent of VeriSign.  VeriSign reserves the right to terminate your access to
the VeriSign Registrar WHOIS database in its sole discretion, including
without limitation, for excessive querying of the WHOIS database or for failure
to otherwise abide by this policy.  VeriSign reserves the right to modify these
terms at any time.  By submitting this query, you agree to abide by this policy.


Registrant:
Quest Technologies, Inc (BIND2-DOM)
	2107 O St. NW
	Washington, DC 20037
	US

	Domain Name: BIND.COM

	Administrative Contact:
	WebAct Administration  (HFJTVUVSUO)  abuse@WEBACT.COM
	WebAct
	2107 O St. NW
	Washington, DC 20037
	US
	202-872-0883
	Fax- 208-460-8163
	Technical Contact:
	WebAct Network Operations Center  (DWOHKUSAGO)  noc@WEBACT.COM
	WebAct
	2107 O St. NW
	Washington, DC 20037
	US
	202-872-0883
	Fax- 208-460-8163
	Billing Contact:
	WebAct Accounts Payable  (XYYGBUVAFO)  billing@WEBACT.COM
	WebAct
	2107 O St. NW
	Washington, DC 20037
	US
	202-872-0883
	Fax- 208-460-8163

	Record last updated on 07-Jan-2002.
	Record expires on 24-Aug-2002.
	Record created on 23-Aug-1996.
	Database last updated on 5-Mar-2002 03:30:00 EST.

	Domain servers in listed order:

	NS1.DNS.WEBACT.COM           207.76.173.19
	NS2.DNS.WEBACT.COM           207.76.173.20
	NS3.DNS.WEBACT.COM           207.76.173.128
	NS4.DNS.WEBACT.COM           207.76.173.129

Zone transfer is a way to get a carbon copy of a zone file from a DNS. Some ISP are blocking this feature to pervent massive security weakness scan (security through obscurity).

Software

Misconfigured DNS servers can cause very hard to debug problems. These problems can remain undetected for month.

If you are serious about DNS you have four options:

To not use BIND 4.x
To not use BIND 8.x
To not use BIND 9.x
To use a software to manage your BIND files

It should be obvious for the reader that I do like D. J. Bernstein’s DJBDNS.

But if after have spent some time reading the DJBDNS site you still want to use BIND, you should use management software for BIND. BIND configuration files are confusing and mistake prone. A badly placed character in a configuration file could cause BIND to refuse reloading or starting.

I was very pleased with a web software called NameSurfer I advise you to take a look at something similar (it was far from free) if you do decide to host your own zones.

Conclusion

When you manage your DNS:

have at least one authoritative DNS outside your network
have a clear zone file template for your customers
splitting service on different IP to force customer to use the right FQDN
differenciate MX, SECONDARY MX, and SMTP to be able to scale your mail
use subdelegation
use some tools to keep your reverse DNS correct

##Glossary

A reserved class C IP is an IP address you can not find on the internet. it is reserved for office and private network use. The available ranges are listed at: http://again.net/cidr. You can as well consult the rfc1918. Authoritative DNS

An authoritative DNS, a abuse of language for DNS servers containing authoritative DNS records, is a DNS which contains the source information for a domain and is registered as such within Internet and answers as such when asked. Glue Record

A glue record is an IP kept by a DNS in order to be able to locate another DNS server. This is used when a DNS is is authoritative for its own domain name.

IE: if ns0.domain.net is authoritative for domain.net, the DNS servers in charge of the .net record need to record the IP of ns0.domain.net in order for other DNS servers to contact it. SOA

NS: DNS record which indicates to the DNS server which server should act authoritatively.

A: DNS record which indicates to the DNS server which server contains DNS information for a given zone

PTR: DNS record which indicates the IP address for a given name

Split horizon, DNS record which indicates the name of a server given its IP. This is not managed automagically by DNS from the A and CNAME record. As a consequence the information can be missing or wrong.

Split horizon DNS are used in conjunction with NAT and firewalls. It means that the DNS answers to the internal DNS queries for local hosts and that it can figure out the IP of external hosts as well.

FQDN: Fully Qualified Domain Name, the name used to name a computer with DNS including the full domain name. Ie: smtp.cnn.com is a FQDN, www or cnn.com are not

The DNS root servers, www.domain.net is in fact an abreviation for www.domain.net. which last dot represent the root DNS server of the Internet (Internet DNS server are a tree structure).

Conclusion

I hope you found this information useful even if this document did not present the problems associated with:

reverse zone
secondary problems
zone transfert from ISP to ISP.