The BrowserMob Blog | All about browsers, performance testing, and load testing

Dec/09

4

Google Public DNS vs OpenDNS vs Your ISP’s DNS – measuring performance

124 Comments · Posted by Patrick Lightbody in Uncategorized

Like many in the industry, we were surprised and intrigued by the announcement yesterday that Google would be entering the DNS business. The basic logic was clear: Google has a vested interest in the internet being fast, and so they want to offer a free public utility to help it be faster.

Of course, some were doubtful. OpenDNS, probably the company that has the most to lose by this decision, responded quickly. Some questioned its security, while others pointed out that Google gains a lot more than you might think by serving DNS: it would now know everywhere you were going, regardless of whether you went through Google Search or whether the site had Google Analytics installed.

While we’re not going to get in to the broader debate of whether this move is Good or Evil, we were curious if their service really offered significant performance benefits. So we extracted the DNS code from our BrowserMob website monitoring service (which itself is based on the fantastic xbill Java DNS library), and built a quick-n-dirty tool to measure exactly that.

The results

Here’s how it works: our tool queries the Alexa worldwide top 1000 sites. It does it against Google Public DNS, OpenDNS, and (optionally) the DNS servers of your choosing. Since we’re located in Portland, OR and have a Qwest internet connection, we used Qwest’s DNS servers in our test. The results surprised us: 

Starting test...
 
Test 1: Google
85109 ms for 1000 records
Test 2: Google
67586 ms for 1000 records
Test 3: Google
67318 ms for 1000 records
Test 1: OpenDNS
92521 ms for 1000 records
Test 2: OpenDNS
45793 ms for 1000 records
Test 3: OpenDNS
47899 ms for 1000 records
Test 1: Your DNS
62541 ms for 1000 records
Test 2: Your DNS
25561 ms for 1000 records
Test 3: Your DNS
25879 ms for 1000 records

Taking the lowest time and dividing by 1000 tells us the average DNS lookup times from our location in Portland, OR:

  • Google – 67.3 ms
  • OpenDNS – 45.8 ms
  • Qwest DNS – 25.6 ms

What this told us was that despite all the hoopla about performance, our trusty old DNS server we’ve always been using is still the fastest. And when you think about it, it shouldn’t be a surprise: there are fewer hops from the computer to the DNS server because it’s the same ISP.

We were very surprised, however, to see how much faster OpenDNS was compared to Google. While both are slower than your local ISP’s, OpenDNS promises a bunch of features which might help make up for the performance difference. While Google isn’t promising any features right now, they are, as always, pledging to not be evil. Considering that some ISPs have in the past redirect DNS lookups to serve their commercial interests, Google’s pledge is worth noting.

Running your own test

These results were from a Qwest DSL connection in Portland, OR. We’re curious what your results are, so we’ve made the test available for everyone. It requires Java and can be run like so:

java -jar browsermob-dns-perf.jar

If you wish to test your ISP’s DNS, just add to the command line one or more IPs: 

java -jar browsermob-dns-perf.jar 123.456.789.012

We hope that you will comment on this blog with the results you get. We’re very curious to see what the worldwide performance results are.

A note about the test

DNS is somewhat hard to test for performance, since there are many moving pieces. The results can depend on whether the server has a cached entry, how far you are from the DNS server, how far the DNS server is from the other servers, etc. This is why we run the test three times for each service an why we think the right approach here is to take the best score.

However, we do understand one could easily argue that the first result is much more critical. We sort of agree, but the problem is that because these are public services you can’t tell how “fresh” the cache is on the servers you are testing against.

In fact, we should note that Google’s FAQ argues that cache misses are big performance issue and the main benefit they bring to the table. Unfortunately, this is extremely difficult, if not impossible, for us to independently test. As such, we decided that the most consistent and useful result would be the faster time for each service, even if that means it was faster due to 100% cache hits.

[Post to Twitter] Tweet This Post 

· ·

124 comments

  • jackal · February 21, 2010 at 6:34 pm

    @dave: I wouldn’t think so. It might add a couple of milliseconds, but if you have a fast nameserver, the forwarding time should be negligible. (And your NS will then cache the results, speeding up other lookups within your network.)

    The only thing I’m worried about is the CDN issue mentioned by Michael and Dave above. Anyone know how to get around that? (I can’t see a way.)

  • David · February 21, 2010 at 8:49 pm

    Makawao, HI w/ Hawaiiantel DNS Server 72.235.80.12

    Test 1: Google
    155875 ms for 1000 records
    Test 2: Google
    156375 ms for 1000 records
    Test 3: Google
    138996 ms for 1000 records
    Test 1: OpenDNS
    209118 ms for 1000 records
    Test 2: OpenDNS
    121587 ms for 1000 records
    Test 3: OpenDNS
    156141 ms for 1000 records
    Test 1: Your DNS
    563146 ms for 1000 records
    Test 2: Your DNS
    217105 ms for 1000 records
    Test 3: Your DNS
    196951 ms for 1000 records

  • Patrick Lightbody · February 21, 2010 at 8:53 pm

    @dave

    Your results across two test runs look somewhat consistent for Google vs OpenDNS, but clearly Your DNS was MUCH faster the second run. What I’m thinking is that perhaps Namebench knows how to bypass caching on your internal DNS and our tool isn’t. But that’s just a guess :)

    Patrick

  • kevin · March 3, 2010 at 8:45 pm

    Comcast – Colorado Springs:

    Starting test…

    Test 1: Google
    97652 ms for 1000 records
    Test 2: Google
    94895 ms for 1000 records
    Test 3: Google
    94641 ms for 1000 records
    Test 1: OpenDNS
    229075 ms for 1000 records
    Test 2: OpenDNS
    45765 ms for 1000 records
    Test 3: OpenDNS
    34702 ms for 1000 records
    Test 1: Your DNS
    269050 ms for 1000 records
    Test 2: Your DNS
    206425 ms for 1000 records
    Test 3: Your DNS
    168130 ms for 1000 records

  • Michael · March 14, 2010 at 8:17 am

    San Francisco – AT&T 68.94.156.1

    Test 1: Google
    91869 ms for 1000 records
    Test 2: Google
    56978 ms for 1000 records
    Test 3: Google
    50271 ms for 1000 records
    Test 1: OpenDNS
    51395 ms for 1000 records
    Test 2: OpenDNS
    20597 ms for 1000 records
    Test 3: OpenDNS
    17484 ms for 1000 records
    Test 1: Your DNS
    109757 ms for 1000 records
    Test 2: Your DNS
    100129 ms for 1000 records
    Test 3: Your DNS
    101781 ms for 1000 records

  • Bob Barcus · March 15, 2010 at 3:11 pm

    I did similar testing comparing my ISP with OpenDNS and Google Public DNS with namebench and DNS Benchmark. My ISP easily beat them both, but I had an issue with how often my ISP’s DNS was updated. Right now I’m using OpenDNS and I’m very pleased with the speed and features that it offers.

  • Ola Hamfors · April 3, 2010 at 10:55 am

    From Stockholm, Sweden ISP DNS server 84.246.88.10

    Test 1: Google
    61882 ms for 1000 records
    Test 2: Google
    35087 ms for 1000 records
    Test 3: Google
    37038 ms for 1000 records
    Test 1: OpenDNS
    128097 ms for 1000 records
    Test 2: OpenDNS
    52702 ms for 1000 records
    Test 3: OpenDNS
    41355 ms for 1000 records
    Test 1: Your DNS
    144592 ms for 1000 records
    Test 2: Your DNS
    10565 ms for 1000 records
    Test 3: Your DNS
    6639 ms for 1000 records

  • Steve · April 13, 2010 at 8:26 am

    Melbourne, Australia.

    iPrimus DNS 203.134.64.66

    Test 1: Google
    251519 ms for 1000 records
    Test 2: Google
    268318 ms for 1000 records
    Test 3: Google
    256436 ms for 1000 records
    Test 1: OpenDNS
    461394 ms for 1000 records
    Test 2: OpenDNS
    486686 ms for 1000 records
    Test 3: OpenDNS
    457596 ms for 1000 records
    Test 1: Your DNS
    406216 ms for 1000 records
    Test 2: Your DNS
    88108 ms for 1000 records
    Test 3: Your DNS
    64943 ms for 1000 records

  • Visu · April 16, 2010 at 6:41 am

    Mumbai, India

    ISP – MTNL
    DNS – 59.185.0.23

    Starting test…

    Test 1: Google
    175734 ms for 1000 records
    Test 2: Google
    192138 ms for 1000 records
    Test 3: Google
    181130 ms for 1000 records
    Test 1: OpenDNS
    245980 ms for 1000 records
    Test 2: OpenDNS
    258948 ms for 1000 records
    Test 3: OpenDNS
    278589 ms for 1000 records
    Test 1: Your DNS
    290552 ms for 1000 records
    Test 2: Your DNS
    125095 ms for 1000 records
    Test 3: Your DNS
    158967 ms for 1000 records

    I have been having problems with my ISP’s DNS Server for long. So I am using Google Public DNS now. By this test looks like I did the right thing.

  • Theo · April 20, 2010 at 10:08 am

    MTN Sudan HSDPA Mobile Broadband (212.0.138.13)

    (roundtrip pings out of Sudan about 400-500ms in the first place)

    Test 1: Google
    772560 ms for 1000 records
    Test 2: Google
    514982 ms for 1000 records
    Test 3: Google
    696747 ms for 1000 records
    Test 1: OpenDNS
    682921 ms for 1000 records
    Test 2: OpenDNS
    706675 ms for 1000 records
    Test 3: OpenDNS
    544984 ms for 1000 records
    Test 1: Your DNS
    1021057 ms for 1000 records
    Test 2: Your DNS
    802216 ms for 1000 records
    Test 3: Your DNS
    991916 ms for 1000 records

    ‘Truly dire’ is the only way to describe MTN’s DNS. Given the whopping roundtrip, to be twice as slow to the local DNS really takes something.

  • Keli · May 6, 2010 at 2:20 am

    Hey thanks, all the contributors and commentators for the stats on DNS Operators performance from different parts of the world

  • jason · May 28, 2010 at 6:02 pm

    In Japan…..Fukuoka

    Starting test…

    Test 1: Google
    99074 ms for 1000 records
    Test 2: Google
    77391 ms for 1000 records
    Test 3: Google
    76086 ms for 1000 records
    Test 1: OpenDNS
    228711 ms for 1000 records
    Test 2: OpenDNS
    151404 ms for 1000 records
    Test 3: OpenDNS
    164576 ms for 1000 records
    Test 1: Your DNS
    164012 ms for 1000 records
    Test 2: Your DNS
    145369 ms for 1000 records
    Test 3: Your DNS
    133506 ms for 1000 records

    Google was much faster (as I suspected) ISP was NTT its servers were driving me bananas with disconnects during simple household usage ~brother playing MMORPG he likes and me downloading from a Ubuntu repo and wife surfing on Facebook.. I kept disconnecting sprad-like.

    Now, there are no disconnects in the house. However, this problem could just have been a momentary connection issue.,, However, now I ‘feel better with the whole family on Googles’ public DNS.

  • hak01 · June 3, 2010 at 10:52 pm

    BSNL Kerala

    Starting test…

    Test 1: Google
    211547 ms for 1000 records
    Test 2: Google
    139719 ms for 1000 records
    Test 3: Google
    144656 ms for 1000 records
    Test 1: OpenDNS
    373016 ms for 1000 records
    Test 2: OpenDNS
    324140 ms for 1000 records
    Test 3: OpenDNS
    365875 ms for 1000 records
    Test 1: Your DNS
    391047 ms for 1000 records
    Test 2: Your DNS
    106375 ms for 1000 records
    Test 3: Your DNS
    99953 ms for 1000 records

    Seems to be ISP doing a great job!

  • eric · June 6, 2010 at 2:18 pm

    Amsterdam, The Netherlands

    ISP : http://www.upc.nl/

    Starting test…

    Test 1: Google
    50209 ms for 1000 records
    Test 2: Google
    43788 ms for 1000 records
    Test 3: Google
    21910 ms for 1000 records
    Test 1: OpenDNS
    36418 ms for 1000 records
    Test 2: OpenDNS
    16407 ms for 1000 records
    Test 3: OpenDNS
    21476 ms for 1000 records
    Test 1: Your DNS
    18107 ms for 1000 records
    Test 2: Your DNS
    12058 ms for 1000 records
    Test 3: Your DNS
    12077 ms for 1000 records

  • gohan2091 · June 17, 2010 at 2:27 pm

    How do I run the java file? I have no idea

  • Wayne · June 17, 2010 at 7:45 pm

    TimeWarner Cable – SoCal (Venice)

    OpenDNS Wins. Both beat my ISPs dns.

    Test 1: Google
    193098 ms for 1000 records
    Test 2: Google
    64314 ms for 1000 records
    Test 3: Google
    56439 ms for 1000 records
    Test 1: OpenDNS
    156233 ms for 1000 records
    Test 2: OpenDNS
    27934 ms for 1000 records
    Test 3: OpenDNS
    18356 ms for 1000 records
    Test 1: Your DNS
    232032 ms for 1000 records
    Test 2: Your DNS
    215850 ms for 1000 records
    Test 3: Your DNS
    192384 ms for 1000 records

  • Tim · June 17, 2010 at 10:28 pm

    Castlemaine, Australia

    Test 1: Google
    339688 ms for 1000 records
    Test 2: Google
    263859 ms for 1000 records
    Test 3: Google
    255125 ms for 1000 records
    Test 1: OpenDNS
    309672 ms for 1000 records
    Test 2: OpenDNS
    245156 ms for 1000 records
    Test 3: OpenDNS
    253657 ms for 1000 records
    Test 1: Your DNS
    240547 ms for 1000 records
    Test 2: Your DNS
    48031 ms for 1000 records
    Test 3: Your DNS
    37953 ms for 1000 records

  • John (Verizon FiOS · June 18, 2010 at 4:46 am

    Test 1: Google
    41293 ms for 1000 records
    Test 2: Google
    35022 ms for 1000 records
    Test 3: Google
    29187 ms for 1000 records
    Test 1: OpenDNS
    34648 ms for 1000 records
    Test 2: OpenDNS
    22994 ms for 1000 records
    Test 3: OpenDNS
    25366 ms for 1000 records
    Test 1: Your DNS
    77412 ms for 1000 records
    Test 2: Your DNS
    59701 ms for 1000 records
    Test 3: Your DNS
    50513 ms for 1000 records

  • Tim · June 18, 2010 at 1:01 pm

    Charter Cable, Carolina Beach, NC

    Test 1: Google
    66547 ms for 1000 records
    Test 2: Google
    72422 ms for 1000 records
    Test 3: Google
    84625 ms for 1000 records

    Test 1: OpenDNS
    156844 ms for 1000 records
    Test 2: OpenDNS
    96828 ms for 1000 records
    Test 3: OpenDNS
    84594 ms for 1000 records

    Test 1: Your DNS (24.178.162.3)
    132734 ms for 1000 records
    Test 2: Your DNS
    124547 ms for 1000 records
    Test 3: Your DNS
    121078 ms for 1000 records

  • Mark · June 20, 2010 at 4:17 pm

    Charter Redding, CA 71.9.127.107

    Test 1: Google
    153687 ms for 1000 records
    Test 2: Google
    65406 ms for 1000 records
    Test 3: Google
    51360 ms for 1000 records

    Test 1: OpenDNS
    101681 ms for 1000 records
    Test 2: OpenDNS
    47991 ms for 1000 records
    Test 3: OpenDNS
    45988 ms for 1000 records

    Test 1: Your DNS
    109970 ms for 1000 records
    Test 2: Your DNS
    99867 ms for 1000 records
    Test 3: Your DNS
    107057 ms for 1000 records

  • Mike · June 29, 2010 at 6:36 pm

    Test 1: Google
    107766 ms for 1000 records

    Test 2: Google
    41719 ms for 1000 records

    Test 3: Google
    36916 ms for 1000 records

    Test 1: OpenDNS
    155806 ms for 1000 records

    Test 2: OpenDNS
    47252 ms for 1000 records

    Test 3: OpenDNS
    43198 ms for 1000 records

    Test 1: Your DNS
    119822 ms for 1000 records

    Test 2: Your DNS
    112872 ms for 1000 records

    Test 3: Your DNS
    107520 ms for 1000 records

    The test says that it doesn’t use caching, so I wonder why both Google and OpenDNS showed significant gains after the first test?

  • Drew · July 2, 2010 at 11:16 pm

    Comcast. Chicago

    Test 1: Google
    95199 ms for 1000 records
    Test 2: Google
    51248 ms for 1000 records
    Test 3: Google
    43111 ms for 1000 records
    Test 1: OpenDNS
    166291 ms for 1000 records
    Test 2: OpenDNS
    22823 ms for 1000 records
    Test 3: OpenDNS
    26757 ms for 1000 records
    Test 1: Your DNS
    85246 ms for 1000 records
    Test 2: Your DNS
    66241 ms for 1000 records
    Test 3: Your DNS
    34530 ms for 1000 records

  • Paddo · July 23, 2010 at 7:29 am

    Cape Town, South Africa

    My DNS is my ISP – WebAfrica

    Test 1: Google
    262281 ms for 1000 records
    Test 2: Google
    238313 ms for 1000 records
    Test 3: Google
    248641 ms for 1000 records
    Test 1: OpenDNS
    375562 ms for 1000 records
    Test 2: OpenDNS
    373594 ms for 1000 records
    Test 3: OpenDNS
    481578 ms for 1000 records
    Test 1: Your DNS
    748828 ms for 1000 records
    Test 2: Your DNS
    270516 ms for 1000 records
    Test 3: Your DNS
    129672 ms for 1000 records

    Google and OpenDNS times I can understand but WebAfrica should be better.

  • Thor · July 25, 2010 at 10:15 pm

    Hungary, T-Online DNS 84.2.44.1

    Starting test…

    Test 1: Google
    74250 ms for 1000 records
    Test 2: Google
    74578 ms for 1000 records
    Test 3: Google
    65985 ms for 1000 records
    Test 1: OpenDNS
    280953 ms for 1000 records
    Test 2: OpenDNS
    79812 ms for 1000 records
    Test 3: OpenDNS
    100891 ms for 1000 records
    Test 1: Your DNS
    203156 ms for 1000 records
    Test 2: Your DNS
    184407 ms for 1000 records
    Test 3: Your DNS
    173828 ms for 1000 records

    Google rocks! Almost 3x times faster.

1 2 3

Leave a Reply

<< Flash and Flex automation options using Selenium

Getting Elasticfox to work with Amazon’s new US west coast location >>

Find it!

Theme Design by devolux.nh2.me
To top

Tweet This Post links powered by Tweet This v1.3.9, a WordPress plugin for Twitter.