Anyone know where I can get a list of EVERY domain on the planet?
 

... or at least the major TLDs? I presume some registrars offer a bulk WHOIS arrangement. T.I.A.

I also know Alexa sells lists of top domains with rankings etc.. They sell them pretty pricey..

Top 100,000 domain lists for $6500.00

The root servers? Good luck getting access to them. :)

Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.

WHOIS data would be a nice addition, but all I really need is the domain name.

Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file... :)

what planet are we talking about exactly?

Way too slow.

:1orglaugh

I already have a crawler running, but I want to feed it domains that it won't find by itself.

Just re-read this, I misunderstood your suggestion. Doing a reverse lookup on all IPs from 0.0.0.1 to 255.255.255.255 is going to result in a lot of DNS fails, and there's no way it will find every single domain... at least with a HTTP based crawler it will find (on average) at least a few new hostnames for each site it fetches.

You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.

I sale short Planet domain lists :Graucho

:1orglaugh:1orglaugh:1orglaugh

Thanks, I might contact you further into the future.

Any hints at how many domains there are in total? I've "budgeted" for about 500 million in my database.

There are somewhere around 70 million in the .com zone.

If you have a reseller account with a registrar like tucows you can request a weekly dump.

I got one here in my back pocket.

I have such a list, how much are you willing to spend?

Network Solutions used to offer this, back in 2001/2002 I used to have access to their program that did this and the files were absolutely massive. Just the .net database was about 2-3 gigs of gzip files containing nothing more than the domain name. Today these files have got to be enormous.
WG

http://www.google.com/search?hl=en&q...=Google+Search

You want a list with 16+ billion domains?

Those search results will include hostnames (eg "inurl:blogspot.com" returns 9 million results alone) and most likely multiple pages from a particular hostname. I'm more inclined to believe Brujah's figure of 70 mil .com domains :)

BTW inurl:.com returns a bunch of well known sites - who needs to pay for Alexa's top 1,000? :thumbsup

contact verisign they used to sell the master list for marketing purposes
i think the list went for 10 k about 6 years ago
i know a couple of call centers who bought the list

For $10k I think I'll let my crawler keep wandering.....

:2 cents:

The com/net zone files can be obtained for free via FTP by applying to:

http://www.verisign.com/information-...ge_001052.html

For .org, the relevant link is:

http://www.pir.org/RegistrarResource...ileAccess.aspx

Some other TLD operators have similar programs, but the above will give you a good start.