Anyone know where I can get a list of EVERY domain on the planet? - GoFuckYourself.com

rowan · 12-23-2007, 11:17 AM

... or at least the major TLDs? I presume some registrars offer a bulk WHOIS arrangement. T.I.A.

baddog · 12-23-2007, 11:21 AM

www.google.com

media · 12-23-2007, 11:30 AM

Quote:

Originally Posted by baddog

www.google.com

I also know Alexa sells lists of top domains with rankings etc.. They sell them pretty pricey..

Top 100,000 domain lists for $6500.00

minusonebit · 12-23-2007, 11:31 AM

The root servers? Good luck getting access to them.

Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.

rowan · 12-23-2007, 11:45 AM

Quote:

Originally Posted by minusonebit

The root servers? Good luck getting access to them.

Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.

WHOIS data would be a nice addition, but all I really need is the domain name.

minusonebit · 12-23-2007, 11:50 AM

Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

Jayvis · 12-23-2007, 11:54 AM

what planet are we talking about exactly?

quantum-x · 12-23-2007, 12:09 PM

Quote:

Originally Posted by minusonebit

Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

Way too slow.

Jarmusch · 12-23-2007, 12:16 PM

Quote:

Originally Posted by Jayvis

what planet are we talking about exactly?

rowan · 12-23-2007, 12:28 PM

Quote:

Originally Posted by minusonebit

Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

I already have a crawler running, but I want to feed it domains that it won't find by itself.

rowan · 12-23-2007, 12:38 PM

Quote:

Originally Posted by minusonebit

Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

Just re-read this, I misunderstood your suggestion. Doing a reverse lookup on all IPs from 0.0.0.1 to 255.255.255.255 is going to result in a lot of DNS fails, and there's no way it will find every single domain... at least with a HTTP based crawler it will find (on average) at least a few new hostnames for each site it fetches.

Brujah · 12-23-2007, 12:41 PM

You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.

JFK · 12-23-2007, 12:41 PM

I sale short Planet domain lists

Iron Fist · 12-23-2007, 12:44 PM

Quote:

Originally Posted by baddog

www.google.com

rowan · 12-23-2007, 12:55 PM

Quote:

Originally Posted by Brujah

You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.

Thanks, I might contact you further into the future.

Any hints at how many domains there are in total? I've "budgeted" for about 500 million in my database.

Brujah · 12-23-2007, 02:25 PM

There are somewhere around 70 million in the .com zone.

Sansa · 12-23-2007, 02:36 PM

If you have a reseller account with a registrar like tucows you can request a weekly dump.

V_RocKs · 12-23-2007, 03:01 PM

I got one here in my back pocket.

Sebastian Sands · 12-23-2007, 04:18 PM

I have such a list, how much are you willing to spend?

WiredGuy · 12-23-2007, 04:28 PM

Network Solutions used to offer this, back in 2001/2002 I used to have access to their program that did this and the files were absolutely massive. Just the .net database was about 2-3 gigs of gzip files containing nothing more than the domain name. Today these files have got to be enormous.
WG

DirtyProfits · 12-23-2007, 05:06 PM

http://www.google.com/search?hl=en&q...=Google+Search

You want a list with 16+ billion domains?

rowan · 12-23-2007, 07:17 PM

Quote:

Originally Posted by DirtyProfits

http://www.google.com/search?hl=en&q...=Google+Search

You want a list with 16+ billion domains?

Those search results will include hostnames (eg "inurl:blogspot.com" returns 9 million results alone) and most likely multiple pages from a particular hostname. I'm more inclined to believe Brujah's figure of 70 mil .com domains

BTW inurl:.com returns a bunch of well known sites - who needs to pay for Alexa's top 1,000?

gideongallery · 12-23-2007, 10:19 PM

contact verisign they used to sell the master list for marketing purposes
i think the list went for 10 k about 6 years ago
i know a couple of call centers who bought the list

rowan · 12-24-2007, 06:03 AM

Quote:

Originally Posted by gideongallery

contact verisign they used to sell the master list for marketing purposes
i think the list went for 10 k about 6 years ago
i know a couple of call centers who bought the list

For $10k I think I'll let my crawler keep wandering.....

Barefootsies · 12-24-2007, 06:08 AM

Quote:

Originally Posted by Jayvis

what planet are we talking about exactly?

GeorgeK · 12-25-2007, 12:58 PM

The com/net zone files can be obtained for free via FTP by applying to:

http://www.verisign.com/information-...ge_001052.html

For .org, the relevant link is:

http://www.pir.org/RegistrarResource...ileAccess.aspx

Some other TLD operators have similar programs, but the above will give you a good start.

12-23-2007, 11:17 AM	#1
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Anyone know where I can get a list of EVERY domain on the planet? ... or at least the major TLDs? I presume some registrars offer a bulk WHOIS arrangement. T.I.A.

12-23-2007, 11:31 AM	#4
minusonebit So Fucking Banned Join Date: Feb 2006 Location: [email protected] Posts: 7,391	The root servers? Good luck getting access to them. Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure. Last edited by minusonebit; 12-23-2007 at 11:32 AM..

12-23-2007, 12:41 PM	#12
Brujah Beer Money Baron Industry Role: Join Date: Jan 2001 Location: brujah / gmail Posts: 22,157	You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me. __________________ Free Premium Domain Lists and Tools at Clickmojo.com For Sale: Obscenity.com

12-23-2007, 12:41 PM	#13
JFK FUBAR the ORIGINATOR Industry Role: Join Date: Jan 2002 Location: FUBARLAND Posts: 67,374	I sale short Planet domain lists __________________ FUBAR Webmasters - The FUBAR Times - FUBAR Webmasters Mobile - FUBARTV.XXX For promo opps contact jfk at fubarwebmasters dot com

12-23-2007, 02:25 PM	#16
Brujah Beer Money Baron Industry Role: Join Date: Jan 2001 Location: brujah / gmail Posts: 22,157	There are somewhere around 70 million in the .com zone. __________________ Free Premium Domain Lists and Tools at Clickmojo.com For Sale: Obscenity.com

12-23-2007, 11:21 AM	#2
baddog So Fucking Banned Industry Role: Join Date: Apr 2001 Location: the beach, SoCal Posts: 107,089	www.google.com

12-23-2007, 11:50 AM	#6
minusonebit So Fucking Banned Join Date: Feb 2006 Location: [email protected] Posts: 7,391	Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

12-23-2007, 11:54 AM	#7
Jayvis Confirmed User Industry Role: Join Date: Nov 2005 Location: Cincinnati, Ohio Posts: 5,428	what planet are we talking about exactly?

12-23-2007, 02:36 PM	#17
Sansa Confirmed User Join Date: Apr 2007 Posts: 293	If you have a reseller account with a registrar like tucows you can request a weekly dump.

12-23-2007, 03:01 PM	#18
V_RocKs Damn Right I Kiss Ass! Industry Role: Join Date: Dec 2003 Location: Cowtown, USA Posts: 32,409	I got one here in my back pocket.

12-23-2007, 04:18 PM	#19
Sebastian Sands Confirmed User Industry Role: Join Date: Mar 2005 Location: ICQ: 211-417-740 Posts: 5,223	I have such a list, how much are you willing to spend?

12-23-2007, 04:28 PM	#20
WiredGuy Pounding Googlebot Industry Role: Join Date: Aug 2002 Location: Canada Posts: 34,482	Network Solutions used to offer this, back in 2001/2002 I used to have access to their program that did this and the files were absolutely massive. Just the .net database was about 2-3 gigs of gzip files containing nothing more than the domain name. Today these files have got to be enormous. WG __________________ I play with Google.

12-23-2007, 05:06 PM	#21
DirtyProfits Confirmed User Join Date: May 2005 Posts: 1,885	http://www.google.com/search?hl=en&q...=Google+Search You want a list with 16+ billion domains?

12-23-2007, 10:19 PM	#23
gideongallery Confirmed User Join Date: Aug 2003 Posts: 7,082	contact verisign they used to sell the master list for marketing purposes i think the list went for 10 k about 6 years ago i know a couple of call centers who bought the list __________________ “When crimes occur through the mail, you don’t shut the post office down,” Steve Wozniak

12-25-2007, 12:58 PM	#26
GeorgeK Confirmed User Join Date: Dec 2003 Location: Toronto, Canada Posts: 423	The com/net zone files can be obtained for free via FTP by applying to: http://www.verisign.com/information-...ge_001052.html For .org, the relevant link is: http://www.pir.org/RegistrarResource...ileAccess.aspx Some other TLD operators have similar programs, but the above will give you a good start. __________________ I buy good domain names. Send lists to George (at) LOFFS.com