Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Post New Thread Reply

Register GFY Rules Calendar
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed.

 
Thread Tools
Old 12-23-2007, 11:17 AM   #1
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Anyone know where I can get a list of EVERY domain on the planet?

... or at least the major TLDs? I presume some registrars offer a bulk WHOIS arrangement. T.I.A.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:21 AM   #2
baddog
So Fucking Banned
 
Industry Role:
Join Date: Apr 2001
Location: the beach, SoCal
Posts: 107,089
www.google.com
baddog is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:30 AM   #3
media
Confirmed Moneymaker
 
media's Avatar
 
Industry Role:
Join Date: Apr 2002
Location: Eugene, OR It's Like Jail, Only with Trees!
Posts: 9,852
Quote:
Originally Posted by baddog View Post
I also know Alexa sells lists of top domains with rankings etc.. They sell them pretty pricey..

Top 100,000 domain lists for $6500.00
__________________
I'm here for the violence!
media is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:31 AM   #4
minusonebit
So Fucking Banned
 
Join Date: Feb 2006
Posts: 7,391
The root servers? Good luck getting access to them.

Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.

Last edited by minusonebit; 12-23-2007 at 11:32 AM..
minusonebit is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:45 AM   #5
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by minusonebit View Post
The root servers? Good luck getting access to them.

Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.
WHOIS data would be a nice addition, but all I really need is the domain name.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:50 AM   #6
minusonebit
So Fucking Banned
 
Join Date: Feb 2006
Posts: 7,391
Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
minusonebit is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 11:54 AM   #7
Jayvis
Confirmed User
 
Industry Role:
Join Date: Nov 2005
Location: Cincinnati, Ohio
Posts: 5,428
what planet are we talking about exactly?
Jayvis is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:09 PM   #8
quantum-x
Confirmed User
 
quantum-x's Avatar
 
Join Date: Feb 2002
Location: ICQ: 251425 Fr/Au/Ca
Posts: 6,863
Quote:
Originally Posted by minusonebit View Post
Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
Way too slow.
quantum-x is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:16 PM   #9
Jarmusch
 
 
Industry Role:
Join Date: May 2003
Posts: 12,464
Quote:
Originally Posted by Jayvis View Post
what planet are we talking about exactly?
Jarmusch is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:28 PM   #10
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by minusonebit View Post
Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
I already have a crawler running, but I want to feed it domains that it won't find by itself.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:38 PM   #11
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by minusonebit View Post
Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
Just re-read this, I misunderstood your suggestion. Doing a reverse lookup on all IPs from 0.0.0.1 to 255.255.255.255 is going to result in a lot of DNS fails, and there's no way it will find every single domain... at least with a HTTP based crawler it will find (on average) at least a few new hostnames for each site it fetches.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:41 PM   #12
Brujah
Beer Money Baron
 
Brujah's Avatar
 
Industry Role:
Join Date: Jan 2001
Location: brujah / gmail
Posts: 22,157
You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.
__________________
Brujah is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:41 PM   #13
JFK
FUBAR the ORIGINATOR
 
JFK's Avatar
 
Industry Role:
Join Date: Jan 2002
Location: FUBARLAND
Posts: 67,374
I sale short Planet domain lists
__________________

FUBAR Webmasters - The FUBAR Times - FUBAR Webmasters Mobile - FUBARTV.XXX
For promo opps contact jfk at fubarwebmasters dot com
JFK is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:44 PM   #14
Iron Fist
Too lazy to set a custom title
 
Join Date: Dec 2006
Posts: 23,400
Quote:
Originally Posted by baddog View Post
__________________
i like waffles
Iron Fist is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 12:55 PM   #15
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by Brujah View Post
You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.
Thanks, I might contact you further into the future.

Any hints at how many domains there are in total? I've "budgeted" for about 500 million in my database.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 02:25 PM   #16
Brujah
Beer Money Baron
 
Brujah's Avatar
 
Industry Role:
Join Date: Jan 2001
Location: brujah / gmail
Posts: 22,157
There are somewhere around 70 million in the .com zone.
__________________
Brujah is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 02:36 PM   #17
Sansa
Confirmed User
 
Join Date: Apr 2007
Posts: 293
If you have a reseller account with a registrar like tucows you can request a weekly dump.
Sansa is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 03:01 PM   #18
V_RocKs
Damn Right I Kiss Ass!
 
Industry Role:
Join Date: Dec 2003
Location: Cowtown, USA
Posts: 32,409
I got one here in my back pocket.
V_RocKs is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 04:18 PM   #19
Sebastian Sands
Confirmed User
 
Industry Role:
Join Date: Mar 2005
Location: ICQ: 211-417-740
Posts: 5,223
I have such a list, how much are you willing to spend?
Sebastian Sands is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 04:28 PM   #20
WiredGuy
Pounding Googlebot
 
Industry Role:
Join Date: Aug 2002
Location: Canada
Posts: 34,482
Network Solutions used to offer this, back in 2001/2002 I used to have access to their program that did this and the files were absolutely massive. Just the .net database was about 2-3 gigs of gzip files containing nothing more than the domain name. Today these files have got to be enormous.
WG
__________________
I play with Google.
WiredGuy is online now   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 05:06 PM   #21
DirtyProfits
Confirmed User
 
Join Date: May 2005
Posts: 1,885
http://www.google.com/search?hl=en&q...=Google+Search

You want a list with 16+ billion domains?
DirtyProfits is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 07:17 PM   #22
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by DirtyProfits View Post
http://www.google.com/search?hl=en&q...=Google+Search

You want a list with 16+ billion domains?
Those search results will include hostnames (eg "inurl:blogspot.com" returns 9 million results alone) and most likely multiple pages from a particular hostname. I'm more inclined to believe Brujah's figure of 70 mil .com domains

BTW inurl:.com returns a bunch of well known sites - who needs to pay for Alexa's top 1,000?
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-23-2007, 10:19 PM   #23
gideongallery
Confirmed User
 
Join Date: Aug 2003
Posts: 7,082
contact verisign they used to sell the master list for marketing purposes
i think the list went for 10 k about 6 years ago
i know a couple of call centers who bought the list
__________________

“When crimes occur through the mail, you don’t shut the post office down,” Steve Wozniak
gideongallery is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-24-2007, 06:03 AM   #24
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by gideongallery View Post
contact verisign they used to sell the master list for marketing purposes
i think the list went for 10 k about 6 years ago
i know a couple of call centers who bought the list
For $10k I think I'll let my crawler keep wandering.....
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-24-2007, 06:08 AM   #25
Barefootsies
Choice is an Illusion
 
Barefootsies's Avatar
 
Industry Role:
Join Date: Feb 2005
Location: Land of Obama
Posts: 42,635
Quote:
Originally Posted by Jayvis View Post
what planet are we talking about exactly?
__________________
Should You Email Your Members?

Link1 | Link2 | Link3

Enough Said.

"Would you rather live like a king for a year or like a prince forever?"
Barefootsies is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-25-2007, 12:58 PM   #26
GeorgeK
Confirmed User
 
Join Date: Dec 2003
Location: Toronto, Canada
Posts: 423
The com/net zone files can be obtained for free via FTP by applying to:

http://www.verisign.com/information-...ge_001052.html

For .org, the relevant link is:

http://www.pir.org/RegistrarResource...ileAccess.aspx

Some other TLD operators have similar programs, but the above will give you a good start.
__________________
I buy good domain names. Send lists to George (at) LOFFS.com
GeorgeK is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Post New Thread Reply
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >

Bookmarks



Advertising inquiries - marketing at gfy dot com

Contact Admin - Advertise - GFY Rules - Top

©2000-, AI Media Network Inc



Powered by vBulletin
Copyright © 2000- Jelsoft Enterprises Limited.