Anyone know where I can get a list of EVERY domain on the planet?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • rowan
    Too lazy to set a custom title
    • Mar 2002
    • 17393

    #1

    Anyone know where I can get a list of EVERY domain on the planet?

    ... or at least the major TLDs? I presume some registrars offer a bulk WHOIS arrangement. T.I.A.
  • baddog
    So Fucking Banned
    • Apr 2001
    • 107089

    #2
    www.google.com

    Comment

    • media
      Confirmed Moneymaker
      • Apr 2002
      • 9853

      #3
      Originally posted by baddog
      I also know Alexa sells lists of top domains with rankings etc.. They sell them pretty pricey..

      Top 100,000 domain lists for $6500.00
      I'm here for the violence!

      Comment

      • minusonebit
        So Fucking Banned
        • Feb 2006
        • 7391

        #4
        The root servers? Good luck getting access to them.

        Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.
        Last edited by minusonebit; 12-23-2007, 09:32 AM.

        Comment

        • rowan
          Too lazy to set a custom title
          • Mar 2002
          • 17393

          #5
          Originally posted by minusonebit
          The root servers? Good luck getting access to them.

          Seriously, I don't think there would be such a list. Even if one register agreed to let you access their whois servers on an unlimited basis, they'd still have to query servers outside their net and those would not permit such bulk activity I am sure.
          WHOIS data would be a nice addition, but all I really need is the domain name.

          Comment

          • minusonebit
            So Fucking Banned
            • Feb 2006
            • 7391

            #6
            Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...

            Comment

            • Jayvis
              Confirmed User
              • Nov 2005
              • 5430

              #7
              what planet are we talking about exactly?

              Comment

              • quantum-x
                Confirmed User
                • Feb 2002
                • 6863

                #8
                Originally posted by minusonebit
                Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
                Way too slow.
                PrettyInCash.com - BoozedGFs.com - TeenGFs.com - JizzGFs.com- MilfUploads.com -

                Comment

                • Jarmusch
                   
                  • May 2003
                  • 12479

                  #9
                  Originally posted by Jayvis
                  what planet are we talking about exactly?

                  Comment

                  • rowan
                    Too lazy to set a custom title
                    • Mar 2002
                    • 17393

                    #10
                    Originally posted by minusonebit
                    Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
                    I already have a crawler running, but I want to feed it domains that it won't find by itself.

                    Comment

                    • rowan
                      Too lazy to set a custom title
                      • Mar 2002
                      • 17393

                      #11
                      Originally posted by minusonebit
                      Maybe write a script to crawl across the net and dig every IP and record all the domains there? Gonna be a big file...
                      Just re-read this, I misunderstood your suggestion. Doing a reverse lookup on all IPs from 0.0.0.1 to 255.255.255.255 is going to result in a lot of DNS fails, and there's no way it will find every single domain... at least with a HTTP based crawler it will find (on average) at least a few new hostnames for each site it fetches.

                      Comment

                      • Brujah
                        Beer Money Baron
                        • Jan 2001
                        • 22157

                        #12
                        You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.

                        Comment

                        • JFK
                          FUBAR the ORIGINATOR
                          • Jan 2002
                          • 67373

                          #13
                          I sale short Planet domain lists

                          FUBAR Webmasters - The FUBAR Times - FUBAR Webmasters Mobile - FUBARTV.XXX
                          For promo opps contact jfk at fubarwebmasters dot com

                          Comment

                          • Iron Fist
                            Too lazy to set a custom title
                            • Dec 2006
                            • 23400

                            #14
                            Originally posted by baddog
                            i like waffles

                            Comment

                            • rowan
                              Too lazy to set a custom title
                              • Mar 2002
                              • 17393

                              #15
                              Originally posted by Brujah
                              You need the registry zonefiles. Most have a system to allow you to apply for access. If you just need a one-time deal, contact me.
                              Thanks, I might contact you further into the future.

                              Any hints at how many domains there are in total? I've "budgeted" for about 500 million in my database.

                              Comment

                              • Brujah
                                Beer Money Baron
                                • Jan 2001
                                • 22157

                                #16
                                There are somewhere around 70 million in the .com zone.

                                Comment

                                • Sansa
                                  Confirmed User
                                  • Apr 2007
                                  • 293

                                  #17
                                  If you have a reseller account with a registrar like tucows you can request a weekly dump.
                                  Beat the shitty economy by becoming a loan shark.

                                  Comment

                                  • V_RocKs
                                    Damn Right I Kiss Ass!
                                    • Nov 2003
                                    • 32449

                                    #18
                                    I got one here in my back pocket.

                                    Comment

                                    • Sebastian Sands
                                      Confirmed User
                                      • Mar 2005
                                      • 5223

                                      #19
                                      I have such a list, how much are you willing to spend?

                                      Comment

                                      • WiredGuy
                                        Pounding Googlebot
                                        • Aug 2002
                                        • 34512

                                        #20
                                        Network Solutions used to offer this, back in 2001/2002 I used to have access to their program that did this and the files were absolutely massive. Just the .net database was about 2-3 gigs of gzip files containing nothing more than the domain name. Today these files have got to be enormous.
                                        WG
                                        I play with Google.

                                        Comment

                                        • DirtyProfits
                                          Confirmed User
                                          • May 2005
                                          • 1885

                                          #21
                                          http://www.google.com/search?hl=en&q...=Google+Search

                                          You want a list with 16+ billion domains?

                                          Comment

                                          • rowan
                                            Too lazy to set a custom title
                                            • Mar 2002
                                            • 17393

                                            #22
                                            Originally posted by DirtyProfits
                                            http://www.google.com/search?hl=en&q...=Google+Search

                                            You want a list with 16+ billion domains?
                                            Those search results will include hostnames (eg "inurl:blogspot.com" returns 9 million results alone) and most likely multiple pages from a particular hostname. I'm more inclined to believe Brujah's figure of 70 mil .com domains

                                            BTW inurl:.com returns a bunch of well known sites - who needs to pay for Alexa's top 1,000?

                                            Comment

                                            • gideongallery
                                              Confirmed User
                                              • Aug 2003
                                              • 7082

                                              #23
                                              contact verisign they used to sell the master list for marketing purposes
                                              i think the list went for 10 k about 6 years ago
                                              i know a couple of call centers who bought the list

                                              “When crimes occur through the mail, you don’t shut the post office down,” Steve Wozniak

                                              Comment

                                              • rowan
                                                Too lazy to set a custom title
                                                • Mar 2002
                                                • 17393

                                                #24
                                                Originally posted by gideongallery
                                                contact verisign they used to sell the master list for marketing purposes
                                                i think the list went for 10 k about 6 years ago
                                                i know a couple of call centers who bought the list
                                                For $10k I think I'll let my crawler keep wandering.....

                                                Comment

                                                • Barefootsies
                                                  Choice is an Illusion
                                                  • Feb 2005
                                                  • 42635

                                                  #25
                                                  Originally posted by Jayvis
                                                  what planet are we talking about exactly?
                                                  Should You Email Your Members?

                                                  Link1 | Link2 | Link3

                                                  Enough Said.

                                                  "Would you rather live like a king for a year or like a prince forever?"

                                                  Comment

                                                  • GeorgeK
                                                    Confirmed User
                                                    • Dec 2003
                                                    • 423

                                                    #26
                                                    The com/net zone files can be obtained for free via FTP by applying to:

                                                    http://www.verisign.com/information-...ge_001052.html

                                                    For .org, the relevant link is:

                                                    http://www.pir.org/RegistrarResource...ileAccess.aspx

                                                    Some other TLD operators have similar programs, but the above will give you a good start.
                                                    I buy good domain names. Send lists to George (at) LOFFS.com

                                                    Comment

                                                    Working...