Quote:
Originally posted by jay23
I agree, but how do you decide what sites to spider ?
|
All adult sites are spidered. As more are found when spidering the sites already in the system, they are then included.
If someone happens to request a site that isn't currently being spidered, then naturally it will be added, as well as all the sites linking to it, and the sites linking to them.
Currently there are at least 1,795,718 unique domains