GoFuckYourself.com - Adult Webmaster Forum

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - google question (https://gfy.com/showthread.php?t=494713)

Chris

07-21-2005 01:02 PM

google question

do you have to have robotos.txt to get google to spider??/

hive	07-21-2005 01:16 PM

No.

http://www.google.com/addurl/?continue=/addurl

Your best bet for a speedy index inclusion is asking some people who run related sites for links.

Chris

07-21-2005 01:21 PM

Quote:

Originally Posted by hive

No.

http://www.google.com/addurl/?continue=/addurl

Your best bet for a speedy index inclusion is asking some people who run related sites for links.

well google is hitting my site already but my stats show robots.txt keeps getting requested

Nismo

07-21-2005 01:22 PM

hell no. I have tons of google listings and no robots.txt

nojob

07-21-2005 01:26 PM

no you do not.

psili

07-21-2005 01:36 PM

Quote:

Originally Posted by JupZChris

well google is hitting my site already but my stats show robots.txt keeps getting requested

I think the spiders default behavior is to request that file, so if it's there the spider will know what to index or not.

xfile

07-21-2005 01:42 PM

I don't use robots.txt.

OzMan

07-21-2005 07:46 PM

Yes, to get listed fast on these servers it would be moooy bwayno to have a el robotos punta txt

All times are GMT -7. The time now is 01:28 PM.