GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   robots.txt....WTF? (https://gfy.com/showthread.php?t=205328)

Greg B 12-06-2003 11:15 AM
robots.txt....WTF?
 
Okay, been a while, but what's the advantage of having a robots.txt file?

Spunky 12-06-2003 11:17 AM
SE use them to spider your pages?

gleb 12-06-2003 11:19 AM
Quote:
Originally posted by spunky1
SE use them to spider your pages?
yupz, rules for search engines http://www.searchengineworld.com/rob...s_tutorial.htm


All times are GMT -7. The time now is 06:09 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123