Thread
:
robots.txt file
View Single Post
06-02-2003, 01:39 PM
StupidFuckingWhore
Confirmed User
Join Date: Apr 2003
Posts: 233
robots.txt file
i noticed i don't have a robots.txt file for my sites, is that important? and if so is this all i need in it to have all bots crawl my site?
User-agent: *
Disallow:
StupidFuckingWhore
View Public Profile
Find More Posts by StupidFuckingWhore