Thread: robots.txt file
View Single Post
Old 06-02-2003, 01:39 PM  
StupidFuckingWhore
Confirmed User
 
Join Date: Apr 2003
Posts: 233
robots.txt file
i noticed i don't have a robots.txt file for my sites, is that important? and if so is this all i need in it to have all bots crawl my site?

User-agent: *
Disallow:
StupidFuckingWhore is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote