Robots.txt getting 404 hits?
 

Anyone ever seen 404 hits trying to hit robots.txt? I don't even use a robots.txt file on the domain that I know of. Msn, Yahoo, Googlebot etc are all visiting just find and those show up in the logs. Site only gets around 40 hits a day right now so I'm not sure where the hell this is coming from. Could it be just a weird se spider looking for robots.txt to know what the fuck to follow?

all spiders look for robots.txt Just put a blank robots.txt in your root if you're tired of seeing the 404's

gfy works great sometimes when its on its groove..

is their a plus to adding anything to the robots.txt file?

I haven't noticed but I hate seeing 1,234,135,465 404's from it so I have a default robots.txt

you can tell certain known robots that you might not want using your site to fuck off.

you can use this tool to validate your robots.txt once you have one in place

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

you might want to use a simple one like this

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/


to keep bots/se's out of directories you dont want crawled

thanks for the replies :)

get that gay shit outta here..
wtf

since when are gay signatures with full front nudity allowed on gfy..

anyways.. im not anti-gay.. but come on.. thats nasty..

Smokey fucks trannies, leave him alone.
Dont be a trannyfobe.

i <3 my robots.txt