|
Robots.txt getting 404 hits?
Anyone ever seen 404 hits trying to hit robots.txt? I don't even use a robots.txt file on the domain that I know of. Msn, Yahoo, Googlebot etc are all visiting just find and those show up in the logs. Site only gets around 40 hits a day right now so I'm not sure where the hell this is coming from. Could it be just a weird se spider looking for robots.txt to know what the fuck to follow?
|