Anyone ever seen 404 hits trying to hit robots.txt? I don't even use a robots.txt file on the domain that I know of. Msn, Yahoo, Googlebot etc are all visiting just find and those show up in the logs. Site only gets around 40 hits a day right now so I'm not sure where the hell this is coming from. Could it be just a weird se spider looking for robots.txt to know what the fuck to follow?
-
-
all spiders look for robots.txt Just put a blank robots.txt in your root if you're tired of seeing the 404's -
gfy works great sometimes when its on its groove..hatisblack at yahoo.comComment
-
is their a plus to adding anything to the robots.txt file?Comment
-
I haven't noticed but I hate seeing 1,234,135,465 404's from it so I have a default robots.txtOriginally posted by SosaComment
-
you can tell certain known robots that you might not want using your site to fuck off.Originally posted by Sosahatisblack at yahoo.comComment
-
you can use this tool to validate your robots.txt once you have one in place
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
you might want to use a simple one like this
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
to keep bots/se's out of directories you dont want crawledhatisblack at yahoo.comComment
-
thanks for the replies
Comment
-
get that gay shit outta here..
wtf
since when are gay signatures with full front nudity allowed on gfy..
anyways.. im not anti-gay.. but come on.. thats nasty..Comment
-
Smokey fucks trannies, leave him alone.
Dont be a trannyfobe.
•
Sell Patches & Pills •Comment
-
i <3 my robots.txt
Often times I wonder why
There's love and hate, theres live or die.
When sickness comes I must decide:
When feelings go, theres suicide.
Comment
Comment