Robots.txt getting 404 hits? - GoFuckYourself.com

Sosa · 08-17-2005, 09:58 PM

Anyone ever seen 404 hits trying to hit robots.txt? I don't even use a robots.txt file on the domain that I know of. Msn, Yahoo, Googlebot etc are all visiting just find and those show up in the logs. Site only gets around 40 hits a day right now so I'm not sure where the hell this is coming from. Could it be just a weird se spider looking for robots.txt to know what the fuck to follow?

JD · 08-17-2005, 10:13 PM

all spiders look for robots.txt Just put a blank robots.txt in your root if you're tired of seeing the 404's

SmokeyTheBear · 08-17-2005, 10:16 PM

gfy works great sometimes when its on its groove..

Sosa · 08-17-2005, 10:17 PM

is their a plus to adding anything to the robots.txt file?

JD · 08-17-2005, 10:18 PM

Quote:

Originally Posted by Sosa

is their a plus to adding anything to the robots.txt file?

I haven't noticed but I hate seeing 1,234,135,465 404's from it so I have a default robots.txt

SmokeyTheBear · 08-17-2005, 10:18 PM

Quote:

Originally Posted by Sosa

is their a plus to adding anything to the robots.txt file?

you can tell certain known robots that you might not want using your site to fuck off.

SmokeyTheBear · 08-17-2005, 10:22 PM

you can use this tool to validate your robots.txt once you have one in place

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

you might want to use a simple one like this

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

to keep bots/se's out of directories you dont want crawled

Sosa · 08-17-2005, 10:38 PM

thanks for the replies

the(G)thang · 08-17-2005, 10:40 PM

get that gay shit outta here..
wtf

since when are gay signatures with full front nudity allowed on gfy..

anyways.. im not anti-gay.. but come on.. thats nasty..

emthree · 08-18-2005, 12:10 AM

Smokey fucks trannies, leave him alone.
Dont be a trannyfobe.

nofx · 08-18-2005, 12:15 AM

i <3 my robots.txt

08-17-2005, 09:58 PM	#1
Sosa In Tushy Land Join Date: Oct 2002 Location: Nebraska Posts: 40,149	Robots.txt getting 404 hits? Anyone ever seen 404 hits trying to hit robots.txt? I don't even use a robots.txt file on the domain that I know of. Msn, Yahoo, Googlebot etc are all visiting just find and those show up in the logs. Site only gets around 40 hits a day right now so I'm not sure where the hell this is coming from. Could it be just a weird se spider looking for robots.txt to know what the fuck to follow?

08-17-2005, 10:16 PM	#3
SmokeyTheBear ►SouthOfHeaven Join Date: Jun 2004 Location: PlanetEarth MyBoardRank: GerbilMaster My-Penis-Size: extralarge MyWeapon: Computer Posts: 28,609	gfy works great sometimes when its on its groove.. __________________ hatisblack at yahoo.com

08-17-2005, 10:22 PM	#7
SmokeyTheBear ►SouthOfHeaven Join Date: Jun 2004 Location: PlanetEarth MyBoardRank: GerbilMaster My-Penis-Size: extralarge MyWeapon: Computer Posts: 28,609	you can use this tool to validate your robots.txt once you have one in place http://www.searchengineworld.com/cgi-bin/robotcheck.cgi you might want to use a simple one like this User-agent: * Disallow: /cgi-bin/ Disallow: /images/ to keep bots/se's out of directories you dont want crawled __________________ hatisblack at yahoo.com

08-17-2005, 10:40 PM	#9
the(G)thang Confirmed User Join Date: Jan 2002 Location: Toronto, Ontario, Canada Posts: 491	get that gay shit outta here.. wtf since when are gay signatures with full front nudity allowed on gfy.. anyways.. im not anti-gay.. but come on.. thats nasty.. __________________ [email protected]

08-18-2005, 12:10 AM	#10
emthree Dialer Kingpin Join Date: Jun 2003 Location: New York Posts: 10,816	Smokey fucks trannies, leave him alone. Dont be a trannyfobe. __________________ • Sell Patches & Pills •

08-17-2005, 10:13 PM	#2
JD Too lazy to set a custom title Industry Role: Join Date: Sep 2003 Posts: 22,651	all spiders look for robots.txt Just put a blank robots.txt in your root if you're tired of seeing the 404's

08-17-2005, 10:17 PM	#4
Sosa In Tushy Land Join Date: Oct 2002 Location: Nebraska Posts: 40,149	is their a plus to adding anything to the robots.txt file?

08-17-2005, 10:38 PM	#8
Sosa In Tushy Land Join Date: Oct 2002 Location: Nebraska Posts: 40,149	thanks for the replies

08-18-2005, 12:15 AM	#11
nofx Too lazy to set a custom title Join Date: Nov 2002 Location: Virgin Mary's womb Posts: 16,826	i <3 my robots.txt __________________ Often times I wonder why There's love and hate, theres live or die. When sickness comes I must decide: When feelings go, theres suicide.