GoFuckYourself.com - Adult Webmaster Forum - View Single Post

jasonir · 01-28-2004, 11:59 PM

Hmmm..
I swear they USED to have a robots.txt file.

Anyways...

It's not crawled because of the GET data tacked on to the ends of the URLs.

01-28-2004, 11:59 PM
jasonir Confirmed User Join Date: Aug 2002 Location: Toro'no Posts: 1,887	Hmmm.. I swear they USED to have a robots.txt file. Anyways... It's not crawled because of the GET data tacked on to the ends of the URLs.