GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Search engines and password protected directories?

rowan · 03-25-2012, 09:02 PM

A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.

Some sites have set up their system so that googlebot gets access to protected content, so it is indexed.

03-25-2012, 09:02 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also. Some sites have set up their system so that googlebot gets access to protected content, so it is indexed.