Search engines and password protected directories?

[Mutt]

Search engine robots can't get to content password protected by htaccess right?

[rowan]

A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.

Some sites have set up their system so that googlebot gets access to protected content, so it is indexed.

[ottopottomouse]

if you do let googlebot in anybody with a user agent switcher is in too.

[Emil]

I have another question, is water good if I'm thirsty?

Will searchengines index my lawn?

Is the searchengine-bot really a spider?

[anexsia]

I have another question, is water good if I'm thirsty?

Will searchengines index my lawn?

Is the searchengine-bot really a spider?

yes

[rowan]

if you do let googlebot in anybody with a user agent switcher is in too.

Of course, that's why you restrict access with both the user-agent and IP range.

[u-Bob]

I have another question, is water good if I'm thirsty?

http://gfy.com/showthread.php?t=1047...ht=dehydration