|
|
|
|
||||
|
|
03-25-2012, 08:28 PM
|
#1 |
|
Private School Jewel
Join Date: Sep 2002
Posts: 31,936
|
Search engines and password protected directories?
Search engine robots can't get to content password protected by htaccess right?
|
|
|
|
03-25-2012, 09:02 PM
|
#2 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 15,657
|
A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.
Some sites have set up their system so that googlebot gets access to protected content, so it is indexed. |
|
|
|
|
03-25-2012, 11:30 PM
|
#3 |
|
She is ugly, bad luck.
Industry Role:
Join Date: Jan 2010
Posts: 11,985
|
if you do let googlebot in anybody with a user agent switcher is in too.
__________________
↑ see post ↑ |
|
|
|
|
03-26-2012, 12:09 AM
|
#4 |
|
Registered User
Join Date: Feb 2007
Location: Sweden
Posts: 5,069
|
I have another question, is water good if I'm thirsty?
Will searchengines index my lawn? Is the searchengine-bot really a spider?
__________________
ก็็็็็็็็็็็็็็็็็็็็ กิิิิิิิิิิิิU.S. NATIONAL DEBT CLOCK 
|
|
|
|
|
03-26-2012, 12:11 AM
|
#5 | |
|
Registered User
Industry Role:
Join Date: May 2010
Posts: 4,604
|
Quote:
 |
|
|
|
|
|
03-26-2012, 12:36 AM
|
#6 | |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 15,657
|
Quote:
|
|
|
|
|
|
03-26-2012, 03:32 AM
|
#7 | |
|
there's no $$$ in porn
Industry Role:
Join Date: Jul 2005
Location: icq: 195./568.-230 (btw: not getting offline msgs)
Posts: 33,071
|
Quote:
|
|
|
|
|
|
|
|
|
Linear Mode
