|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
03-25-2012, 08:28 PM
|
#1 |
|
Too lazy to set a custom title
Industry Role:
Join Date: Sep 2002
Posts: 34,431
|
Search engines and password protected directories?
Search engine robots can't get to content password protected by htaccess right?
__________________
I moved my sites to Vacares Hosting. I've saved money, my hair is thicker, lost some weight too! Thanks Sly!
|
|
|
|
03-25-2012, 09:02 PM
|
#2 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,373
|
A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.
Some sites have set up their system so that googlebot gets access to protected content, so it is indexed. |
|
|
|
|
03-25-2012, 11:30 PM
|
#3 |
|
She is ugly, bad luck.
Industry Role:
Join Date: Jan 2010
Posts: 13,177
|
if you do let googlebot in anybody with a user agent switcher is in too.
__________________
↑ see post ↑ 13101 |
|
|
|
|
03-26-2012, 12:09 AM
|
#4 |
|
Confirmed User
Join Date: Feb 2007
Location: Sweden
Posts: 5,582
|
I have another question, is water good if I'm thirsty?
Will searchengines index my lawn? Is the searchengine-bot really a spider?
__________________
U.S. NATIONAL DEBT CLOCK 
|
|
|
|
|
03-26-2012, 12:11 AM
|
#5 | |
|
Confirmed User
Industry Role:
Join Date: May 2010
Posts: 5,735
|
Quote:
 |
|
|
|
|
|
03-26-2012, 12:36 AM
|
#6 | |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,373
|
Quote:
|
|
|
|
|
|
03-26-2012, 03:32 AM
|
#7 | |
|
there's no $$$ in porn
Industry Role:
Join Date: Jul 2005
Location: icq: 195./568.-230 (btw: not getting offline msgs)
Posts: 33,063
|
Quote:
|
|
|
|
|
