|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
03-06-2002, 10:50 AM
|
#1 |
|
Confirmed User
Join Date: Feb 2002
Location: Seattle, WA
Posts: 1,527
|
Meta Tags
Does anyone know the meta tags to insert so people can't mass-download from your members area with a bot?
Or a good idea to block the way a bot massive grabs your content... |
|
|
|
03-06-2002, 10:51 AM
|
#2 |
|
Confirmed User
Industry Role:
Join Date: Jan 2002
Location: Cherry Hill, NJ
Posts: 3,615
|
Robots.txt file in your root directory of your site can keep SE's from indexing you members area, I'm not sure about downloading.
|
|
|
|
|
03-06-2002, 11:01 AM
|
#3 |
|
Confirmed User
Join Date: Feb 2002
Location: Toronto, ON
Posts: 962
|
Meta Tags are read and complied with voluntarily. If you're trying to stop a bot with rogue intentions, this will not do it.
__________________
SIG TOO BIG! Maximum 120x60 button and no more than 3 text lines of DEFAULT SIZE and COLOR. Unless your sig is for a GFY top banner sponsor, then you may use a 624x80 instead of a 120x60. |
|
|
|
|
03-06-2002, 11:15 AM
|
#4 |
|
Confirmed User
Join Date: Feb 2001
Location: atlanta, GA
Posts: 6,432
|
meta tags seem useless nowadays
|
|
|
|
|
03-06-2002, 11:16 AM
|
#5 |
|
aka K-Man
Industry Role:
Join Date: Oct 2001
Location: The Gutter
Posts: 29,292
|
how can a spider index your members section (SE spider that is) if it is pword protected??
__________________
Crypto HODLr Crypto mining Angel investor |
|
|
|
|
03-06-2002, 12:14 PM
|
#6 |
|
There can be only one
Industry Role:
Join Date: Aug 2001
Location: Somewhere else
Posts: 39,075
|
I use both the metas as well as the robots.txt file AND robot.txt file (without the "s") to cover some older backward shit... just to make sure I got 'em all covered.
|
|
|
|
|
03-06-2002, 12:25 PM
|
#7 |
|
Confirmed User
Join Date: Feb 2002
Location: Seattle, WA
Posts: 1,527
|
i was thinking of serving pages as .asp extensions
the bots don't reconize this and don't know how to ripp the content via links when they hit an asp what do you think? |
|
|
|
