Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Post New Thread Reply

Register GFY Rules Calendar Mark Forums Read
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
robots.txt robots.txt
Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed.

 
Thread Tools
Old 12-15-2004, 01:51 PM   #1
alex79
Confirmed User
 
Join Date: Jun 2002
Location: france
Posts: 996
robots.txt
i want that the spiders don't look automaticaly for domain.com/robots.txt file.. i DON"T want to block spiders.. i just don't want they looking for robots.txt.. how can i do that?

also, it is possible to change the robots.txt location from meta tags? for example to put robots.txt file in domain.com/xxx/spiders.txt file and tell spiders that there is the robots file
alex79 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 01:54 PM   #2
European Lee
Confirmed User
 
Join Date: Dec 2002
Location: Daytona Beach
Posts: 7,133
Why woud you want to stop them looking for one?

Just dont upload one, if there isnt one on the server, it will return a 404 error and youll at least know your site is being spidered.

Just a thought.

Regards,

Lee
European Lee is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 01:55 PM   #3
Evil1
Confirmed User
 
Industry Role:
Join Date: Mar 2002
Posts: 3,893
Why do the french always have to cause problems?
__________________
Evil1 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 01:58 PM   #4
BackToMine
Confirmed User
 
Join Date: Nov 2004
Posts: 541
Quote:
Originally posted by Evil1
Why do the french always have to cause problems?
BackToMine is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:01 PM   #5
DutchTeenCash
I like Dutch Girls
 
DutchTeenCash's Avatar
 
Join Date: Feb 2003
Location: dutchteencash.com
Posts: 21,684
Quote:
Originally posted by Evil1
Why do the french always have to cause problems?
DutchTeenCash is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:02 PM   #6
Jim Adbrite
Registered User
 
Join Date: Dec 2004
Location: San Francisco
Posts: 20
why would you want to do this?
__________________
Sell text ads on your site!
ICQ: 190218261
Jim Adbrite is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:07 PM   #7
alex79
Confirmed User
 
Join Date: Jun 2002
Location: france
Posts: 996
Quote:
Originally posted by Jim Adbrite
why would you want to do this?
becouse i want to use .httacces and php, and when the spiders will looking for robots.txt they will be redirected through httacces to a php file that will make a query to a mysql database and will slowdown my server .. that's why
alex79 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:10 PM   #8
macho
Confirmed User
 
Join Date: Nov 2001
Location: Internet
Posts: 2,875
Quote:
Originally posted by Evil1
Why do the french always have to cause problems?
__________________
GayPay X2 -- #1in the niche! $35 per sign! Up to 70% revenue share! http://www.gaypay.com
macho is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:11 PM   #9
Jim Adbrite
Registered User
 
Join Date: Dec 2004
Location: San Francisco
Posts: 20
so make your script not be called when someone tries to access robots.txt
__________________
Sell text ads on your site!
ICQ: 190218261
Jim Adbrite is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:12 PM   #10
TheSwed
Confirmed User
 
TheSwed's Avatar
 
Join Date: Feb 2004
Location: Sweden.
Posts: 3,483
Quote:
Originally posted by Evil1
Why do the french always have to cause problems?
*lmao*
TheSwed is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:14 PM   #11
European Lee
Confirmed User
 
Join Date: Dec 2002
Location: Daytona Beach
Posts: 7,133
Quote:
Originally posted by Jim Adbrite
so make your script not be called when someone tries to access robots.txt
Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale

Regards,

Lee
European Lee is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:18 PM   #12
Kevsh
Confirmed User
 
Join Date: Dec 2004
Location: TO
Posts: 8,619
Quote:
Originally posted by alex79
... will make a query to a mysql database and will slowdown my server .. that's why
Monsieur, how many times do you think robots look for robots.txt in a given day? And are you aware that MySql can handle multiple queries?

... unless of course you are running the site on a 386 with 2mb of RAM, I can't see any situation that should cause a (noticeable) slowdown on your server.
Kevsh is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:19 PM   #13
European Lee
Confirmed User
 
Join Date: Dec 2002
Location: Daytona Beach
Posts: 7,133
Quote:
Originally posted by Kevsh
I can't see any situation that should cause a (noticeable) slowdown on your server.
I can

Regards,

Lee
European Lee is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:20 PM   #14
Dalai lama
Strength and Honor
 
Join Date: Jul 2004
Location: Europe
Posts: 16,540
Quote:
Originally posted by Evil1
Why do the french always have to cause problems?
__________________

A program you can trust.
Gallerybooster Run multiply TGPs of 1 script
Dalai lama is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:25 PM   #15
alex79
Confirmed User
 
Join Date: Jun 2002
Location: france
Posts: 996
Quote:
Originally posted by European Lee
Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale

Regards,

Lee
good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?
alex79 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 12-15-2004, 02:29 PM   #16
European Lee
Confirmed User
 
Join Date: Dec 2002
Location: Daytona Beach
Posts: 7,133
Quote:
Originally posted by alex79
good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?
No idea.

All i know is it can be done, we usually ask our host to set that kind of stuff up.

Regards,

Lee
European Lee is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Post New Thread Reply
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
robots.txt robots.txt
« Previous Thread | Next Thread »

Bookmarks
Thread Tools



Advertising inquiries - marketing at gfy dot com

Contact Admin - Advertise - GFY Rules - Top

©2000-, AI Media Network Inc



Powered by vBulletin
Copyright © 2000- Jelsoft Enterprises Limited.