robots.txt - GoFuckYourself.com

alex79 · 12-15-2004, 01:51 PM

i want that the spiders don't look automaticaly for domain.com/robots.txt file.. i DON"T want to block spiders.. i just don't want they looking for robots.txt.. how can i do that?

also, it is possible to change the robots.txt location from meta tags? for example to put robots.txt file in domain.com/xxx/spiders.txt file and tell spiders that there is the robots file

European Lee · 12-15-2004, 01:54 PM

Why woud you want to stop them looking for one?

Just dont upload one, if there isnt one on the server, it will return a 404 error and youll at least know your site is being spidered.

Just a thought.

Regards,

Lee

Evil1 · 12-15-2004, 01:55 PM

Why do the french always have to cause problems?

BackToMine · 12-15-2004, 01:58 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

DutchTeenCash · 12-15-2004, 02:01 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

Jim Adbrite · 12-15-2004, 02:02 PM

why would you want to do this?

alex79 · 12-15-2004, 02:07 PM

Quote:

Originally posted by Jim Adbrite
why would you want to do this?

becouse i want to use .httacces and php, and when the spiders will looking for robots.txt they will be redirected through httacces to a php file that will make a query to a mysql database and will slowdown my server .. that's why

macho · 12-15-2004, 02:10 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

Jim Adbrite · 12-15-2004, 02:11 PM

so make your script not be called when someone tries to access robots.txt

TheSwed · 12-15-2004, 02:12 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

*lmao*

European Lee · 12-15-2004, 02:14 PM

Quote:

Originally posted by Jim Adbrite
so make your script not be called when someone tries to access robots.txt

Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale

Regards,

Lee

Kevsh · 12-15-2004, 02:18 PM

Quote:

Originally posted by alex79
... will make a query to a mysql database and will slowdown my server .. that's why

Monsieur, how many times do you think robots look for robots.txt in a given day? And are you aware that MySql can handle multiple queries?

... unless of course you are running the site on a 386 with 2mb of RAM, I can't see any situation that should cause a (noticeable) slowdown on your server.

European Lee · 12-15-2004, 02:19 PM

Quote:

Originally posted by Kevsh
I can't see any situation that should cause a (noticeable) slowdown on your server.

I can

Regards,

Lee

Dalai lama · 12-15-2004, 02:20 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

alex79 · 12-15-2004, 02:25 PM

Quote:

Originally posted by European Lee
Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale

Regards,

Lee

good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?

European Lee · 12-15-2004, 02:29 PM

Quote:

Originally posted by alex79
good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?

No idea.

All i know is it can be done, we usually ask our host to set that kind of stuff up.

Regards,

Lee

12-15-2004, 01:51 PM	#1
alex79 Confirmed User Join Date: Jun 2002 Location: france Posts: 996	robots.txt i want that the spiders don't look automaticaly for domain.com/robots.txt file.. i DON"T want to block spiders.. i just don't want they looking for robots.txt.. how can i do that? also, it is possible to change the robots.txt location from meta tags? for example to put robots.txt file in domain.com/xxx/spiders.txt file and tell spiders that there is the robots file

12-15-2004, 01:54 PM	#2
European Lee Confirmed User Join Date: Dec 2002 Location: Daytona Beach Posts: 7,133	Why woud you want to stop them looking for one? Just dont upload one, if there isnt one on the server, it will return a 404 error and youll at least know your site is being spidered. Just a thought. Regards, Lee __________________ YOUR TRAFFIC IS SAFE WITH US - CONDOM CASH PUTTING THE GAY ADULT INDUSTRY ON YOUR PC!! AFFORDABLE 2257 COMPLIANT GAY + BI CONTENT

12-15-2004, 01:55 PM	#3
Evil1 Confirmed User Industry Role: Join Date: Mar 2002 Posts: 3,893	Why do the french always have to cause problems? __________________

12-15-2004, 02:02 PM	#6
Jim Adbrite Registered User Join Date: Dec 2004 Location: San Francisco Posts: 20	why would you want to do this? __________________ Sell text ads on your site! ICQ: 190218261

12-15-2004, 02:11 PM	#9
Jim Adbrite Registered User Join Date: Dec 2004 Location: San Francisco Posts: 20	so make your script not be called when someone tries to access robots.txt __________________ Sell text ads on your site! ICQ: 190218261