GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - robots.txt (https://gfy.com/showthread.php?t=403312)

alex79

12-15-2004 01:51 PM

robots.txt

i want that the spiders don't look automaticaly for domain.com/robots.txt file.. i DON"T want to block spiders.. i just don't want they looking for robots.txt.. how can i do that?

also, it is possible to change the robots.txt location from meta tags? for example to put robots.txt file in domain.com/xxx/spiders.txt file and tell spiders that there is the robots file

European Lee

12-15-2004 01:54 PM

Why woud you want to stop them looking for one?

Just dont upload one, if there isnt one on the server, it will return a 404 error and youll at least know your site is being spidered.

Just a thought.

Regards,

Lee

Evil1

12-15-2004 01:55 PM

Why do the french always have to cause problems?

BackToMine

12-15-2004 01:58 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

:1orglaugh :1orglaugh :1orglaugh :1orglaugh :1orglaugh :1orglaugh

DutchTeenCash

12-15-2004 02:01 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

:1orglaugh

Jim Adbrite

12-15-2004 02:02 PM

why would you want to do this?

alex79

12-15-2004 02:07 PM

Quote:

Originally posted by Jim Adbrite
why would you want to do this?

becouse i want to use .httacces and php, and when the spiders will looking for robots.txt they will be redirected through httacces to a php file that will make a query to a mysql database and will slowdown my server .. that's why

macho

12-15-2004 02:10 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

:1orglaugh :thumbsup

Jim Adbrite

12-15-2004 02:11 PM

so make your script not be called when someone tries to access robots.txt

TheSwed

12-15-2004 02:12 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

:1orglaugh *lmao*

European Lee

12-15-2004 02:14 PM

Quote:

Originally posted by Jim Adbrite
so make your script not be called when someone tries to access robots.txt

Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale :Graucho

Regards,

Lee

Kevsh

12-15-2004 02:18 PM

Quote:

Originally posted by alex79
... will make a query to a mysql database and will slowdown my server .. that's why

Monsieur, how many times do you think robots look for robots.txt in a given day? And are you aware that MySql can handle multiple queries?

... unless of course you are running the site on a 386 with 2mb of RAM, I can't see any situation that should cause a (noticeable) slowdown on your server.

European Lee

12-15-2004 02:19 PM

Quote:

Originally posted by Kevsh
I can't see any situation that should cause a (noticeable) slowdown on your server.

I can :Graucho

Regards,

Lee

Dalai lama

12-15-2004 02:20 PM

Quote:

Originally posted by Evil1
Why do the french always have to cause problems?

:1orglaugh :1orglaugh :1orglaugh :1orglaugh

alex79

12-15-2004 02:25 PM

Quote:

Originally posted by European Lee
Actually, if the guy is trying to do what i think he is trying to do that would be impossible.

Hmm he could however use an apache module redirect for text files if he is on a unix based server to accomplish what he wants.

I just hope he is doing what i beleive he is trying to do and if so, if he is going to have this script available for resale :Graucho

Regards,

Lee

good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?

European Lee

12-15-2004 02:29 PM

Quote:

Originally posted by alex79
good ideea to redirect all the .txt files from that domain to a blank page.. can u tell me please how can i do that from apache config file?

No idea.

All i know is it can be done, we usually ask our host to set that kind of stuff up.

Regards,

Lee

All times are GMT -7. The time now is 09:28 AM.