robots.txt question... - GoFuckYourself.com

brizzad · 01-07-2004, 07:25 PM

If I don't want the SE bots going into a directory, let's say whatever.com/members, what would I put in my robots.txt? Thanks

jimthefiend · 01-07-2004, 07:26 PM

### BEGIN FILE ###
#
# allow-all
#
#
# The use of robots or other automated means to access this site
# without express permission is strictly prohibited.
# Notwithstanding the foregoing, we may permit automated access to
# access certain of these pages but soley for the limited purpose of
# including content in publicly available search engines. Any other
# use of robots or failure to obey the robots exclusion standards set
# forth at <http://www.robotstxt.org/wc/exclusion.html> is strictly
# prohibited and will result in me beating your ass.
#

User-agent: *
Disallow: /members
Disallow: /working
User-agent: grub-client
User-agent: grub
User-agent: looksmart
User-agent: WebZip
User-agent: b2w/0.1
User-agent: copernic
Disallow: /

### END FILE ###

Oracle Porn · 01-07-2004, 07:26 PM

google.com/bot.html

gornyhuy · 01-07-2004, 07:26 PM

Doesn't anyone use google before posting here?

Disallow: /directory_name/

see:
http://www.searchengineworld.com/rob...s_tutorial.htm

for more detail.

Fu-Q · 01-07-2004, 07:28 PM

Quote:

Originally posted by gornyhuy
Doesn't anyone use google before posting here?

why would they? someone else will do it for them

brizzad · 01-07-2004, 07:31 PM

Thanks

sorry for being lazy

- AFN - · 01-08-2004, 03:57 AM

just change your bots file

V_RocKs · 01-08-2004, 03:59 AM

Google bitch, google.

dotwind · 01-08-2004, 04:13 AM

Quote:

Originally posted by jimthefiend
is strictly
# prohibited and will result in me beating your ass.

that's good

01-07-2004, 07:25 PM	#1
brizzad holla Join Date: Jul 2003 Location: KFC Posts: 11,769	robots.txt question... If I don't want the SE bots going into a directory, let's say whatever.com/members, what would I put in my robots.txt? Thanks

01-07-2004, 07:26 PM	#3
Oracle Porn Affiliate Industry Role: Join Date: Oct 2002 Location: Icq: 94-399-723 Posts: 24,433	google.com/bot.html __________________

01-07-2004, 07:26 PM	#4
gornyhuy Chafed. Join Date: May 2002 Location: Face Down in Pussy Posts: 18,041	Doesn't anyone use google before posting here? Disallow: /directory_name/ see: http://www.searchengineworld.com/rob...s_tutorial.htm for more detail. __________________ icq:159548293

01-08-2004, 03:57 AM	#7
- AFN - Confirmed User Join Date: Jun 2003 Location: NYC Baaaabeee Posts: 3,101	just change your bots file __________________ cat cash* \| tr -d '\r' \| tr -s '/n' > money

01-07-2004, 07:26 PM	#2
jimthefiend So Fucking Banned Join Date: Oct 2003 Location: icq: 121189 Posts: 18,889	### BEGIN FILE ### # # allow-all # # # The use of robots or other automated means to access this site # without express permission is strictly prohibited. # Notwithstanding the foregoing, we may permit automated access to # access certain of these pages but soley for the limited purpose of # including content in publicly available search engines. Any other # use of robots or failure to obey the robots exclusion standards set # forth at <http://www.robotstxt.org/wc/exclusion.html> is strictly # prohibited and will result in me beating your ass. # User-agent: * Disallow: /members Disallow: /working User-agent: grub-client User-agent: grub User-agent: looksmart User-agent: WebZip User-agent: b2w/0.1 User-agent: copernic Disallow: / ### END FILE ###

01-07-2004, 07:31 PM	#6
brizzad holla Join Date: Jul 2003 Location: KFC Posts: 11,769	Thanks sorry for being lazy

01-08-2004, 03:59 AM	#8
V_RocKs Damn Right I Kiss Ass! Industry Role: Join Date: Dec 2003 Location: Cowtown, USA Posts: 32,409	Google bitch, google.