If I don't want the SE bots going into a directory, let's say whatever.com/members, what would I put in my robots.txt? Thanks
-
-
### BEGIN FILE ###
#
# allow-all
#
#
# The use of robots or other automated means to access this site
# without express permission is strictly prohibited.
# Notwithstanding the foregoing, we may permit automated access to
# access certain of these pages but soley for the limited purpose of
# including content in publicly available search engines. Any other
# use of robots or failure to obey the robots exclusion standards set
# forth at <http://www.robotstxt.org/wc/exclusion.html> is strictly
# prohibited and will result in me beating your ass.
#
User-agent: *
Disallow: /members
Disallow: /working
User-agent: grub-client
User-agent: grub
User-agent: looksmart
User-agent: WebZip
User-agent: b2w/0.1
User-agent: copernic
Disallow: /
### END FILE ### -
google.com/bot.html
Comment
-
Doesn't anyone use google before posting here?
Disallow: /directory_name/
see:
http://www.searchengineworld.com/rob...s_tutorial.htm
for more detail.
icq:159548293Comment
-
why would they? someone else will do it for themOriginally posted by gornyhuy
Doesn't anyone use google before posting here?AIM - X8U
ICQ - 195301080Comment
-
Thanks
sorry for being lazyComment
-
just change your bots filecat cash* | tr -d '\r' | tr -s '/n' > moneyComment
-
Google bitch, google.Comment
-
that's goodOriginally posted by jimthefiend
is strictly
# prohibited and will result in me beating your ass.
Custom large/complex systems programing.<br>ASP.Net, C#, XML, MS SQL, WebServices(SOAP).<br>ICQ: 235719545Comment
Comment