|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
05-10-2009, 05:56 AM
|
#1 |
|
Confirmed User
Join Date: Nov 2003
Location: 237 619 975
Posts: 8,944
|
Robots.txt VS meta robots INDEX
I have a robots text that disables the entire site for indexing
# Make changes for all web spiders User-agent: * Disallow: / but I have robots meta data at the top of each page saying to index it. Which will happen? I don't want it to get indexed but I dont want to disable the plugin adding the robots meta data either. Which one of these rules will trump the other? Thanks in advance - and please don't guess.
__________________
|
|
|
|
05-10-2009, 06:20 AM
|
#2 |
|
Confirmed User
Join Date: Nov 2003
Location: 237 619 975
Posts: 8,944
|
come on guys, I know all the smart people are working Sunday morning. Hook me up with the info please.
__________________
|
|
|
|
|
05-10-2009, 06:24 AM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Dec 2002
Posts: 1,605
|
robots.txt
|
|
|
|
|
05-10-2009, 08:52 AM
|
#4 |
|
there's no $$$ in porn
Industry Role:
Join Date: Jul 2005
Location: icq: 195./568.-230 (btw: not getting offline msgs)
Posts: 33,063
|
robots.txt
think about it for a sec  |
|
|
|
|
05-10-2009, 08:54 AM
|
#5 |
|
<&(©¿©)&>
Industry Role:
Join Date: Jul 2002
Location: Chicago
Posts: 47,882
|
they should follow robots.txt first
__________________
Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager  Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager |
|
|
|
|
05-10-2009, 09:05 AM
|
#6 |
|
Too lazy to set a custom title
Join Date: Nov 2006
Posts: 16,532
|
I have no idea
|
|
|
|
|
05-10-2009, 09:09 AM
|
#7 |
|
Confirmed User
Industry Role:
Join Date: Sep 2004
Location: Montrealquebecanada
Posts: 5,500
|
robots.txt is looked for first, and followed - so it won't bother hitting the metas if you no-index the site.
Yahoo! will take forever to index and list you if you don't have a robots.txt, for example, even if you have a robots meta tag.
__________________
 YOU Are Industry News! Press Releases: pr[at]payoutmag.com Facebook: Payout Magazine! Facebook: MIKEB! ICQ: 248843947 Skype: Mediaguy1 |
|
|
|
