DamianJ

I hate the daily mail with a passion, however, their robots' text is the best job ad ever:

# Robots.txt for http://www.dailymail.co.uk/
# All robots will spider the domain

# Begin standard rules

# Meltwater block
User-agent: Meltwater
Disallow: /

# Apply rules to all user agents updated 15/01/09
# Added Sitemaps.xml

User-agent: *

Disallow: /*redirect.php?

Disallow: /femail/article-1292332/Return-anarchy-sea-After-talented-pupils-died-riotous-end-exam-parties-Newquay-summer-changed.html

Disallow: /*reportAbuseInComment.html?
Disallow: /*search.html?searchPhrase=$
Disallow: /*search.html?pageOffset
Disallow: /*previousThread.html
Disallow: /*createThread.html
Disallow: /*emailArticle.html$
Disallow: /*nextThread.html
Disallow: /*readLater.html$
Disallow: /*myStories.html$
Disallow: /*search.html?s=y&authornamef=
Disallow: /*logout?redirectPath=
Disallow: /*login?redirectPath=
Disallow: /*createThread.html
Disallow: /*reportAbuse.html
Disallow: /*refer_product.php?
Disallow: /*startIndex=
Disallow: /*?pageSize
Disallow: /*?start=
Disallow: /SITE=DM/
Disallow: /js
Disallow: /*debateUserSearch.html
Disallow: /*debateSearchResults.html
Disallow: /*debateTagSearch.html
Disallow: /*textbased/channel
Disallow: /*goto.php?
Disallow: /*?printingPage=true$
Disallow: /tvshowbiz/tvlistings/
Disallow: /home/ireland/
Disallow: /home/scotland/

# August 12th, MailOnline are looking for a talented SEO Manager so if you found this then you're the kind of techie we need!
# Send your CV to holly dot ward at mailonline dot co dot uk

# Begin standard rules
# Apply rules to all user agents updated 08/06/08
ACAP-crawler: *
# User-agent: *
ACAP-disallow-crawl: /*search.html?searchPhrase=$
# Disallow: /*search.html?searchPhrase=$
ACAP-disallow-crawl: /*search.html?pageOffset
# Disallow: /*search.html?pageOffset
ACAP-disallow-crawl: /*previousThread.html$
# Disallow: /*previousThread.html$
ACAP-disallow-crawl: /*createThread.html$
# Disallow: /*createThread.html$
ACAP-disallow-crawl: /*emailArticle.html$
# Disallow: /*emailArticle.html$
ACAP-disallow-crawl: /*nextThread.html$
# Disallow: /*nextThread.html$
ACAP-disallow-crawl: /*readLater.html$
# Disallow: /*readLater.html$
ACAP-disallow-crawl: /*myStories.html$
# Disallow: /*myStories.html$
ACAP-disallow-crawl: /*search.html?s=y&authornamef=
# Disallow: /*search.html?s=y&authornamef=
ACAP-disallow-crawl: /*logout?redirectPath=
# Disallow: /*logout?redirectPath=
ACAP-disallow-crawl: /*login?redirectPath=
# Disallow: /*login?redirectPath=
ACAP-disallow-crawl: /*createThread.html
# Disallow: /*createThread.html
ACAP-disallow-crawl: /*reportAbuse.html
# Disallow: /*reportAbuse.html
ACAP-disallow-crawl: /*refer_product.php?
# Disallow: /*refer_product.php?
ACAP-disallow-crawl: /*startIndex=
# Disallow: /*startIndex=
ACAP-disallow-crawl: /*?pageSize
# Disallow: /*?pageSize
ACAP-disallow-crawl: /*?start=
# Disallow: /*?start=
ACAP-disallow-crawl: /SITE=DM/
# Disallow: /SITE=DM/
ACAP-disallow-crawl: /js
# Disallow: /js
ACAP-disallow-crawl: /*debateUserSearch.html
# Disallow: /*debateUserSearch.html
ACAP-disallow-crawl: /*debateSearchResults.html
# Disallow: /*debateSearchResults.html
ACAP-disallow-crawl: /*debateTagSearch.html
# Disallow: /*debateTagSearch.html
ACAP-disallow-crawl: /*textbased/channel
# Disallow: /*textbased/channel
ACAP-disallow-crawl: /*goto.php?
# Disallow: /*goto.php?
ACAP-disallow-crawl: /*reportAbuseInComment.html?
# Disallow: /*reportAbuseInComment.html?

# Sitemap files
Sitemap: http://www.dailymail.co.uk/newssitemap.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2010.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2009.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2008.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2007.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2006.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2005.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2004.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2003.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2002.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2001.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~2000.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1999.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1998.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1997.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1996.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1995.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1994.xml
Sitemap: http://www.dailymail.co.uk/sitemap-a...-year~1993.xml
Sitemap: http://www.dailymail.co.uk/videositemap.xml

http://www.dailymail.co.uk/robots.txt