![]() |
Robots.txt
What do YOU have in your robots file?
|
Mine is pretty small at the moment. Had better ones in the past, not sure where they went though.
--- User-Agent: GetRight Disallow: / User-Agent: Go!Zilla Disallow: / User-Agent: Teleport Disallow: / |
User-agent: URL2File
Disallow: / User-agent: TE Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Offline Explorer Disallow: / User-agent: WebSauger Disallow: / User-agent: Mister PiX Disallow: / User-agent: Internet Ninja Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Teleport Disallow: / User-agent: asterias Disallow: / User-agent: wget Disallow: / User-agent: Web Downloader Disallow: / User-agent: nearsite Disallow: / User-agent: netattache Disallow: / User-agent: sitesnagger Disallow: / User-agent: webwhacker Disallow: / User-agent: digout4u Disallow: / User-agent: extractorpro Disallow: / User-agent: go-ahead-got-it Disallow: / User-agent: HTTPClient Disallow: / User-agent: TovekTools Web Indexer Disallow: / User-agent: LinkWalker Disallow: / User-agent: NEWT ActiveX Disallow: / User-agent: GetRight Disallow: / User-agent: WebZIP Disallow: / User-agent: WebReaper Disallow: / User-agent: HTTrack Disallow: / |
Um... Do you really think any of those site-sucking programs pay attention to robots.txt directives?
|
<H1>COCK SMOKERS!!</H1>
|
All times are GMT -7. The time now is 05:16 AM. |
Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123