GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   Robots.txt (https://gfy.com/showthread.php?t=111929)

weth0ney 02-28-2003 10:36 AM

Robots.txt
 
What do YOU have in your robots file?

ZoiNk 02-28-2003 10:46 AM

Mine is pretty small at the moment. Had better ones in the past, not sure where they went though.

---

User-Agent: GetRight
Disallow: /

User-Agent: Go!Zilla
Disallow: /

User-Agent: Teleport
Disallow: /

extreme 02-28-2003 12:33 PM

User-agent: URL2File
Disallow: /
User-agent: TE
Disallow: /
User-agent: BaiDuSpider
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: Internet Ninja
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: asterias
Disallow: /
User-agent: wget
Disallow: /
User-agent: Web Downloader
Disallow: /
User-agent: nearsite
Disallow: /
User-agent: netattache
Disallow: /
User-agent: sitesnagger
Disallow: /
User-agent: webwhacker
Disallow: /
User-agent: digout4u
Disallow: /
User-agent: extractorpro
Disallow: /
User-agent: go-ahead-got-it
Disallow: /
User-agent: HTTPClient
Disallow: /
User-agent: TovekTools Web Indexer
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: NEWT ActiveX
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: HTTrack
Disallow: /

NetRodent 02-28-2003 12:50 PM

Um... Do you really think any of those site-sucking programs pay attention to robots.txt directives?

Manny The Tranny 03-29-2003 09:22 PM

<H1>COCK SMOKERS!!</H1>


All times are GMT -7. The time now is 05:16 AM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123