|
OK im bored so...
Freshbot will not crawl anything but indexes, html and text, deepbot will not crawl links with more then 2-3 variables in them (I.E. bum.php?ass=1&yes=no&butt=fat&turd=1) or urls with session IDs.
BONUS FACT: Inkatomi's spider WILL crawl things like vBulletin and other scripts with sessions.
__________________
Amen
|