What is the best way to mass download an entire site? - GoFuckYourself.com

DWB · 06-13-2007, 07:37 PM

I'd like to be able to log-in, and have this download every file with as little effort as possible.

I know this is done everyday, what is the software called?

GooSearch · 06-13-2007, 07:39 PM

frontpage :p

ur killin me ya know this

HighSociety · 06-13-2007, 07:57 PM

this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry

DWB · 06-13-2007, 07:59 PM

Quote:

Originally Posted by made2ordervideos

this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry

Ohhhhh. I like the sound of that.

You have one to sell?

GooSearch · 06-13-2007, 08:01 PM

Quote:

Originally Posted by made2ordervideos

this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry

we been beat up for a few days.. i'll take one too

HighSociety · 06-13-2007, 08:01 PM

the virus or the program that downloads entire sites? I do know of a members area that has this in it. The program you can probably find on google

jact · 06-13-2007, 08:02 PM

wget will do it.

tony286 · 06-13-2007, 08:03 PM

Quote:

Originally Posted by made2ordervideos

this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry

Thats a great idea actually, some scum bag ripped one of my sites and put it up on a bt.

HighSociety · 06-13-2007, 08:05 PM

It works well

psili · 06-13-2007, 08:15 PM

Quote:

Originally Posted by jact

wget will do it.

Never tried it, but might have to agree with "wget" or the ton of graphical clients that crawl and download from sites.

You find a site that still uses basic authentication and offers video downloads that aren't DRM'd, you could pretty much own their shit with free software out there.

Fuck it. Hit up a password sharing site, find all that use basic auth who you have a grudge against, and work their shit.

Using basic auth and allowing free downloads is like dropping a wallet on the street. -- You find it. It's yours.

geedub · 06-13-2007, 08:18 PM

wget fo shizzle

fluffygrrl · 06-13-2007, 08:18 PM

wget, it even has a windows port now.

ultimatebbwdotcom · 06-13-2007, 08:29 PM

Theres hundreds of progs that download entire sites - d/l limits and block lists cure most of the problems...... like the sound of the virus though, any more info?

camgenie · 06-13-2007, 08:40 PM

Quote:

Originally Posted by jact

wget will do it.

wget -r [site]

or

man wget

RedShoe · 06-13-2007, 08:54 PM

I'd like to wget a fuckin boot up their ass!

RawAlex · 06-13-2007, 08:58 PM

You can lead wget down the garden path and fill their disk space up pretty quickly with junk - apparently well formed jpeg files that are really large and don't contain images. You can use links that normal surfers won't see to do this (1x1 pixels linked, example, near the top of each page). Track whoever hits those links, and ban the IP. It will certainly slow them down.

Good luck.

DWB · 06-13-2007, 09:35 PM

Quote:

Originally Posted by RawAlex

You can lead wget down the garden path and fill their disk space up pretty quickly with junk - apparently well formed jpeg files that are really large and don't contain images. You can use links that normal surfers won't see to do this (1x1 pixels linked, example, near the top of each page). Track whoever hits those links, and ban the IP. It will certainly slow them down.

Good luck.

That's pretty slick.

GrouchyAdmin · 06-13-2007, 10:15 PM

Wget, of course. wget can be set to ignore robots.txt, so a good thing to do is make a hidden directory full of crap, say /d/, and just fill it with a bunch of random data, and have your analyzer watch for that. There are many programs written for UNIX that can even realtime watch your apache logs and fire off an email with the person's IP (and if using normal authentication methods, their username).

uno · 06-13-2007, 10:29 PM

Quote:

Originally Posted by jact

wget will do it.

jact knows his shit.

Shoehorn! · 06-13-2007, 10:43 PM

Quote:

Originally Posted by made2ordervideos

this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry

That is an awesome idea.

bobby666 · 06-13-2007, 10:57 PM

i used webzip for mirroring or downloading websites when i was a pc user

06-13-2007, 07:37 PM	#1
DWB Registered User Industry Role: Join Date: Jul 2003 Location: Encrypted. Access denied. Posts: 31,779	What is the best way to mass download an entire site? I'd like to be able to log-in, and have this download every file with as little effort as possible. I know this is done everyday, what is the software called?

06-13-2007, 07:39 PM	#2
GooSearch Confirmed User Join Date: Oct 2006 Location: FLA Posts: 2,538	frontpage :p ur killin me ya know this __________________ GooSearch Real Text Ads Without The Drama Coming Back Bigger and Better Then Ever

06-13-2007, 07:57 PM	#3
HighSociety Confirmed User Join Date: Jun 2005 Posts: 1,786	this is why you put a virus in your members area that can't be seen but will be downloaded when a program is used to suck your site dry __________________ http://www.highsociety.com http://www.playgirl.com http://www.cheri.com Jonathan "JC" Maldini ICQ: 223 643

06-13-2007, 08:01 PM	#6
HighSociety Confirmed User Join Date: Jun 2005 Posts: 1,786	the virus or the program that downloads entire sites? I do know of a members area that has this in it. The program you can probably find on google __________________ http://www.highsociety.com http://www.playgirl.com http://www.cheri.com Jonathan "JC" Maldini ICQ: 223 643

06-13-2007, 08:02 PM	#7
jact Confirmed User Join Date: Sep 2002 Location: Oakville, Canada Posts: 9,134	wget will do it. __________________ Free agent

06-13-2007, 08:05 PM	#9
HighSociety Confirmed User Join Date: Jun 2005 Posts: 1,786	It works well __________________ http://www.highsociety.com http://www.playgirl.com http://www.cheri.com Jonathan "JC" Maldini ICQ: 223 643

06-13-2007, 08:18 PM	#11
geedub Confirmed User Industry Role: Join Date: Jun 2005 Location: concrete jungle Posts: 3,489	wget fo shizzle __________________ Reliable web host that actually cares, tell em geedub sent ya. Vacares

06-13-2007, 08:18 PM	#12
fluffygrrl So Fucking Banned Join Date: May 2006 Posts: 2,187	wget, it even has a windows port now.

06-13-2007, 08:29 PM	#13
ultimatebbwdotcom Confirmed User Join Date: Mar 2006 Posts: 591	Theres hundreds of progs that download entire sites - d/l limits and block lists cure most of the problems...... like the sound of the virus though, any more info?

06-13-2007, 08:54 PM	#15
RedShoe 赤い靴 call me 202-456-1111 Industry Role: Join Date: Feb 2001 Location: The Valley Posts: 14,831	I'd like to wget a fuckin boot up their ass! __________________ SPECIALTY COSTUMES • PROPS • FX Superheroes • Monsters • Robots PM for details For any manufacturing needs. Adult or otherwise. aka BonsHigh on Insta Bonsai weed plants

06-13-2007, 08:58 PM	#16
RawAlex So Fucking Banned Join Date: Oct 2003 Location: In a house. Posts: 9,465	You can lead wget down the garden path and fill their disk space up pretty quickly with junk - apparently well formed jpeg files that are really large and don't contain images. You can use links that normal surfers won't see to do this (1x1 pixels linked, example, near the top of each page). Track whoever hits those links, and ban the IP. It will certainly slow them down. Good luck.

06-13-2007, 10:15 PM	#18
GrouchyAdmin Now choke yourself! Industry Role: Join Date: Apr 2006 Posts: 12,085	Wget, of course. wget can be set to ignore robots.txt, so a good thing to do is make a hidden directory full of crap, say /d/, and just fill it with a bunch of random data, and have your analyzer watch for that. There are many programs written for UNIX that can even realtime watch your apache logs and fire off an email with the person's IP (and if using normal authentication methods, their username). __________________ AIM: GrouchyGfy

06-13-2007, 10:57 PM	#21
bobby666 boots are my religion Join Date: Nov 2005 Location: Heart of europe Posts: 21,765	i used webzip for mirroring or downloading websites when i was a pc user __________________