I have a huge list of urls, need app to remove duplicates

**MickeyG** · 07-15-2004, 12:32 PM

import them into a database with a index field then you can export to a clean list.

**QualityMpegs** · 07-15-2004, 12:40 PM

I'm sure somebody here could program that for real cheap for ya

**WiredGuy** · 07-15-2004, 12:42 PM

Import it into MS Access, keep distinct rows and you're done.
WG

**colpanic** · 07-15-2004, 12:45 PM

or if you are in some sort of Unix environment (or Linux)...

cat urls.txt | sort | uniq > uniqueurls.txt

**rowan** · 07-15-2004, 12:47 PM

Originally posted by colpanic
or if you are in some sort of Unix environment (or Linux)...

cat urls.txt | sort | uniq > uniqueurls.txt

No need for uniq or cat, you can use sort -u

sort -u urls.txt > uniqueurls.txt

**jact** · 07-15-2004, 12:48 PM

Originally posted by rowan
No need for uniq or cat, you can use sort -u

sort -u urls.txt > uniqueurls.txt

Bingo, problem solved.

**RFlagg** · 07-15-2004, 12:49 PM

Free Bookmark (favorites) manager

found on google in approx 23 secs...

I'm starting to think GFY stands for "Googled For You"

**VeriSexy** · 07-15-2004, 12:55 PM

Thanks guys, found a free program called raptor and worked great

**Fukeneh** · 07-15-2004, 01:01 PM

learn to use a shell. its an amazing tool.

**budz** · 07-15-2004, 01:03 PM

$sliced = explode("\n", $pizza);
$new_array = array_unique($sliced);
echo implode("\n",$new_array);

Comment