GoFuckYourself.com - Adult Webmaster Forum - I have a huge list of urls, need app to remove duplicates

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - I have a huge list of urls, need app to remove duplicates (https://gfy.com/showthread.php?t=326469)

VeriSexy

07-15-2004 01:30 PM

I have a huge list of urls, need app to remove duplicates

Any good free apps that will help remove duplicate urls? I have a list of thousands of urls.

MickeyG

07-15-2004 01:32 PM

import them into a database with a index field then you can export to a clean list.

QualityMpegs

07-15-2004 01:40 PM

I'm sure somebody here could program that for real cheap for ya

WiredGuy

07-15-2004 01:42 PM

Import it into MS Access, keep distinct rows and you're done.
WG

colpanic

07-15-2004 01:45 PM

or if you are in some sort of Unix environment (or Linux)...

cat urls.txt | sort | uniq > uniqueurls.txt

rowan

07-15-2004 01:47 PM

Quote:

Originally posted by colpanic
or if you are in some sort of Unix environment (or Linux)...

cat urls.txt | sort | uniq > uniqueurls.txt

No need for uniq or cat, you can use sort -u :)

sort -u urls.txt > uniqueurls.txt

jact	07-15-2004 01:48 PM

Quote:

Originally posted by rowan
No need for uniq or cat, you can use sort -u :)

sort -u urls.txt > uniqueurls.txt

Bingo, problem solved.

RFlagg

07-15-2004 01:49 PM

Free Bookmark (favorites) manager

found on google in approx 23 secs...

I'm starting to think GFY stands for "Googled For You"

:eyecrazy

VeriSexy

07-15-2004 01:55 PM

Thanks guys, found a free program called raptor and worked great :)

Fukeneh

07-15-2004 02:01 PM

learn to use a shell. its an amazing tool.

budz	07-15-2004 02:03 PM

$sliced = explode("\n", $pizza);
$new_array = array_unique($sliced);
echo implode("\n",$new_array);

All times are GMT -7. The time now is 11:21 PM.