Thread: I have a huge list of urls, need app to remove duplicates
View Single Post
Old 07-15-2004, 01:47 PM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally posted by colpanic
or if you are in some sort of Unix environment (or Linux)...


cat urls.txt | sort | uniq > uniqueurls.txt
No need for uniq or cat, you can use sort -u

sort -u urls.txt > uniqueurls.txt
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote