UNIX question -- how to sort and remove duplicates?
 

I have a huge file that's full of tons of duplicate lines -- I want to remove all the duplicate lines from the file.

I did a "man sort" and I didn't see any option to remove/ignore duplicate lines --

Anyone know how to do it? I'm sure it's easy...

cat filename.txt|sort|uniq > newfilename.txt

Awesome I did "info sort"

I found it...

"
`-u'
`--unique'
Normally, output only the first of a sequence of lines that compare
equal. For the `--check' (`-c') option, check that no pair of
consecutive lines compares equal."

Cool I like that better... I figured there was a way to pipe it instead of using -o filename

uniq filename newfilename

sort < in.txt | uniq > out.txt

Trying to remove all the duplicate emails from a spam list? :winkwink:

J/k dude :1orglaugh

thats exactly what I was going to say ;p

you know there is some great spam list software out there that can handle multiple millions :Graucho

If you want to know what I was really doing... I had some code for one of my scripts to periodically remove dead trades -- using a formula -- blah -- anyway the code was buggy and the result was lots of duplicate entries over the course of many months the script still worked fine but the database was getting full of useless records... basically I fixed my code -- but then I needed to repair the database by removing the duplicates...

Anyway thanks for the help... maybe some day I will try some kind of email marketing but so far I've been too lazy to do it...