UNIX question -- how to sort and remove duplicates?

**nuclei** · 07-25-2002, 09:44 AM

cat filename.txt|sort|uniq > newfilename.txt

**TheFLY** · 07-25-2002, 09:45 AM

Awesome I did "info sort"

I found it...

"
`-u'
`--unique'
Normally, output only the first of a sequence of lines that compare
equal. For the `--check' (`-c') option, check that no pair of
consecutive lines compares equal."

**TheFLY** · 07-25-2002, 09:46 AM

Originally posted by nuclei
cat filename.txt|sort|uniq > newfilename.txt

Cool I like that better... I figured there was a way to pipe it instead of using -o filename

**DennisS** · 07-25-2002, 09:48 AM

uniq filename newfilename

**Big E** · 07-26-2002, 10:28 AM

sort < in.txt | uniq > out.txt

**Dreamman010** · 07-26-2002, 12:09 PM

Originally posted by TheFLY
I have a huge file that's full of tons of duplicate lines -- I want to remove all the duplicate lines from the file.

I did a "man sort" and I didn't see any option to remove/ignore duplicate lines --

Anyone know how to do it? I'm sure it's easy...

Trying to remove all the duplicate emails from a spam list?

J/k dude

**DrGuile** · 07-26-2002, 12:17 PM

Originally posted by Dreamman010

Trying to remove all the duplicate emails from a spam list?

J/k dude

thats exactly what I was going to say ;p

**pr0** · 07-26-2002, 03:13 PM

you know there is some great spam list software out there that can handle multiple millions

**TheFLY** · 07-26-2002, 06:46 PM

If you want to know what I was really doing... I had some code for one of my scripts to periodically remove dead trades -- using a formula -- blah -- anyway the code was buggy and the result was lots of duplicate entries over the course of many months the script still worked fine but the database was getting full of useless records... basically I fixed my code -- but then I needed to repair the database by removing the duplicates...

Anyway thanks for the help... maybe some day I will try some kind of email marketing but so far I've been too lazy to do it...

UNIX question -- how to sort and remove duplicates?

UNIX question -- how to sort and remove duplicates?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment