UNIX question -- how to sort and remove duplicates? - GoFuckYourself.com

TheFLY · 07-25-2002, 10:39 AM

I have a huge file that's full of tons of duplicate lines -- I want to remove all the duplicate lines from the file.

I did a "man sort" and I didn't see any option to remove/ignore duplicate lines --

Anyone know how to do it? I'm sure it's easy...

nuclei · 07-25-2002, 10:44 AM

cat filename.txt|sort|uniq > newfilename.txt

TheFLY · 07-25-2002, 10:45 AM

Awesome I did "info sort"

I found it...

"
`-u'
`--unique'
Normally, output only the first of a sequence of lines that compare
equal. For the `--check' (`-c') option, check that no pair of
consecutive lines compares equal."

TheFLY · 07-25-2002, 10:46 AM

Quote:

Originally posted by nuclei
cat filename.txt|sort|uniq > newfilename.txt

Cool I like that better... I figured there was a way to pipe it instead of using -o filename

DennisS · 07-25-2002, 10:48 AM

uniq filename newfilename

Big E · 07-26-2002, 11:28 AM

sort < in.txt | uniq > out.txt

Dreamman010 · 07-26-2002, 01:09 PM

Quote:

Originally posted by TheFLY
I have a huge file that's full of tons of duplicate lines -- I want to remove all the duplicate lines from the file.

I did a "man sort" and I didn't see any option to remove/ignore duplicate lines --

Anyone know how to do it? I'm sure it's easy...

Trying to remove all the duplicate emails from a spam list?

J/k dude

DrGuile · 07-26-2002, 01:17 PM

Quote:

Originally posted by Dreamman010

Trying to remove all the duplicate emails from a spam list?

J/k dude

thats exactly what I was going to say ;p

pr0 · 07-26-2002, 04:13 PM

you know there is some great spam list software out there that can handle multiple millions

TheFLY · 07-26-2002, 07:46 PM

If you want to know what I was really doing... I had some code for one of my scripts to periodically remove dead trades -- using a formula -- blah -- anyway the code was buggy and the result was lots of duplicate entries over the course of many months the script still worked fine but the database was getting full of useless records... basically I fixed my code -- but then I needed to repair the database by removing the duplicates...

Anyway thanks for the help... maybe some day I will try some kind of email marketing but so far I've been too lazy to do it...

07-25-2002, 10:39 AM	#1
TheFLY So Fucking Banned Join Date: Jan 2001 Location: http://www.thefly.net/ --- Quit your job and live off steady traffic. Posts: 11,856	UNIX question -- how to sort and remove duplicates? I have a huge file that's full of tons of duplicate lines -- I want to remove all the duplicate lines from the file. I did a "man sort" and I didn't see any option to remove/ignore duplicate lines -- Anyone know how to do it? I'm sure it's easy...

07-25-2002, 10:44 AM	#2
nuclei old school fart Industry Role: Join Date: May 2001 Location: Florida Posts: 1,015	cat filename.txt\|sort\|uniq > newfilename.txt __________________ The next generation of SEO

07-25-2002, 10:48 AM	#5
DennisS Confirmed User Join Date: Jan 2002 Location: N.Y.C Posts: 265	uniq filename newfilename __________________

07-26-2002, 04:13 PM	#9
pr0 rockin tha trailerpark Industry Role: Join Date: May 2001 Location: ~Coastal~ Posts: 23,088	you know there is some great spam list software out there that can handle multiple millions __________________ __________ Loadedca$h - get sum! - Revengebucks - mmm rebills! - webair (gotz sErVrz)

07-25-2002, 10:45 AM	#3
TheFLY So Fucking Banned Join Date: Jan 2001 Location: http://www.thefly.net/ --- Quit your job and live off steady traffic. Posts: 11,856	Awesome I did "info sort" I found it... " `-u' `--unique' Normally, output only the first of a sequence of lines that compare equal. For the `--check' (`-c') option, check that no pair of consecutive lines compares equal."

07-26-2002, 11:28 AM	#6
Big E Registered User Industry Role: Join Date: Mar 2002 Location: San Diego, CA Posts: 935	sort < in.txt \| uniq > out.txt

07-26-2002, 07:46 PM	#10
TheFLY So Fucking Banned Join Date: Jan 2001 Location: http://www.thefly.net/ --- Quit your job and live off steady traffic. Posts: 11,856	If you want to know what I was really doing... I had some code for one of my scripts to periodically remove dead trades -- using a formula -- blah -- anyway the code was buggy and the result was lots of duplicate entries over the course of many months the script still worked fine but the database was getting full of useless records... basically I fixed my code -- but then I needed to repair the database by removing the duplicates... Anyway thanks for the help... maybe some day I will try some kind of email marketing but so far I've been too lazy to do it...