mailing list cleaner to rid duplicate emails **no pics inside** - GoFuckYourself.com

squeezeboobs · 02-23-2008, 03:19 PM

Help a girl out!!

Anyone heard of a software or script that would go through tons of txt or csv files I have with email addresses and combine them into one that takes out duplicates?

I have over 30,000 emails that I've collected over the years from different sites of mine and I want to combine them all into a new email list software that I have.

Any help is appreciated.

GrouchyAdmin · 02-23-2008, 03:22 PM

Excel can do it. Just import your data, sort, remove dupes.

You're likely stuck at a max of 65,000 records per sheet, though.

squeezeboobs · 02-23-2008, 03:32 PM

how can I remove dups?

I dont see anything to do that in the menus.

GrouchyAdmin · 02-23-2008, 03:33 PM

http://support.microsoft.com/kb/262277

squeezeboobs · 02-23-2008, 03:42 PM

ok I found this of the Microcrap site:

1. Select all the rows, including the column headers, in the list you want to filter.

ShowTip
Click the top left cell of the range, and then drag to the bottom right cell.
2. On the Data menu, point to Filter, and then click Advanced Filter.
3. In the Advanced Filter dialog box, click Filter the list, in place.
4. Select the Unique records only check box, and then click OK.

The filtered list is displayed and the duplicate rows are hidden.
5. On the Edit menu, click Office Clipboard.

The Clipboard task pane is displayed.
6. Make sure the filtered list is still selected, and then click Copy Copy button.

The filtered list is highlighted with bounding outlines and the selection appears as an item at the top of the Clipboard.
7. On the Data menu, point to Filter, and then click Show All.

The original list is re-displayed.
8. Press the DELETE key.

The original list is deleted.
9. In the Clipboard, click on the filtered list item.

The filtered list appears in the same location as the original list.

Linguist · 02-23-2008, 03:43 PM

Put them all into one file and then in Linux or Cygwin:

cat file.txt | sort | uniq > new_file.txt

(this would only work if you have an e-mail per line and nothing else, so if you have names, opt-in data, etc then it wouldn't work as easily)

squeezeboobs · 02-23-2008, 03:44 PM

GrouchyAdmin hey thx

squeezeboobs · 02-23-2008, 03:45 PM

Linguist whoa whoa now that's crazy. Who runs linux on the desktop.

now that's white and nerdy!

Linguist · 02-23-2008, 03:46 PM

http://cygwin.com ;)

GrouchyAdmin · 02-23-2008, 03:48 PM

Quote:

Originally Posted by squeezeboobs

GrouchyAdmin hey thx

No problem.

Quote:

Originally Posted by Linguist

http://cygwin.com ;)

Then you've got bigger issues. Some native Cygwin ports do/don't automatically distinct case between CR (unix) and CR+LF (dos). Not a good plan. You'd be better off using Perl than even awk in this case.

Linguist · 02-23-2008, 03:51 PM

Quote:

Originally Posted by GrouchyAdmin

No problem.

Then you've got bigger issues. Some native Cygwin ports do/don't automatically distinct case between CR (unix) and CR+LF (dos). Not a good plan. You'd be better off using Perl than even awk in this case.

Just tried making a list of e-mails in Notepad out of curiosity and it works here *shrug*

I'd expect that utilities whose job is to work with lines would be ported properly (like sort, cat, etc)

Dave PSC · 02-23-2008, 03:58 PM

Linux and VMware on a desktop - great combination!

squeezeboobs · 02-23-2008, 04:00 PM

now is there a way to combine a bunch of csv files into one file and then I can run the filter tool.

hungry hungry hippy · 02-23-2008, 04:36 PM

eListPro .. oldschool

squeezeboobs · 02-23-2008, 05:22 PM

I'll check that out....

later that night...

Wow you were right. It combines all my lists into 1 file, cleans them of malformed emails, filters out domains I don't want and then sorts by domain or randomizes.

13k clean emails in 1 list!!

Who's buying??

02-23-2008, 03:19 PM	#1
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	mailing list cleaner to rid duplicate emails no pics inside Help a girl out!! Anyone heard of a software or script that would go through tons of txt or csv files I have with email addresses and combine them into one that takes out duplicates? I have over 30,000 emails that I've collected over the years from different sites of mine and I want to combine them all into a new email list software that I have. Any help is appreciated. __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 03:22 PM	#2
GrouchyAdmin Now choke yourself! Industry Role: Join Date: Apr 2006 Posts: 12,085	Excel can do it. Just import your data, sort, remove dupes. You're likely stuck at a max of 65,000 records per sheet, though. __________________ AIM: GrouchyGfy

02-23-2008, 03:32 PM	#3
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	how can I remove dups? I dont see anything to do that in the menus. __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 03:33 PM	#4
GrouchyAdmin Now choke yourself! Industry Role: Join Date: Apr 2006 Posts: 12,085	http://support.microsoft.com/kb/262277 __________________ AIM: GrouchyGfy

02-23-2008, 03:42 PM	#5
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	ok I found this of the Microcrap site: 1. Select all the rows, including the column headers, in the list you want to filter. ShowTip Click the top left cell of the range, and then drag to the bottom right cell. 2. On the Data menu, point to Filter, and then click Advanced Filter. 3. In the Advanced Filter dialog box, click Filter the list, in place. 4. Select the Unique records only check box, and then click OK. The filtered list is displayed and the duplicate rows are hidden. 5. On the Edit menu, click Office Clipboard. The Clipboard task pane is displayed. 6. Make sure the filtered list is still selected, and then click Copy Copy button. The filtered list is highlighted with bounding outlines and the selection appears as an item at the top of the Clipboard. 7. On the Data menu, point to Filter, and then click Show All. The original list is re-displayed. 8. Press the DELETE key. The original list is deleted. 9. In the Clipboard, click on the filtered list item. The filtered list appears in the same location as the original list. __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 03:43 PM	#6
Linguist Confirmed User Join Date: Apr 2004 Location: Toronto, ON Posts: 1,706	Put them all into one file and then in Linux or Cygwin: cat file.txt \| sort \| uniq > new_file.txt (this would only work if you have an e-mail per line and nothing else, so if you have names, opt-in data, etc then it wouldn't work as easily) __________________ 315-310 Last edited by Linguist; 02-23-2008 at 03:44 PM..

02-23-2008, 03:44 PM	#7
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	GrouchyAdmin hey thx __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 03:45 PM	#8
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	Linguist whoa whoa now that's crazy. Who runs linux on the desktop. now that's white and nerdy! __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 03:46 PM	#9
Linguist Confirmed User Join Date: Apr 2004 Location: Toronto, ON Posts: 1,706	http://cygwin.com ;) __________________ 315-310

02-23-2008, 03:58 PM	#12
Dave PSC Confirmed User Join Date: Feb 2008 Location: Philadelphia Posts: 224	Linux and VMware on a desktop - great combination! __________________ www.pinkstarcash.com Our newest site!! www.glamurebabes.com Dave ICQ: 434741855

02-23-2008, 04:00 PM	#13
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	now is there a way to combine a bunch of csv files into one file and then I can run the filter tool. __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193

02-23-2008, 04:36 PM	#14
hungry hungry hippy Confirmed User Join Date: Mar 2007 Posts: 249	eListPro .. oldschool

02-23-2008, 05:22 PM	#15
squeezeboobs Confirmed User Industry Role: Join Date: Nov 2003 Location: Boston MA Posts: 887	I'll check that out.... later that night... Wow you were right. It combines all my lists into 1 file, cleans them of malformed emails, filters out domains I don't want and then sorts by domain or randomizes. 13k clean emails in 1 list!! Who's buying?? __________________ (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) (oYo) ICQ 149295193