![]() |
![]() |
![]() |
||||
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
![]() ![]() |
|
Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
Thread Tools |
![]() |
#1 |
Retired
Industry Role:
Join Date: Dec 2002
Posts: 21,246
|
need to de-dupe keyword list... solution?
I have a list of keywords, one phrase or keyword per line
the list has alot of duplicates... whats the best way to strip them out? help please ![]()
__________________
2 lifeguards for Jessica |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#2 |
Chafed.
Join Date: May 2002
Location: Face Down in Pussy
Posts: 18,041
|
One approach:
-Import to excel -sort alphabetically -run a formula comparing each entry to the one above and below, and mark it as a dupe (or delete it) -for example =IF(OR(A3=A4,A3=A2),"Duplicate","") -then sort by the duplicate status and delete ish. |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#3 |
Chafed.
Join Date: May 2002
Location: Face Down in Pussy
Posts: 18,041
|
Here is a less manual excel approach that I haven't tested, but it looks damn sexy:
http://www.rondebruin.nl/easyfilter.htm |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#4 |
Retired
Industry Role:
Join Date: Dec 2002
Posts: 21,246
|
what about a solution for people that dont have excel?
I dont have any office applications
__________________
2 lifeguards for Jessica |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#5 |
Confirmed User
Join Date: May 2005
Posts: 2,737
|
If you have access to *nix, try:
$ cat list.txt|uniq > newlist.txt |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#6 |
Confirmed User
Industry Role:
Join Date: Dec 2007
Posts: 331
|
in excel you dont need a formula to remove dupes, theres a feature to show only non dupes. in older versions its called something like 'show original content' in 2007 under data tab its just called remove dupes
|
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#7 |
Retired
Industry Role:
Join Date: Dec 2002
Posts: 21,246
|
I tried that on freebsd and it just made a copy of the same file with a new name
__________________
2 lifeguards for Jessica |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#8 |
Retired
Industry Role:
Join Date: Dec 2002
Posts: 21,246
|
someone help me out with the syntax error on line 18 please?
Code:
#!/usr/bin/perl use strict; my $FileName = 'file.txt'; # Modify file name as needed. my(@List,%List,@NewList)= (); sub Abandon { print join ' ',@_; exit; } # sub Abandon print "Content-type: text/plain\n\n"; Abandon("Unable to read file $FileName") unless open R,"<$FileName"; @List = ; close R; Abandon("Unable to create temporary file ${FileName}.tmp.txt") unless open W,">${FileName}.tmp.txt"; for(@List) { print W $_; } close W; for(@List) { next if $List{$_}; $List{$_}++; push @NewList,$_; } Abandon('Something wrong.',"Backup file is ${FileName}.tmp.txt") unless open W,">$FileName"; for(@NewList) { print W $_; } close W; unlink "${FileName}.tmp.txt"; print 'D O N E';
__________________
2 lifeguards for Jessica |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#9 |
Confirmed User
Industry Role:
Join Date: Sep 2003
Location: NZ
Posts: 673
|
You must sort before you can uniq:
cat infile | sort | uniq > outputfile
__________________
-- react |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#10 | |
Retired
Industry Role:
Join Date: Dec 2002
Posts: 21,246
|
Quote:
thanks man ![]()
__________________
2 lifeguards for Jessica |
|
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#11 |
Chafed.
Join Date: May 2002
Location: Face Down in Pussy
Posts: 18,041
|
While we are on the subject, does anybody have a good query for deduping mysql tables across multiple fields?
|
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#12 |
Confirmed User
Industry Role:
Join Date: Sep 2003
Location: NZ
Posts: 673
|
That multiple fields bit isn't super clear.. but if you want to combine data in several columns of one table into a single unique column create a new table with one column that has unique index on it. Then for each of the columns in the old table:
insert ignore into newtable (newcolumn) select oldcolumn1 from oldtable; insert ignore into newtable (newcolumn) select oldcolumn2 from oldtable; If you just want to keep all unique rows then create new table with the same column structure, create a unique index across all columns, then: insert ignore into newtable select * from oldtable
__________________
-- react |
![]() |
![]() ![]() ![]() ![]() ![]() |
![]() |
#13 |
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
|
![]() |
![]() ![]() ![]() ![]() ![]() |