need to de-dupe keyword list... solution?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Mr Pheer
    So Fucking Banned
    • Dec 2002
    • 22082

    #1

    need to de-dupe keyword list... solution?

    I have a list of keywords, one phrase or keyword per line

    the list has alot of duplicates... whats the best way to strip them out?

    help please
  • gornyhuy
    Chafed.
    • May 2002
    • 18041

    #2
    One approach:
    -Import to excel
    -sort alphabetically
    -run a formula comparing each entry to the one above and below, and mark it as a dupe (or delete it)
    -for example =IF(OR(A3=A4,A3=A2),"Duplicate","")
    -then sort by the duplicate status and delete

    ish.

    icq:159548293

    Comment

    • gornyhuy
      Chafed.
      • May 2002
      • 18041

      #3
      Here is a less manual excel approach that I haven't tested, but it looks damn sexy:
      http://www.rondebruin.nl/easyfilter.htm

      icq:159548293

      Comment

      • Mr Pheer
        So Fucking Banned
        • Dec 2002
        • 22082

        #4
        what about a solution for people that dont have excel?

        I dont have any office applications

        Comment

        • mrkris
          Confirmed User
          • May 2005
          • 2737

          #5
          If you have access to *nix, try:

          $ cat list.txt|uniq > newlist.txt

          PHP-MySQL-Rails | ICQ: 342500546

          Comment

          • severe
            Confirmed User
            • Dec 2007
            • 331

            #6
            in excel you dont need a formula to remove dupes, theres a feature to show only non dupes. in older versions its called something like 'show original content' in 2007 under data tab its just called remove dupes

            Comment

            • Mr Pheer
              So Fucking Banned
              • Dec 2002
              • 22082

              #7
              Originally posted by mrkris
              If you have access to *nix, try:

              $ cat list.txt|uniq > newlist.txt
              I tried that on freebsd and it just made a copy of the same file with a new name

              Comment

              • Mr Pheer
                So Fucking Banned
                • Dec 2002
                • 22082

                #8
                someone help me out with the syntax error on line 18 please?

                Code:
                #!/usr/bin/perl
                use strict;
                
                my $FileName = 'file.txt'; # Modify file name as needed.
                
                my(@List,%List,@NewList)= ();
                
                sub Abandon
                {
                print join '
                ',@_;
                exit;
                } # sub Abandon
                
                print "Content-type: text/plain\n\n";
                
                Abandon("Unable to read file $FileName") unless open R,"<$FileName";
                @List = ;
                close R;
                
                Abandon("Unable to create temporary file ${FileName}.tmp.txt") unless open W,">${FileName}.tmp.txt";
                for(@List) { print W $_; }
                close W;
                
                for(@List)
                {
                next if $List{$_};
                $List{$_}++;
                push @NewList,$_;
                }
                
                Abandon('Something wrong.',"Backup file is ${FileName}.tmp.txt") unless open W,">$FileName";
                for(@NewList) { print W $_; }
                close W;
                
                unlink "${FileName}.tmp.txt";
                
                print 'D O N E';

                Comment

                • react
                  Confirmed User
                  • Sep 2003
                  • 673

                  #9
                  You must sort before you can uniq:

                  cat infile | sort | uniq > outputfile
                  --
                  react

                  Comment

                  • Mr Pheer
                    So Fucking Banned
                    • Dec 2002
                    • 22082

                    #10
                    Originally posted by react
                    You must sort before you can uniq:

                    cat infile | sort | uniq > outputfile
                    w00t!!!

                    thanks man

                    Comment

                    • gornyhuy
                      Chafed.
                      • May 2002
                      • 18041

                      #11
                      While we are on the subject, does anybody have a good query for deduping mysql tables across multiple fields?

                      icq:159548293

                      Comment

                      • react
                        Confirmed User
                        • Sep 2003
                        • 673

                        #12
                        That multiple fields bit isn't super clear.. but if you want to combine data in several columns of one table into a single unique column create a new table with one column that has unique index on it. Then for each of the columns in the old table:

                        insert ignore into newtable (newcolumn) select oldcolumn1 from oldtable;
                        insert ignore into newtable (newcolumn) select oldcolumn2 from oldtable;

                        If you just want to keep all unique rows then create new table with the same column structure, create a unique index across all columns, then:

                        insert ignore into newtable select * from oldtable
                        --
                        react

                        Comment

                        • rowan
                          Too lazy to set a custom title
                          • Mar 2002
                          • 17393

                          #13
                          Originally posted by react
                          You must sort before you can uniq:

                          cat infile | sort | uniq > outputfile
                          No need for uniq in that case... or cat

                          sort -u infile > outfile

                          Comment

                          Working...