Damn Duplicates -- p00f begone!

[Barry-xlovecam]

I wrote a fast file cleaner the grep use is interesting ...
Clean your lists up, etc. ...

#!/usr/bin/perl
####################################
# nodupes.cgi
#you may use free as-is with  no warranty
#make the outfile's chmod 666 if using the webserver
#chmod this script to 755
####################################
use CGI::Carp qw/fatalsToBrowser/;
use CGI qw/:standard/;

print "Content-type: text/html\n\n";

my $query = "$ENV{'QUERY_STRING'}";
	if ($query =~ s/[^a-zA-Z0-9\_]//g) {print qq~HUH???~;       exit;}

my $infile="somesiteurl.txt";
my $outfile="somesiteurlduped.txt";

   open(INPUT, "<", $infile) || die "infile not found\n";
      my @array=(<INPUT>);
   open(OUTPUT, ">>",$outfile )|| die "outfile not found\n";


         [COLOR="Yellow"]my %seen = ();
         my @unique = grep { ! $seen{ $_ }++ } @array;[/COLOR]

            foreach my $unique(@unique){
                  chomp $unique;
                  print OUTPUT "$unique\n"
                    }

close OUTPUT;
close INPUT;

[V_RocKs]

Nice.... reminds me of 2000...

[mikke]

can you port it to brainfuck?

[Barry-xlovecam]

can you port it to brainfuck?

Something understandable ...

[fris]

Something understandable ...

or you could just use cat file.txt | sort -u

;)

much quicker

[Barry-xlovecam]

cat infile.txt|sort -u > outfile.txt

No spaces and the outfile

That is a lot easier fris, ty

[alextokyo]

Someone said something about a poof?

[Barry-xlovecam]

p00f not poof -- comprehension problems?

[fris]

cat infile.txt|sort -u > outfile.txt

No spaces and the outfile

That is a lot easier fris, ty

awk way to do it, to remove dups without sorting

awk '!x[$0]++' file.txt

perl without sorting

perl -ne 'print if !$a{$_}++'

this would remove dupe entries on a file with a single column

awk

awk '{ if ($1 in stored_lines) x=1; else print; stored_lines[$1]=1 }' infile.txt > outfile.txt

perl

perl -ane 'print unless $x{$F[0]}++' infile > outfile

sunday gfy bonus

count and show duplicate file names

find . -type f  |sed "s#.*/##g" |sort |uniq -c -d

extra bonus

fild duplicate files based on filesize, then md5 hash

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate