[ph-perl] Re: [plug] perl - removing duplicate line entries …

Top Page

Reply to this message
Author: ph-perl Mailing List
Date:  
To: roderick tapang, The Main Philippine Linux Users' Group (PLUG) Discussion List
CC: ph-perl
Subject: [ph-perl] Re: [plug] perl - removing duplicate line entries in a file
On Wed, May 04, 2005 at 10:12:18AM +0800, roderick tapang wrote:
> how do you remove duplicate line entries in a file
> using perl? we have some logfiles that are xferred
> to a windows machine (don't ask why) for analysis
> and we only have ActivePerl available in it and one
> staff needs to have the duplicate lines remove.
> i really don't do perl so can anyone give out some
> lines of codes for this?
>
> (i posted this in ph-perl but didn't get any response
> yet).
>


Try this:

#!/usr/bin/perl
# usage: remove_dup_in_place.pl file1 file2 ... fileN

while (<>) {
  if ($ARGV ne $oldargv) {
    open(ARGVOUT, ">$ARGV");
    $oldargv = $ARGV;
    undef %hash;
  }
  $hash{$_}++;
  print ARGVOUT sort keys %hash 
    if (eof and defined fileno(ARGVOUT));
}


exit;

--
$_=q:; # SHERWIN #
70;72;69;6e;74;20;
27;4a;75;73;74;20;
61;6e;6f;74;68;65;
72;20;50;65;72;6c;
20;6e;6f;76;69;63;
65;27;:;;s=~?(..);
?=pack q$C$,hex$1;
;;;=egg;;;;eval;;;
_________________________________________________
Philippine Linux Users' Group (PLUG) Mailing List
plug@??? (#PLUG @ irc.free.net.ph)
Read the Guidelines: http://linux.org.ph/lists
Searchable Archives: http://archives.free.net.ph