Re: [plug] perl - removing duplicate line entries in a file

Top Page

Reply to this message
Author: Ambrosio Berdijo
Date:  
To: roderick tapang, The Main Philippine Linux Users' Group (PLUG) Discussion List
Subject: Re: [plug] perl - removing duplicate line entries in a file
Assuming you want to do the job without installing the
12MB Win32 port of ActivePerl,

if your don't mind re-arranging the data.

    gsort [filename] | uniq


if you need to keep the data in order

# uniq.awk 
{
    if (data[$0]++ == 0)
        lines[++count] = $0
}


END {
    for (i = 1; i <= count; i++)
        print lines[i]
}


gawk -f uniq.awk [filename]

All this power by just downloading the gsort, uniq and
gawk Win32 binaries from
http://unxutils.sourceforge.net/

Ambo

--- roderick tapang <retapang@???> wrote:
> how do you remove duplicate line entries in a file
> using perl? we have some logfiles that are xferred
> to a windows machine (don't ask why) for analysis
> and we only have ActivePerl available in it and one



__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
_________________________________________________
Philippine Linux Users' Group (PLUG) Mailing List
plug@??? (#PLUG @ irc.free.net.ph)
Read the Guidelines: http://linux.org.ph/lists
Searchable Archives: http://archives.free.net.ph