Re: [plug] perl - removing duplicate line entries in a file

Top Page

Reply to this message
Author: Ambrosio Berdijo
To: roderick tapang, The Main Philippine Linux Users' Group (PLUG) Discussion List
Subject: Re: [plug] perl - removing duplicate line entries in a file
Assuming you want to do the job without installing the
12MB Win32 port of ActivePerl,

if your don't mind re-arranging the data.

    gsort [filename] | uniq

if you need to keep the data in order

# uniq.awk 
    if (data[$0]++ == 0)
        lines[++count] = $0

    for (i = 1; i <= count; i++)
        print lines[i]

gawk -f uniq.awk [filename]

All this power by just downloading the gsort, uniq and
gawk Win32 binaries from


--- roderick tapang <retapang@???> wrote:
> how do you remove duplicate line entries in a file
> using perl? we have some logfiles that are xferred
> to a windows machine (don't ask why) for analysis
> and we only have ActivePerl available in it and one

Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
Philippine Linux Users' Group (PLUG) Mailing List
plug@??? (#PLUG @
Read the Guidelines:
Searchable Archives: