Eating memory

[Jonathan Stowe]

On Tue, 2006-05-23 at 12:53, Simon Wilcox wrote:
> Sorry, I have a perl question :-)
> 
> I've been working on a simple script to munge some logs and produce some
> stats but top suggests that it's loading the whole file into memory when I
> think it should be working line by line. This is obviously not good when
> there are gigs of logs to look through.
> 
> I've reduced the script to the simplest case but it is still not reading
> the file line by line. I must be doing something stupid - can anyone see
> what it is ?
> 
> #! /usr/bin/perl
> 
> use warnings;
> use strict;
> 
> my $filename = $ARGV[0];
> 
> open INFILE, "$filename" or die "Can't open file : $!\n";
> 
> foreach (<INFILE>) {
> }
> 

Don't you mean

while (<INFILE>) {
}

I think you'll that foreach will cause the whole file to be read
upfront.

/J\
> perl -V says:
> 
> Summary of my perl5 (revision 5 version 8 subversion 4) configuration:
>   Platform:
>     osname=linux, osvers=2.6.15.4, archname=i386-linux-thread-multi
>     uname='linux ninsei 2.6.15.4 #1 smp preempt mon feb 20 09:48:53 pst
> 2006 i686 gnulinux '
>     config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
> -Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr
> -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8
> -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
> -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
> -Dsitelib=/usr/local/share/perl/5.8.4 -Dsitearch=/usr/local/lib/perl/5.8.4
> -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
> -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
> -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh
> -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.4 -Dd_dosuid -des'
>     hint=recommended, useposix=true, d_sigaction=define
>     usethreads=define use5005threads=undef useithreads=define
> usemultiplicity=define
>     useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
>     use64bitint=undef use64bitall=undef uselongdouble=undef
>     usemymalloc=n, bincompat5005=undef
>   Compiler:
>     cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
> -DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
> -D_FILE_OFFSET_BITS=64',
>     optimize='-O2',
>     cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN
> -fno-strict-aliasing -I/usr/local/include'
>     ccversion='', gccversion='3.3.5 (Debian 1:3.3.5-13)', gccosandvers=''
>     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
>     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
>     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize=8
>     alignbytes=4, prototype=define
>   Linker and Libraries:
>     ld='cc', ldflags =' -L/usr/local/lib'
>     libpth=/usr/local/lib /lib /usr/lib
>     libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
>     perllibs=-ldl -lm -lpthread -lc -lcrypt
>     libc=/lib/libc-2.3.2.so, so=so, useshrplib=true,
> libperl=libperl.so.5.8.4
>     gnulibc_version='2.3.2'
>   Dynamic Linking:
>     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>     cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'
> 
> 
> Characteristics of this binary (from libperl):
>   Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
> PERL_IMPLICIT_CONTEXT
>   Built under linux
>   Compiled at Mar 23 2006 21:49:08
>   @INC:
>     /etc/perl
>     /usr/local/lib/perl/5.8.4
>     /usr/local/share/perl/5.8.4
>     /usr/lib/perl5
>     /usr/share/perl5
>     /usr/lib/perl/5.8
>     /usr/share/perl/5.8
>     /usr/local/lib/site_perl
>     .
> 
> top reports the file size just before it exits as a little bigger than the
> size of the log file. I've been using a 450Mb file for testing purposes.
> 
> I'm sure I'm doing something stupid but I can't for the life of me see
> what it is.
> 
> Simon.
-- 

This e-mail is sponsored by http://www.integration-house.com/