keys()/each() leak memory with threads::shared #10723

p5pRT · 2010-10-13T19:45:25Z

Migrated from rt.perl.org#78362 (status was 'resolved')

Searchable as RT78362$

p5pRT · 2010-10-13T19:45:28Z

From @jdhedden

Created by @jdhedden

Using keys() and/or each() on threads::shared hashes causes a significant
memory leak. For example:

my $hash = &threads::shared::share({});

for (1..$MAX_LOOP) {
$$hash{1} = 1;
keys(%{$hash});
if ((my $key) = each(%{$hash})) {
delete($$hash{$key});
}
}

The attached test script contains the above as well as some variants
that do not leak but which illustrate that keys()/each() are the
problems.

My suspicion is that the HvKEYS() call in the following code from
dist/threads-shared/shared.xs is the culprit:

/* Len magic for PERL_MAGIC_tied(P) */

U32
sharedsv_array_mg_FETCHSIZE(pTHX_ SV *sv, MAGIC *mg)
{
dTHXc;
SV *ssv = (SV *) mg->mg_ptr;
U32 val;
SHARED_EDIT;
if (SvTYPE(ssv) == SVt_PVAV) {
val = av_len((AV*) ssv);
} else {
/* Not actually defined by tie API but ... */
val = HvKEYS((HV*) ssv);
}
SHARED_RELEASE;
return (val);
}

Perl Info


Flags:
    category=library
    severity=medium
    module=threads::shared

Site configuration information for perl 5.12.2:

Configured by Jerry at Thu Sep 16 09:52:43 EDT 2010.

Summary of my perl5 (revision 5 version 12 subversion 2 patch 42531)
configuration:
  Snapshot of: 7a3b65c9d99f69553fffe01f73d49fe9abf95120
  Platform:
    osname=cygwin, osvers=1.5.25(0.15642), archname=cygwin-thread-multi-64int
    uname='cygwin_nt-5.1 pn100-02-2-364p 1.5.25(0.15642) 2008-06-12
19:34 i686 cygwin '
    config_args='-de -Duse64bitint -Dusethreads
-Dinc_version_list=none -Dnoextensions=IPC/SysV Sys/Syslog Devel/DProf
Devel/Peek re GDBM_File NDBM_File ODBM_File Text/Soundex
Math/BigInt/FastCalc Time/Piece -A append:ccflags= -DNO_MATHOMS  -A
define:optimize=-Os -pipe -funit-at-a-time -march=pentium4
-mfpmath=sse -mieee-fp -mmmx -msse -msse2'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__
-DNO_MATHOMS -fno-strict-aliasing -pipe',
    optimize='-Os -pipe -funit-at-a-time -march=pentium4 -mfpmath=sse
-mieee-fp -mmmx -msse -msse2',
    cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -DNO_MATHOMS
-fno-strict-aliasing -pipe'
    ccversion='', gccversion='3.4.4 (cygming special, gdc 0.12, using
dmd 0.125)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags =' -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--stack,8388608
-Wl,--enable-auto-image-base -s -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib
    libs=-lgdbm -ldl -lcrypt -lgdbm_compat
    perllibs=-ldl -lcrypt
    libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=cygperl5_12_2.dll
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' -s'
    cccdlflags=' ', lddlflags=' --shared  -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--stack,8388608
-Wl,--enable-auto-image-base -s -L/usr/local/lib'

Locally applied patches:



@INC for perl 5.12.2:
    /usr/lib/perl5/site_perl/5.12.2/cygwin
    /usr/lib/perl5/site_perl/5.12.2
    /usr/lib/perl5/5.12.2/cygwin
    /usr/lib/perl5/5.12.2
    .


Environment for perl 5.12.2:
    CYGWIN=ntsec
    HOME=/home/jhedden
    LANG=C
    LANGUAGE=C
    LC_ALL=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/jhedden/bin:/usr/local/src/perl/bin:/link/work/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/c/Program
Files/WiX:/c/djgpp/bin:/c/bb/tools/apache-ant-1.6.5/bin:/c/Program
Files/nant-0.85/bin:/c/bb/tools/jse5/bin:/c/dev-cpp/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/system32/WBEM:/c/blp/API/dde:/c/blp/API:/c/oracle/ora92/bin:/c/Program
Files/Oracle/jre/1.3.1/bin:/c/Program
Files/Oracle/jre/1.1.8/bin:/c/Program
Files/Hummingbird/Connectivity/7.10/Accessories:/usr/bin:/c/Program
Files/Windows Imaging/:/c/Program Files/Citrix/ICAService/:/c/Program
Files/Citrix/System32/:.
    PERLIO=perlio
    PERL_BADLANG (unset)
    SHELL (unset)

p5pRT · 2010-10-13T19:45:28Z

From @jdhedden

leak.pl

p5pRT · 2014-09-23T01:50:02Z

From @jkeenan

On Wed Oct 13 12:45:28 2010, jdhedden@cpan.org wrote:

This is a bug report for perl from jdhedden@cpan.org,
generated with the help of perlbug 1.39 running under perl 5.12.2.

-----------------------------------------------------------------
[Please describe your issue here]

Using keys() and/or each() on threads::shared hashes causes a
significant
memory leak. For example:

my $hash = &threads::shared::share({});

for (1..$MAX_LOOP) {
$$hash{1} = 1;
keys(%{$hash});
if ((my $key) = each(%{$hash})) {
delete($$hash{$key});
}
}

The attached test script contains the above as well as some variants
that do not leak but which illustrate that keys()/each() are the
problems.

My suspicion is that the HvKEYS() call in the following code from
dist/threads-shared/shared.xs is the culprit:

/* Len magic for PERL_MAGIC_tied(P) */

U32
sharedsv_array_mg_FETCHSIZE(pTHX_ SV *sv, MAGIC *mg)
{
dTHXc;
SV *ssv = (SV *) mg->mg_ptr;
U32 val;
SHARED_EDIT;
if (SvTYPE(ssv) == SVt_PVAV) {
val = av_len((AV*) ssv);
} else {
/* Not actually defined by tie API but ... */
val = HvKEYS((HV*) ssv);
}
SHARED_RELEASE;
return (val);
}

Reviewing this older ticket tonight, I built a threaded perl (see attachment for perl -V) at HEAD of blead, then ran the test script which Jerry provided a couple of years back.

This was the output:
#####
$ ./perl -Ilib ~/learn/perl/p5p/78362-leak.pl
Leak test 1 running...
Leak test 2 running...
Leak test 3 running...
Done
#####

The program completed in 0m0.779s. Should I have expected to see some error output? Has the bug somehow been fixed?

Thank you very much.

--
James E Keenan (jkeenan@cpan.org)

p5pRT · 2014-09-23T01:50:02Z

From @jkeenan

78362-leak.pl

p5pRT · 2014-09-23T01:50:02Z

From @jkeenan

Summary of my perl5 (revision 5 version 21 subversion 5) configuration:
Commit id: a0d9739
Platform:
osname=linux, osvers=3.13.0-35-generic, archname=x86_64-linux-thread-multi
uname='linux zareason 3.13.0-35-generic #62-ubuntu smp fri aug 15 01:58:42 utc 2014 x86_64 x86_64 x86_64 gnulinux '
config_args='-des -Dusedevel -Duse64bitint -Dusethreads'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.8.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.19'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
USE_ITHREADS USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
USE_PERL_ATOF USE_REENTRANT_API
Built under linux
Compiled at Sep 22 2014 21:40:56
%ENV:
PERL5LIB="/home/jkeenan/perl5/lib/perl5"
PERLBREW_BASHRC_VERSION="0.67"
PERLBREW_HOME="/home/jkeenan/.perlbrew"
PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.20.0/man"
PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin:/home/jkeenan/perl5/perlbrew/perls/perl-5.20.0/bin"
PERLBREW_PERL="perl-5.20.0"
PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"
PERLBREW_VERSION="0.67"
PERL_LOCAL_LIB_ROOT="/home/jkeenan/perl5"
PERL_MB_OPT="--install_base "/home/jkeenan/perl5""
PERL_MM_OPT="INSTALL_BASE=/home/jkeenan/perl5"
PERL_WORKDIR="gitwork/perl"
@INC:
lib
/home/jkeenan/perl5/lib/perl5
/usr/local/lib/perl5/site_perl/5.21.5/x86_64-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.21.5
/usr/local/lib/perl5/5.21.5/x86_64-linux-thread-multi
/usr/local/lib/perl5/5.21.5
.

p5pRT · 2014-09-23T01:50:03Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2014-09-23T10:02:24Z

From @iabyn

On Mon, Sep 22, 2014 at 06:50:03PM -0700, James E Keenan via RT wrote:

On Wed Oct 13 12:45:28 2010, jdhedden@cpan.org wrote:
Reviewing this older ticket tonight, I built a threaded perl (see attachment for perl -V) at HEAD of blead, then ran the test script which Jerry provided a couple of years back.

This was the output:
#####
$ ./perl -Ilib ~/learn/perl/p5p/78362-leak.pl
Leak test 1 running...
Leak test 2 running...
Leak test 3 running...
Done
#####

The program completed in 0m0.779s. Should I have expected to see some error output? Has the bug somehow been fixed?

Increase $MAX_LOOP so that each loop runs for longer, then execute while
top is running. This will show (on 5.12.x) that the first two tests use
constant virtual memory, while the third test sees quickly increasing VM.

However, it appears to have been fixed sometime between 5.16.0 and 5.18.0

--
Never do today what you can put off till tomorrow.

p5pRT · 2014-09-24T11:11:16Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Sep 24, 2014

p5pRT added Severity Low leak/refcount/malloc type-ithreads type-library labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

keys()/each() leak memory with threads::shared #10723

keys()/each() leak memory with threads::shared #10723

p5pRT commented Oct 13, 2010

p5pRT commented Oct 13, 2010

p5pRT commented Oct 13, 2010

p5pRT commented Sep 23, 2014

p5pRT commented Sep 23, 2014

p5pRT commented Sep 23, 2014

p5pRT commented Sep 23, 2014

p5pRT commented Sep 23, 2014

p5pRT commented Sep 24, 2014

keys()/each() leak memory with threads::shared #10723

keys()/each() leak memory with threads::shared #10723

Comments

p5pRT commented Oct 13, 2010

p5pRT commented Oct 13, 2010

From @jdhedden

Created by @jdhedden

p5pRT commented Oct 13, 2010

From @jdhedden

p5pRT commented Sep 23, 2014

From @jkeenan

p5pRT commented Sep 23, 2014

From @jkeenan

p5pRT commented Sep 23, 2014

From @jkeenan

p5pRT commented Sep 23, 2014

p5pRT commented Sep 23, 2014

From @iabyn

p5pRT commented Sep 24, 2014