little help with memory leak/management by the kernel/libg++/libc?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

little help with memory leak/management by the kernel/libg++/libc?

Jairo19@interhosting.us
   Hola:

This demo program I wrote shows how much memory the process is using (as
given by the kernel /proc/self/status interface) before and after I
request and free memory.

So the issue is that as I request and free memory of different sizes,
the process seems not to relinquish the full amount of requested memory.
My real world program seems to have a small loss of approx 100Kb for
almost each memory request/release cycle, and it needs to do this 500000
times, so you can imagine that I am running out of memory, but the
program is releasing all the memory it asks for. In both cases valgrind
does not report any leaks.

In the table you see that the demo program starts with 180Kb, but ends
using 556Kb. I do not understand why.

[jairo@linux5 scripts]$ make memfree
g++     memfree.cpp   -o memfree
[jairo@linux5 scripts]$ ./memfree

VmStk      VmLib      VmData      VmSize      VmRSS
    84        2540        *180*                2876         *  832*
    84        2540        436                 3132           916
    84        2540        184                2880            916
    84        2540        184                2880            916
    84        2540        748                3444            920
    84        2540        184                2880            916
    84        2540        184                2880            916
    84        2540        624                3320            920
    84        2540        184                2880            920
    84        2540        184                2880            920
    84        2540        *1188*             3884            924
    84        2540        *184*                2880           *920*

    84        2540        184                2880            920
    84        2540        556                3252            928

    84        2540        556                3252            928
    84        2540        556                3252            928
    84        2540        *556*                3252            *940*


Program:
memfree.cpp
#include <fstream>
#include <iostream>
#include <string>

using namespace std;

string  getProcStatus(char*field) {
   string s = "";
   ifstream in;
   in.open("/proc/self/status");
   while(!in.eof()) {
     in >> s;
     if (s.find(field) == string::npos) continue;
     in >> s;
     break;
   }
   in.close();

   return(s);
} // getProcStatus

class bar {
public:
  int one;
  int two;
  int three;
  double four;
  double five;
  double six;
  double seven;
  long long eight;
  int    nine;
  double  ten;
};

int main() {
   bar *memory_get = NULL;
  cout << "Virt mem used before creating data 1:                " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   memory_get = new bar[4000];
  cout << "Virt mem used after creating/before deleting data 1: " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   delete[] memory_get;
  cout << "Virt mem used after deleting data 1:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

  cout << "Virt mem used before creating data 2:                "<<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib") <<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   memory_get = new bar[9000];
  cout << "Virt mem used after creating/before deleting data 2: " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   delete[] memory_get;
  cout << "Virt mem used after deleting data 2:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

  cout << "Virt mem used before creating data 3:                " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   memory_get = new bar[7000];
  cout << "Virt mem used after creating/before deleting data 3: " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   delete[] memory_get;
  cout << "Virt mem used after deleting data 3:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

  cout << "Virt mem used before creating data 4:                "<<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" " <<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   memory_get = new bar[16000];
  cout << "Virt mem used after creating/before deleting data 4: " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   delete[] memory_get;
  cout << "Virt mem used after deleting data 4:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;


  cout<<endl;
  cout << "Virt mem used before creating data 5:                " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   for(unsigned int i = 0; i < 100000; ++i) {
     unsigned int j = 1 + (int) (10.0 * (rand() / (RAND_MAX + 1.0)));
     memory_get = new bar[6000*j];
     delete[] memory_get;
   }
  cout << "Virt mem used after deleting data 5:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

sleep(5);
sleep(5);

  cout<<endl;
  cout << "Virt mem used before creating data 6:                " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;
   for(unsigned int i = 0; i < 1000000; ++i) {
     unsigned int j = 1 + (int) (10.0 * (rand() / (RAND_MAX + 1.0)));
     memory_get = new bar[6000*j];
     delete[] memory_get;
   }
  cout << "Virt mem used after deleting data 6:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

  cout << "Virt mem used after deleting data 7:                 " <<
getProcStatus("VmStk")<<" "<< getProcStatus("VmLib")<<" "<<
getProcStatus("VmData")<<" "<< getProcStatus("VmSize")<<" " <<
getProcStatus("VmRSS")<<"     pointer "<<memory_get << endl;

}

NOTE: same program but using malloc/free of a struct instead of
new/delete of a class has the same result, but with a smaller VmRSS in
almost every instance. Also the programs wait for 10 seconds to see if
the kernel needs time adjusting the status.

There seems to be a relationship between the size after the delete if
the next requested block of memory is smaller (the VmData increases), I
modified the program a bit to first allocate 6000, then 5000, then 4000,
then 3000 and the result is:

VmStk      VmLib      VmData      VmSize      VmRSS
    84        2540        *180*                2880         *  828*
    84        2540        *560 *              3260           912
    84        2540        184                2884            912
    84        2540        184                2884            912
    84        2540        *496*               3196            920
    84        2540        496                3196            920
    84        2540        496                3196            920
    84        2540        496                3196            928
    84        2540        496                3196            932
    84        2540        496                3196            932
    84        2540        496                3196            940
    84        2540        496                3196           948

    84        2540        496                3196            948
    84        2540        496                3196            956

    84        2540        496                3196            956
    84        2540        496                3196            956
    84        2540        *496*               3196            *956*


84 2540 *180** 2880 *828**
84 2540 *560** 3260 912
84 2540 184 2884 912
84 2540 184 2884 912
84 2540 *496** 3196 920
84 2540 496 3196 920
84 2540 496 3196 920
84 2540 496 3196 928
84 2540 496 3196 932
84 2540 496 3196 932
84 2540 496 3196 940
84 2540 496 3196 948

       
       
       
       
84 2540 496 3196 948
84 2540 496 3196 956

       
       
       
       
84 2540 496 3196 956
84 2540 496 3196 956
84 2540 *496** 3196 *956**


I'm using FC7, and FC8, I also tried it in kubuntu,  all with similar
results.

What do you think?

  Thanks in advance.

--
Jairo Medina

Reply | Threaded
Open this post in threaded view
|

Re: little help with memory leak/management by the kernel/libg++/libc?

Ulrich Drepper
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jairo Medina wrote:
> This demo program I wrote shows how much memory the process is using (as
> given by the kernel /proc/self/status interface) before and after I
> request and free memory.

This is no list for programming help.  Take this elsewhere.

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFH+74w2ijCOnn/RHQRAqhMAKC4SFKjXSf1gFfKOlNxieVqRjoYawCeIxDj
nGswsL4MVDE59w6sLxTWEfU=
=ca0j
-----END PGP SIGNATURE-----
Reply | Threaded
Open this post in threaded view
|

Re: little help with memory leak/management by the kernel/libg++/libc?

Steve Munroe
In reply to this post by Jairo19@interhosting.us
[hidden email] wrote on 04/08/2008 01:41:07 PM:

>    Hola:
>
> This demo program I wrote shows how much memory the process is using (as
> given by the kernel /proc/self/status interface) before and after I
> request and free memory.
>
> So the issue is that as I request and free memory of different sizes,
> the process seems not to relinquish the full amount of requested memory.
> My real world program seems to have a small loss of approx 100Kb for
> almost each memory request/release cycle, and it needs to do this 500000
> times, so you can imagine that I am running out of memory, but the
> program is releasing all the memory it asks for. In both cases valgrind
> does not report any leaks.
>

Working as designed. You can read about malloc's internal design by reading
the comments in the source:

http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/malloc/malloc.c?cvsroot=glibc

You may be able to adjust things more to your liking using the mallopt
interface:

http://www.gnu.org/software/libc/manual/html_mono/libc.html#Malloc-Tunable-Parameters

Steven J. Munroe
Linux on Power Toolchain Architect
IBM Corporation, Linux Technology Center