Windows get_connections() and get_process_information() memory leaks

[giampaolo]

From g.rodola on October 26, 2010 20:44:19

======================================================================
ERROR: test_resume (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\user\Desktop\svn\psutil\test\test_psutil.py", line 147, in inner
    return fun(self, *args, **kwargs)
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 94, 
in test_resume
    self.execute('resume')
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 42, in execute
    retvalue = obj(*args, **kwarks)
  File "C:\Python27\lib\site-packages\psutil\__init__.py", line 408, in resume
    self._platform_impl.resume_process()
  File "C:\Python27\lib\site-packages\psutil\_psmswindows.py", line 190, in 
resume_process
    return _psutil_mswindows.resume_process(self.pid)
WindowsError: [Error 6] The handle is invalid

======================================================================
FAIL: test__str__ (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 68, 
in test__str__
    self.execute('__str__')
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 64, in execute
    self.fail("rss1=%s, rss2=%s, difference=%s" %(rss1, rss2, 
difference))
AssertionError: rss1=9527296, rss2=9629696, difference=102400

======================================================================
FAIL: test_get_connections (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\user\Desktop\svn\psutil\test\test_psutil.py", line 147, in inner
    return fun(self, *args, **kwargs)
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 106, 
in test_get_connections
    self.execute('get_connections')
  File "C:\Users\user\Desktop\svn\psutil\test\test_memory_leaks.py", line 64, in execute
    self.fail("rss1=%s, rss2=%s, difference=%s" %(rss1, rss2, 
difference))
AssertionError: rss1=9629696, rss2=9703424, difference=73728

----------------------------------------------------------------------


Tried to debug get_connections() a little and it seems that the leak doesn't 
occur if the intermediate calls to getExtendedTcpTable and getExtendedUdpTable 
are removed (these are necessary though).

I have no clue about __str__. It calls a lot of code (name, ppid, path, 
cmdline, uid, gid) and it's not easy to debug.

Original issue: http://code.google.com/p/psutil/issues/detail?id=122

[giampaolo]

From [email protected] on October 26, 2010 22:15:15

What makes you think it's a memory leak?

[giampaolo]

From g.rodola on October 27, 2010 02:00:58

We have a script [1] we run against functionalities implemented in pure C which 
works as such:

- 1: call C fun 1000 times
- 2: retrieve process memory usage (1)
- 3: call C fun 1000 times
- 4: retrieve process memory usage (2)

If difference between 4 and 2 is > 4 Kbytes we assume the C function has a 
memory leak.
This has shown to work so far, revealing places where we forgot to close 
process handles, call free(), use Py_DECREF [2], etc.
As for get_connections() I suspect it's not something which has to do with Py_* 
objects not being Py_DECREF-ed because the script signals a memory leak even if 
I remove those parts from the code.


[1] 
https://code.google.com/p/psutil/source/browse/trunk/test/test_memory_leaks.py 
[2] http://docs.python.org/c-api/intro.html#reference-counts

[giampaolo]

From g.rodola on November 11, 2010 03:45:16

After bumping up number of loops from 1000 to 2000 the problem magically 
disappeared.
Closing out as invalid for now, but I think our memory leaks script might 
actually have a problem.

Status: Invalid

[giampaolo]

From [email protected] on November 11, 2010 06:24:07

If it goes away with more loops, it could just be garbage collection kicking in 
later, so the memory doesn't get reclaimed immediately.