OSX: memory_maps() segfaults or raise EINVAL

[wrishel]

I get Exception: OSError(22, 'Invalid argument') on OSX 10.13.4 with Python 2.7

Adding this to the loop makes it go away:

for proc in psutil.process_iter():
        if proc.pid == os.getpid(): continue   

[giampaolo]

Please paste the full trace back

On Sun, 10 Jun 2018 at 06:59, wrishel ***@***.***> wrote: I get Exception: OSError(22, 'Invalid argument') on OSX 10.13.4 with Python 2.7 Adding this to the loop makes it go away: for proc in psutil.process_iter(): if proc.pid == os.getpid(): continue — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1291>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAplLAXf0fJCzEADUoq4gBchk7p8xiSSks5t7KeegaJpZM4UhlYK> .

-- Giampaolo - http://grodola.blogspot.com

[wrishel]

Giampaolo, here is the full traceback. I'm embarrassed that I didn't think to do this in the first place.

As always I appreciate how promptly you reply.

Wes

psutil-debug> proc_regionfilename() failed
Traceback (most recent call last):
  File "/Users/Wes/Dropbox/Programming/Python/etpruncnt2.1/lib/gen/queue_coordinator/kill_coordinator.py", line 39, in <module>
    coords = find_all_coordinators()
  File "/Users/Wes/Dropbox/Programming/Python/etpruncnt2.1/lib/gen/queue_coordinator/kill_coordinator.py", line 22, in find_all_coordinators
    pinfo = proc.as_dict()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/__init__.py", line 527, in as_dict
    ret = meth()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/__init__.py", line 1112, in memory_maps
    it = self._proc.memory_maps()
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/_psosx.py", line 330, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/Wes/.virtualenvs/ppyy/lib/python2.7/site-packages/psutil/_psosx.py", line 571, in memory_maps
    return cext.proc_memory_maps(self.pid)
OSError: [Errno 22] Invalid argument

[giampaolo]

Duplicate of #1209.

[wiggin15]

@giampaolo I'm still getting this with the latest code. #1209 seems to deal with zombie processes and task_for_pid but this issue seems to be about the running process and proc_regionfilename:

~ python -c "import psutil; print psutil.Process().memory_maps()"
psutil-debug> proc_regionfilename() failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/arnony/git/external/psutil/psutil/__init__.py", line 1111, in memory_maps
    it = self._proc.memory_maps()
  File "/Users/arnony/git/external/psutil/psutil/_psosx.py", line 476, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/arnony/git/external/psutil/psutil/_psosx.py", line 717, in memory_maps
    return cext.proc_memory_maps(self.pid)
OSError: [Errno 22] Invalid argument

Please consider reopening this issue.

This fails multiple tests when running make test and I'm even getting a segfault at test_memory_maps. There are some things about the implementation of memory_maps on macOS that don't look right, for example:

• if info.is_submap is True, we increase the depth but also increase the address later in the loop, instead of querying the same address with the increased depth
• We don't really handle the case where vm_region_recurse_64 returns != KERN_SUCCESS. We only print a debug log and continue.
• There is an if that checks ((sizeof(buf)) <= 0 - but sizeof(buf) is constant.

I wrote a small program that goes over the memory mapping to see where the problem is here. I'm also getting "errno 22" on the last regions.

[giampaolo]

Mmm segfault is not good. I'm not familiar with that code:
cf18ae528
@whitlockjc Jeremy, any idea?

[giampaolo]

What's even worse this time is that travis nor I can reproduce it. :-\

[giampaolo]

@wiggin15 I think I may have fixed this as of #1436. Could you give it a try?

[wiggin15]

I'm still getting the same thing: OSError: [Errno 22] Invalid argument when running with python and segmentation fault when running through make test, and also when running with ipython...

Please see program I wrote here - it seems that proc_regionfilename starts failing at some point during "regular" traversal with this isolated code and I can't tell why. The vmmap tool successfully traverses the memory mappings of the process, but it's probably not using proc_regionfilename for all the regions. See for example:

Output from test program above program (snipped to where it starts failing with "errno 22"):

000000011c004000-000000011c038000 (depth=0) /usr/lib/dyld (errno=0 ret=13)
000000011c038000-000000011c061000 (depth=0) /usr/lib/dyld (errno=0 ret=13)
00007fcc0ac00000-00007fcc0ad00000 (depth=0) /usr/lib/dyld (errno=22 ret=13)
00007fcc0b000000-00007fcc0b800000 (depth=0) /usr/lib/dyld (errno=22 ret=13)

And snipped output from vmmap of the same process for these addresses:

__DATA                 000000011bfff000-000000011c004000 [   20K    20K    20K     0K] rw-/rwx SM=COW          /usr/lib/dyld
__DATA                 000000011c004000-000000011c038000 [  208K    32K    32K     0K] rw-/rwx SM=PRV          /usr/lib/dyld
MALLOC_TINY            00007fcc0ac00000-00007fcc0ad00000 [ 1024K    20K    20K     0K] rw-/rwx SM=PRV          DefaultMallocZone_0x10e359000
MALLOC_SMALL           00007fcc0b000000-00007fcc0b800000 [ 8192K    12K    12K     0K] rw-/rwx SM=PRV          DefaultMallocZone_0x10e359000

[giampaolo]

I see. Yes, I tried your program and I can't get errno=22 (EINVAL) on OSX 10.11.6 .

[giampaolo]

@wiggin15 I've found this:
https://searchcode.com/file/133107236/libr/debug/p/native/maps/darwin.c
...which calls proc_regionfilename on a certain condition:

		if (info.max_protection!=0 && !contiguous) {
                        ...
			int ret = proc_regionfilename (dbg->pid, address,
				module_name, sizeof (module_name));

On my machine info.max_protection is always 0 so proc_regionfilename is always called. Maybe on yours is not? This is the info structure:

struct vm_region_submap_info_64 {
        vm_prot_t               protection;     /* present access protection */
        vm_prot_t               max_protection; /* max avail through vm_prot */
        vm_inherit_t            inheritance;/* behavior of map/obj on fork */
        memory_object_offset_t  offset;         /* offset into object/map */
        unsigned int            user_tag;       /* user tag on map entry */
        unsigned int            pages_resident; /* only valid for objects */
        unsigned int            pages_shared_now_private; /* only for objects */
        unsigned int            pages_swapped_out; /* only for objects */
        unsigned int            pages_dirtied;   /* only for objects */
        unsigned int            ref_count;       /* obj/map mappers, etc */
        unsigned short          shadow_depth;   /* only for obj */
        unsigned char           external_pager;  /* only for obj */
        unsigned char           share_mode;     /* see enumeration */
        boolean_t               is_submap;      /* submap vs obj */
        vm_behavior_t           behavior;       /* access behavior hint */
        vm32_object_id_t        object_id;      /* obj/map name, not a handle */
        unsigned short          user_wired_count;
        unsigned int            pages_reusable;
};

Basically what I am suggesting is trying to reverse engineer this thing in order to understand under what conditions we should avoid calling proc_regionfilename.

[wiggin15]

I'm afraid that doesn't work either - info.max_protection is not 0 for any region in the process.
I'm also trying to figure out how to fix this but no luck so far... I tried to run "dtruss" to see if I can peek into what vmmap calls and what's causing the segfault, but there are some security limitations on this command so it didn't help.

This doesn't reproduce on our continuous-integration mac machine, which runs macOS 10.12.1, but it does reproduce on more development mac stations in my team, running different types of mac machines and macOS 10.14.3.

[giampaolo]

I see. Thanks for looking into this. This reminds me of open_files() on Windows (but worse). Also memory_maps on OSX only works for the current process and the children spawned by it, even as root. For anything else we get AccessDenied, which AFAICT is a first in psutil. If we can't fix it then I think I am gonna remove memory_maps on OSX in the next major version. I am not thrilled at the idea, but getting a segfault just for calling as_dict() is not acceptable.

[giampaolo]

We have 2 issues here: the segfault and the compatibility breakage. I think a reasonable way to approach this is the following:

• remove the C code and raise a deprecation warning + access denied for all processes right now in 5.6.0 (next release)
• remove the API in 6.0.0

[giampaolo]

Done in b31f3bf. Closing this out.

[giampaolo]

On a second thought also raising AccessDenied for the current process is backward incompatible. Returning an empty list is too. It looks like the only way forward is to remove this API in 5.6.0 (which is a major version BTW).

[giampaolo]

OK, memory_maps() on OSX is gone. While removing it I realized the unit-test on OSX was disabled, meaning this has been broken for a very long time. Considering the unsolvable [1], long standing and critical segfault, I consider the breakage of backward compatibility an acceptable cost. A similar case happened to open_files() on Windows which can return an incomplete result but doesn't segfault.

[1] vmmap source code is closed source, proc_regionfilename is undocumented (perhaps even broken), no solution was found

[bwoodsend]

I'm able to get around this by subprocess calling lsof -p $pid instead of vmmap, and parsing the output. I'm currently implementing it for another project put I'd rather add this implementation to psutil. Would you allow me to do this?

Below is the info yielded by parsing lsof of Python's PID.

COMMAND	PID	USER	FD	TYPE	DEVICE	SIZE/OFF	NODE	NAME
python3.7	1380	runner	cwd	DIR	1,4	352	12893113052	/Users/runner/work/pyinstaller-hook-factory/pyinstaller-hook-factory
python3.7	1380	runner	txt	REG	1,4	4192140	12893124030	/usr/local/miniconda/bin/python3.7
python3.7	1380	runner	txt	REG	1,4	19980	12893124065	/usr/local/miniconda/lib/python3.7/lib-dynload/_heapq.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	51860	12893124099	/usr/local/miniconda/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	106992	12893124060	/usr/local/miniconda/lib/python3.7/lib-dynload/_datetime.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	16436	12893124074	/usr/local/miniconda/lib/python3.7/lib-dynload/_posixsubprocess.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	33728	12893124106	/usr/local/miniconda/lib/python3.7/lib-dynload/select.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	92988	12893124082	/usr/local/miniconda/lib/python3.7/lib-dynload/_socket.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	37476	12893124111	/usr/local/miniconda/lib/python3.7/lib-dynload/zlib.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	100996	12891728561	/usr/local/miniconda/lib/libz.1.2.11.dylib
python3.7	1380	runner	txt	REG	1,4	95464	12893124046	/usr/local/miniconda/lib/python3.7/lib-dynload/_bz2.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	40320	12893124068	/usr/local/miniconda/lib/python3.7/lib-dynload/_lzma.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	154396	12891728467	/usr/local/miniconda/lib/liblzma.5.dylib
python3.7	1380	runner	txt	REG	1,4	16204	12893124098	/usr/local/miniconda/lib/python3.7/lib-dynload/grp.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	46612	12893132255	/usr/local/miniconda/lib/python3.7/site-packages/psutil/_psutil_osx.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	15524	12893132256	/usr/local/miniconda/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-darwin.so
python3.7	1380	runner	txt	REG	1,4	1568368	1152921500312899604	/usr/lib/dyld
python3.7	1380	runner	0	PIPE	0xceb898bcad398973	16384
python3.7	1380	runner	1	PIPE	0xe88391d08adf368f	65536		->0xff6e2faf57a81150
python3.7	1380	runner	2	PIPE	0x590b43d7c0ca7945	16384		->0x37d3f8e5c729229e
python3.7	1380	runner	3	PIPE	0x8f906b3253d4a8b6	16384		->0x7fc1f421784607ca

Online you may see speed concerns regarding lsof but they don't apply to lsof -p for a specific process.

[giampaolo]

Hello. I'd rather not, sorry. In general I prefer to refrain from parsing subprocess output and maintain it internally. I've been there and I know it's painful. Plus memory_maps() it's sort of low-levelish and underused.

[bwoodsend]

Plus memory_maps() it's sort of low-levelish and underused.

I use it. It's the closest I've found to a cross platform way to test shared library dependencies at run time. And I've already written the parser and it's not too ugly 😛.

[P403n1x87]

I have an example of how to iterate over the memory maps in Austin. It certainly isn't documentation, but perhaps can be of help?

https://github.com/P403n1x87/austin/blob/master/src/mac/py_proc.h

It would be nice to have memory_maps back into psutil. I don't have easy access to macOS so I can't really try to do this myself at the moment, unfortunately.

[Maxwell175]

it would be great to be able to rely on this function across all platforms.

[mzpqnxow]

I'm in a situation where I need to get a list of memory mappings- specifically, I need the file-backed mappings (e.g. the various dynamic libraries and other resources, like Java jar files- I need the full path to each on a per-process basis)

I currently do this on Linux with psutil.Process.memory_maps() but don't have this field on MacOS (darwin 20.6.0 xnu-7195.141.14) and

I understand that shelling out to lsof is not an ideal solution, but that is the only way to get this information right now as far as I'm aware (excluding the few snippets of C code here, and of course in the lsof source itself- though the MacOS memory mapping bit might be closed, I suspect it isn't)

It's probably clear, but this is the kind of data I need:

com.apple 77215             _driverkit  txt       REG                1,5     292928 1152921500311993973 /System/DriverKit/usr/lib/system/libsystem_pthread.dylib
com.apple 77215             _driverkit  txt       REG                1,5     157488 1152921500311993975 /System/DriverKit/usr/lib/system/libsystem_trace.dylib
com.apple 77215             _driverkit  txt       REG                1,5     461936 1152921500311993948 /System/DriverKit/usr/lib/libc++abi.dylib
com.apple 77216             _driverkit  txt       REG                1,5     401680 1152921500311993946 /System/DriverKit/usr/lib/libc++.dylib
com.apple 77216             _driverkit  txt       REG                1,5     140272 1152921500311993944 /System/DriverKit/usr/lib/libSystem.dylib
com.apple 77216             _driverkit  txt       REG                1,5     194336 1152921500311993951 /System/DriverKit/usr/lib/system/libcompiler_rt.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1490784 1152921500311993953 /System/DriverKit/usr/lib/system/libcorecrypto.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1009216 1152921500311993955 /System/DriverKit/usr/lib/system/libdispatch.dylib
com.apple 77216             _driverkit  txt       REG                1,5     383824 1152921500311993957 /System/DriverKit/usr/lib/system/libdyld.dylib
com.apple 77216             _driverkit  txt       REG                1,5     120608 1152921500311993959 /System/DriverKit/usr/lib/system/libmacho.dylib
com.apple 77216             _driverkit  txt       REG                1,5     138864 1152921500311993961 /System/DriverKit/usr/lib/system/libsystem_blocks.dylib
com.apple 77216             _driverkit  txt       REG                1,5    1292064 1152921500311993963 /System/DriverKit/usr/lib/system/libsystem_c.dylib

I was hoping that p.open_files() would do what I needed, but no such luck- it is woefully incomplete and only shows descriptors/files that were opened as plain file descriptors, not any that were implicitly opened and mapped by the MACH loader

I'm bummed because I switched away from a custom Python solution that married ps and lsof data together, hoping that psutil would save all my problems- but I'm a bit stuck now and may have to go back to it

Anyway, just a long-winded way to say "+1", I guess ;)

Thanks for the work you all do on this