Real time network IO counters

[giampaolo]

From [email protected] on February 18, 2011 07:04:31

What steps will reproduce the problem?  
1. if can show the Process's IO, i means: psutil support a function
like 
def get_disk_io(pid=None):
    return (write_speed, read_speed)

def get_net_io(pid=None):
    return (recv_speed, send_speed) 2. 3. 

What is the expected output?  


What do you see instead?  


What version of psutil are you using? What Python version?  


On what operating system? Is it 32bit or 64bit version?  
Please provide any additional information below.

Original issue: http://code.google.com/p/psutil/issues/detail?id=150

[giampaolo]

From g.rodola on February 18, 2011 14:03:10

There's already something for process IO counters (see issue 64 ).
By using that, get_disk_io() might be implemented, although I'm not sure 
whether something so high level fits well in a python lib.

As for get_net_io() I'm not sure what we might do. I doubt that kind of 
information is supported on many platforms.

[giampaolo]

From g.rodola on June 11, 2011 11:28:46

Personal reminder.
On linux network IO stats can be retrieved via /proc/net/dev and 
/proc/diskstats for the disk IO.

[giampaolo]

From g.rodola on June 11, 2011 13:38:14

On FreeBSD checkout man devstat and iostat cmdline utility.

[giampaolo]

From [email protected] on June 23, 2011 11:17:51

If this is something you want to implement, I have the network I/O implemented 
for Darwin.  Just need  a reason to create a patch.

[giampaolo]

From g.rodola on June 24, 2011 02:22:22

Please do.
This is something I'd really love to see added.

[giampaolo]

From g.rodola on June 27, 2011 01:43:45

Changing this issue so that we track network IO counters only; disk IO counters 
will be tracked into a separate issue.
What I have in mind is something like this:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=...)

...or, if the the platform has support for it, also include the number of 
packets sent/received:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=..., pkts_sent=..., pkts_recv=...)

Summary: Real time network IO counters

[giampaolo]

From [email protected] on July 21, 2011 16:29:27

This is not possible on Windows without using ETW kernel event tracing, and you 
don't want to do that.

[giampaolo]

From [email protected] on September 06, 2011 21:12:55

I don't see any movement on this.  Is this still on the table?  I could 
implement OS X pretty quickly but I don't see any stubs to base my work off of. 
 I don't mind using the above "psutil.network_io_counters()" and it would 
return the following:

(packets_in, packets_out, bytes_in, bytes_out)

Just let me know.  OS X implementation is working locally so support is just a 
patch away once we can get a few suggestions made known.

[giampaolo]

From g.rodola on September 07, 2011 00:42:57

As I said earlier in the thread, please do provide a patch.
The API I had in mind still stands:

>>> psutil.network_io_counters()
iostat(bytes_sent=..., bytes_recv=..., pkts_sent=..., pkts_recv=...)

Unfortunately I haven't had much time lately, and this issue requires a lot of 
effort because the implementation is completely different on every platform.
Having a patch for OSX would surely speed up the process, altough I'm not sure 
when I'll have the time to look into other platform implementations and start 
writing some actual code.

[giampaolo]

From [email protected] on September 07, 2011 09:35:12

I've submitted patches in the past but usually against an already in place API. 
 It just felt right to make a last ditch effort for some clarity before doing 
things my way only to find out you had something else in mind.  Expect a patch 
sometime today.

[giampaolo]

From g.rodola on September 07, 2011 09:49:34

Yeah, sure.
The most important thing is the C implementation which extracts the information 
from the OS. The upper python layer and API aren't really important at this stage.

[giampaolo]

From [email protected] on September 08, 2011 16:18:18

Attached is a patch that adds network_io_counters for Mac OS X, complete with 
tests.  Since I'm adding a new API I might not had done things they way you 
wanted but I did my best to merge your conventions with the information in this 
patch.  Right now, this works just like mentioned above:

>>> import psutil
>>> psutil.network_io_counters()
iostat(bytes_sent=179001106, bytes_rcvd=1633284276, packets_sent=1564655, 
packets_rcvd=1916980)

Let me know if there is anything I can do to clean this patch up so it can be 
submitted.  Also, I really think we could add another feature to do 
per-interface stats and interface-specific stats, kind of how it's done for 
partition/disk usage.  I was gonna implement that now but I figured I'd wait 
until we had consensus on getting the totals in first.

Attachment: psutil_issue_150.diff

[giampaolo]

From [email protected] on September 08, 2011 21:24:10

I realize now that the test I wrote actually parses the output of "netstat -bi" 
to validate the psutil output.  I started thinking we could use that logic to 
implement this for supporting OSes.  Just an idea...

[giampaolo]

From g.rodola on September 09, 2011 13:44:46

I haven't tried it yet but at a first look your patch looks fine to me.
Do you know if this might work on FreeBSD as well?
I'd say let's leave per-interface alone for now: we're not sure whether we can 
do that on all platforms.

Status: Started

[giampaolo]

From [email protected] on September 09, 2011 13:58:34

I had the same question after doing it OS X specific.  Based on the header 
files I used for reference, it's possible but I don't have a FreeBSD box to 
test on.  I guess I could install one on a VM.

[giampaolo]

From g.rodola on September 09, 2011 14:09:19

I'll try to see whether your patch works on FreeBSD so that you won't have to 
install it.

As for per-interface, I'm looking at Linux implementation now and noticed that 
data can naturally be gathered in a per-interface fashion.
Considering that FreeBSD implemention should be similar to OSX's maybe it makes 
sense to rethink our API right now and return a list of namedtuples for every 
interface including the interface name. Example:

>>> psutil.network_io_counters()
[iostat(interface='eth0', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='eth1', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='lo', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 ...]

[giampaolo]

From [email protected] on September 09, 2011 14:16:24

I was thinking the same thing.  It would be nice to be able to get a list when 
necessary but not make it harder to get a total.  I don't see any other place 
where we do this type of thing.  The closest is psutil.cpu_times() where you 
have a function argument that dictates returning a list of results vs. 
returning a total.  I guess it's up to you.  Just let me know what you'd like 
me to do.  If you can tell me the Python interface(s) for the function(s), I'll 
update my patch accordingly.  Let me know the FreeBSD support too and I'll 
refactor if necessary.  I guess you can just include me where you want. ;)

[giampaolo]

From g.rodola on September 09, 2011 14:34:43

I think in this case it makes more sense to return a list of nametuples as the default.
Perhaps we can add a "total=False" argument and return single a namedtuple 
instead, but this should be done in python, at the highest level.

> If you can tell me the Python interface(s) for the function(s), 
> I'll update my patch accordingly. 

I'd be for sticking with this API:

>>> psutil.network_io_counters()
[iostat(interface='eth0', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='eth1', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 iostat(interface='lo', bytes_sent=1, bytes_rcvd=2, packets_sent=3, 
packets_rcvd=4),
 ...]

...therefore the C function should return a list of (name, bsent, brcv, psent, 
prcvd) tuples.
It is fine for the C extension to return plain tuples. You can then convert 
them to namedtuples in python (psutil/_psosx.py).

[giampaolo]

From g.rodola on September 09, 2011 14:43:37

Linux implementation committed as r1117 .

Labels: -Type-Defect Type-Enhancement OpSys-Linux Milestone-0.3.1 Progress-1in4

[giampaolo]

From g.rodola on September 09, 2011 14:58:14

total parameter added in r1118 .

[giampaolo]

From [email protected] on September 09, 2011 15:17:11

I'll update my patch for OS X based on the updated code in Subversion.

[giampaolo]

From [email protected] on September 09, 2011 16:07:53

I've updated my patch.  Let me know if you need me to change anything.  Also, 
for testing I don't see any real test for this new stuff.  I didn't port the 
test I wrote in the original patch because I wasn't sure why it wasn't used in 
your tests.  I can port it if need be but applying my attached patch shouldn't 
be held up while I do that, assuming you're gonna apply it of course.

Attachment: psutil_issue_150_v2.diff

[giampaolo]

From g.rodola on September 10, 2011 08:57:04

OSX patch committed as r1119 .
I also gave credits to you (you are Jeremy Whitlock, right?).
Let me know if that's fine with you.
I tried to adapt your original OSX test but it keeps failing.

[giampaolo]

From g.rodola on September 10, 2011 09:13:43

I tried to copy & paste your patch on FreeBSD and this is what I get:

psutil/_psutil_bsd.c: In function 'get_network_io_counters':
psutil/_psutil_bsd.c:761: error: 'NET_RT_IFLIST2' undeclared (first use in this function)
psutil/_psutil_bsd.c:761: error: (Each undeclared identifier is reported only once
psutil/_psutil_bsd.c:761: error: for each function it appears in.)
psutil/_psutil_bsd.c:783: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:785: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:785: error: 'RTM_IFINFO2' undeclared (first use in this function)
psutil/_psutil_bsd.c:787: error: invalid use of undefined type 'struct 
if_msghdr2'
psutil/_psutil_bsd.c:795: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:796: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:797: error: dereferencing pointer to incomplete type
psutil/_psutil_bsd.c:798: error: dereferencing pointer to incomplete type
error: command 'cc' failed with exit status 1

I fixed 'NET_RT_IFLIST2' and 'RTM_IFINFO2' errors by including sys/socket.h and 
net/route.h but other errors remain.

[giampaolo]

From g.rodola on September 10, 2011 09:54:25

Update about OSX: it seems bytes/packets received are always 0.
The script in attachment shows the network usage in real time.
On linux it seems to work fine (while I download a big file with the browser it 
more or less reflects the number of bytes shown by the browser download 
manager) but on OSX it clearly doesn't work as download bytes are always 0.

Labels: -Progress-1in4 Progress-2in4

Attachment: network_io.py

[giampaolo]

From [email protected] on September 10, 2011 13:21:47

I don't see the same issue with bytes/packets being 0 except for certain 
devices, which also report the same metrics via "netstat -bi".  Here is an 
example of what I have, my code prior to your applying my patch:

>>> psutil.network_io_counters()
[iostat(interface='lo0', bytes_sent=84498890, bytes_rcvd=84491065, 
packets_sent=430981, packets_rcvd=430991), iostat(interface='gif0', 
bytes_sent=0, bytes_rcvd=0, packets_sent=0, packets_rcvd=0), 
iostat(interface='stf0', bytes_sent=0, bytes_rcvd=0, packets_sent=0, 
packets_rcvd=0), iostat(interface='en0', bytes_sent=342, bytes_rcvd=0, 
packets_sent=0, packets_rcvd=0), iostat(interface='en1', bytes_sent=220966030, 
bytes_rcvd=-1835371131, packets_sent=2140419, packets_rcvd=2745809), 
iostat(interface='fw0', bytes_sent=346, bytes_rcvd=0, packets_sent=0, 
packets_rcvd=0), iostat(interface='vboxnet0', bytes_sent=0, bytes_rcvd=0, 
packets_sent=0, packets_rcvd=0)]

I'm not understanding the problem.  As for my name in the patch, that's fine.  
I usually use "Jeremy Whitlock <[email protected]>" for projects I 
contribute to but I wasn't specific so no big deal.  As I was about to submit 
this comment, I see your script and I am able to see non-0 bytes sent/rcvd.  I 
do see a formatting issue where the k/B/etc. aren't always being displayed or 
are being displayed "stacked"  (0.5kk).  So, is there a problem with the 
network_io_counters() on OS X or with your implementation of the script?  I can 
look further into it regardless.

[giampaolo]

From g.rodola on September 13, 2011 10:25:48

FreeBSD implementation checked in as r1120 .
As for the script problem on OSX I'll take a deeper look to figure out what's 
wrong. Maybe it's my fault.

Labels: -Progress-2in4 Progress-4in4

[giampaolo]

From g.rodola on September 13, 2011 11:16:09

Ok, problem was numbers types which are expressed by using a u_int64_t type.
This is now fixed in r1121 .

[giampaolo]

From g.rodola on September 14, 2011 12:54:43

It seems we can get this information also on Windows without going crazy (at 
least, this is my impression) by using GetIfEntry(): 
http://stackoverflow.com/questions/5213629/how-to-enumerate-all-available-network-interfaces
 http://msdn.microsoft.com/en-us/library/aa365939(v=vs.85).aspx

Labels: -Progress-4in4 Progress-3in4

[giampaolo]

From [email protected] on September 21, 2011 12:59:36

Would you like to refactor the call/result of network_io_counters to be like 
what you suggested/implemented in disk_io_counters?  ( 
https://code.google.com/p/psutil/issues/detail?id=206#c4 )  I can do the work 
if you've not started already.

[giampaolo]

From [email protected] on September 21, 2011 13:45:53

I have this implemented across all currently-implemented OSes.  Here is the 
output of both disk_io_counters and network_io_counters for comparison.  Some 
minor cleanup and I'll submit a patch.

>>> import psutil
>>> psutil.disk_io_counters()
iostat(read_count=349825L, write_count=897726L, read_bytes=5204112896L, 
write_bytes=22961667072L, read_time=1631324184229L, write_time=4326592882937L)
>>> psutil.disk_io_counters(perdisk=True)
{'disk0': iostat(read_count=349826L, write_count=897726L, 
read_bytes=5204116992L, write_bytes=22961667072L, read_time=1631336311003L, 
write_time=4326592882937L)}
>>> psutil.network_io_counters()
iostat(bytes_sent=342302872L, bytes_rcvd=5570448096L, packets_sent=3600502L, 
packets_rcvd=5141945L)
>>> psutil.network_io_counters(pernic=True)
{'gif0': iostat(bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, 
packets_rcvd=0L), 'en0': iostat(bytes_sent=342L, bytes_rcvd=0L, 
packets_sent=0L, packets_rcvd=0L), 'en1': iostat(bytes_sent=317063256L, 
bytes_rcvd=5545205708L, packets_sent
=3522199L, packets_rcvd=5063634L), 'lo0': iostat(bytes_sent=25240482L, 
bytes_rcvd=25243127L, packets_sent=78312L, packets_rcvd=78317L), 'stf0': 
iostat(bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L), 
'vboxnet0': iostat(
bytes_sent=0L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L), 'fw0': 
iostat(bytes_sent=346L, bytes_rcvd=0L, packets_sent=0L, packets_rcvd=0L)}

[giampaolo]

From [email protected] on September 21, 2011 13:53:57

Attached is a patch that will refactor network_io_counters to be more like 
disk_io_counters in how it's called, total being the default, and the return 
type of dict instead of list.  During this I also found that 
network_io_counters on OS X was leaking memory and I've fixed this as well.

Attachment: psutil_issue_150_refactor.diff

[giampaolo]

From [email protected] on September 21, 2011 15:37:06

I have created Issue 210 for tracking the enhanced tests required to prove 
psutil.network_io_counters() is working properly.

[giampaolo]

From g.rodola on September 22, 2011 04:21:12

Your patch looks good to me. You can go on and commit AFAICT.

[giampaolo]

From [email protected] on September 22, 2011 08:04:39

As of r1132 , this has been committed to source control.

[giampaolo]

From [email protected] on October 13, 2011 13:01:59

Windows implementation added in r1153 .

[giampaolo]

From g.rodola on October 21, 2011 16:34:38

Status: FixedInSVN
Labels: -Progress-3in4 Progress-4in4

[giampaolo]

From g.rodola on October 21, 2011 16:44:17

Labels: -Milestone-0.3.1

[giampaolo]

From g.rodola on October 21, 2011 16:45:26

Labels: Milestone-0.4.0

[giampaolo]

From g.rodola on October 28, 2011 20:44:14

Status: Fixed

[giampaolo]

From g.rodola on March 02, 2013 03:59:12

Updated csets after the SVN -> Mercurial migration: r1117 == revision 
99bc14825f65 r1118 == revision 3a92e8ef7a71 r1119 == revision b404050fd676 
r1120 == revision 08e02cf133ba r1121 == revision ac004112a205 r1132 == revision 
516510dc23a3 r1153 == revision df719375414c

[giampaolo]

From [email protected] on June 15, 2013 20:49:00

Are there any plans to extend this support to the Process class as 
"get_net_io", in the same fashion that exists for "get_io_counters"? Would be 
really awesome.

[giampaolo]

From g.rodola on June 16, 2013 02:33:11

You mean per-process net io counters? I don't think there are platforms 
exposing that kind of info except maybe Windows, if my memory serves me right.

Labels: -OpSys-Linux

[giampaolo]

From [email protected] on June 16, 2013 02:49:08

Ya, sorry about that. After I made that request, I researched it a bit and came 
to find the same ruling. Makes perfect sense now why that feature doesn't exist :-)