Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leaks with Python API Operator #163

Closed
haixuanTao opened this issue Jan 2, 2023 · 1 comment · Fixed by #168
Closed

Memory leaks with Python API Operator #163

haixuanTao opened this issue Jan 2, 2023 · 1 comment · Fixed by #168

Comments

@haixuanTao
Copy link
Collaborator

haixuanTao commented Jan 2, 2023 •
edited
Loading

It seems that we have a memory leak in the Python API operator. I am not able to reproduce it in the Rust API Operator but might also be present there.

Reproduce

If you unzip the following python dataflow and run it with:

dora-coordinator --run-dataflow dataflow.yml

my_dataflow.zip

You will be able to see the memory usage growing.

How did I found out this issue?

I found out this issue as I was running the Carla leaderboard and after about 2 hours, I run out of memory. So, we might want to fix this issue for the Carla Leaderboard.

Current Investigation

I'm currently investigating the different reasons for this leak. It seems that pyo3 can easily creates copy of python object. See: https://pyo3.rs/v0.15.1/memory.html . What I know so far:

  • This is true for both iceoryx and zenoh.
  • This is coming from the subscriber.
haixuanTao added a commit that referenced this issue Jan 3, 2023
@haixuanTao
Force removal of Pyo3 Object to avoid memory leak
c951743 
This commit fix the memory leak happening in the dora API operator. This
is seemingly due to pyo3 leaking memory on object created in Rust.
Using standard `drop` did not drop the memory on the `PyBytes` included in
the `PyDict`.

See: PyO3/pyo3#1801

Fixes #163
@haixuanTao haixuanTao mentioned this issue Jan 3, 2023
Merged
@phil-opp
Copy link
Collaborator

phil-opp commented Jan 4, 2023

See PyO3/pyo3#2853

haixuanTao added a commit that referenced this issue Jan 14, 2023
@haixuanTao
Force removal of Pyo3 Object to avoid memory leak
71078bd 
This commit fix the memory leak happening in the dora API operator. This
is seemingly due to pyo3 leaking memory on object created in Rust.
Using standard `drop` did not drop the memory on the `PyBytes` included in
the `PyDict`.

See: PyO3/pyo3#1801

Fixes #163
haixuanTao added a commit that referenced this issue Jan 16, 2023
@haixuanTao
Package DoraStatus into dora python package.
1952195 
To prepare for future feature, this commit initiate a `__init__.py` file
that makes the dora python node api, a mixed python/rust project.

Any python function or variable included in the `__init__.py` will be
distributed along rust function declared in `src/lib.rs`.

#163 has shown the limitation on
depending too much on `pyo3`.
@haixuanTao haixuanTao mentioned this issue Jan 16, 2023
Merged
haixuanTao added a commit that referenced this issue Jan 18, 2023
@haixuanTao
Package DoraStatus into dora python package.
160472e 
To prepare for future feature, this commit initiate a `__init__.py` file
that makes the dora python node api, a mixed python/rust project.

Any python function or variable included in the `__init__.py` will be
distributed along rust function declared in `src/lib.rs`.

#163 has shown the limitation on
depending too much on `pyo3`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Force removal of Pyo3 Object to avoid memory leak dora-rs/dora
2 participants
@phil-opp @haixuanTao