Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Flaky SEGFAULT while testing exec_command #57

Open
webknjaz opened this issue Jun 1, 2020 · 3 comments · Fixed by #60 or #94 · May be fixed by #658
Open

[BUG] Flaky SEGFAULT while testing exec_command #57

webknjaz opened this issue Jun 1, 2020 · 3 comments · Fixed by #60 or #94 · May be fixed by #658
Assignees
@ganeshrn
Labels
bug Something isn't working
Milestone
Jan 2021

Comments

@webknjaz
Copy link
Member

webknjaz commented Jun 1, 2020 •
edited
Loading

https://github.com/ansible/pylibssh/runs/725098816?check_suite_focus=true#step:10:146:

[gw1] [ 16%] PASSED tests/unit/channel_test.py::test_exec_command Fatal Python error: Segmentation fault

Thread 0x00007f13f32a4700 (most recent call first):
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 400 in read
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 432 in from_io
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn

Current thread 0x00007f13f58cf680 (most recent call first):
  File "/home/runner/work/pylibssh/pylibssh/tests/unit/channel_test.py", line 22 in ssh_channel
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/fixtures.py", line 800 in _teardown_yield_fixture
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/fixtures.py", line 871 in finish
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 318 in _callfinalizers
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 328 in _teardown_with_finalization
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 310 in _pop_and_teardown
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 350 in _teardown_towards
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 342 in teardown_exact
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 148 in pytest_runtest_teardown
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 101 in runtestprotocol
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 88 in run_one_test
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 71 in pytest_runtestloop
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 261 in <module>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084 in executetask
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060 in serve
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
[gw1] node down: Not properly terminated
[gw1] [ 91%] FAILED tests/unit/channel_test.py::test_exec_command 
replacing crashed worker gw1
@webknjaz webknjaz added the bug Something isn't working label Jun 1, 2020
@webknjaz webknjaz changed the title Flaky SEGFAULT while testing exec_command [BUG] Flaky SEGFAULT while testing exec_command Jun 1, 2020
@ganeshrn
Copy link
Member

Fixed by PR #60

@webknjaz
Copy link
Member Author

More segfaults appeared recently under both macos and ubuntu:

@webknjaz webknjaz reopened this Jun 18, 2020
webknjaz added a commit that referenced this issue Jun 18, 2020
@webknjaz
Mark test_exec_command as xfail
1348b85 
It causes SEGFAULTs:
#57
ganeshrn added a commit to ganeshrn/pylibssh that referenced this issue Jul 11, 2020
@ganeshrn
Fix exec_command issue
f784e8f 
Fixes ansible#57
Fixes ansible#56

*  Use channel created with Channel class
   instead of creating a new channel with
   exec_command()
@ganeshrn ganeshrn mentioned this issue Jul 11, 2020
Merged
ganeshrn added a commit to ganeshrn/pylibssh that referenced this issue Jul 13, 2020
@ganeshrn
Fix exec_command issue
c79c2ca 
Fixes ansible#57
Fixes ansible#56

*  Use channel created with Channel class
   instead of creating a new channel with
   exec_command()
This was linked to pull requests Jul 30, 2020
Fix exec_command segfault issue #60
Merged
Fix exec_command issue #94
Merged
@ganeshrn ganeshrn reopened this Oct 21, 2020
@webknjaz webknjaz added this to the Jan 2021 milestone Oct 21, 2020
@webknjaz webknjaz mentioned this issue Jul 29, 2022
Open
@webknjaz webknjaz mentioned this issue Feb 3, 2024
Merged
@webknjaz

This comment was marked as outdated.

Sign in to view
@Jakuje Jakuje mentioned this issue Nov 14, 2024
Open
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
10be284 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
e174293
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
2e2d4cf 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
63b8ab7 
Signed-off-by: Jakub Jelen <[email protected]>
@Jakuje Jakuje linked a pull request Nov 14, 2024 that will close this issue
Open
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
17e177d 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 14, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
fffd344 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 15, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
d2fcc9c 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 15, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
88dbb88 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 19, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
398e3af 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 19, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
f0a908a 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
cdeba7e 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
4e92723 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
0020cc8 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
969827d 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
channels: Use dynamically allocated memory for callbacks
7d59c3e 
Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
e62357a 
Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
channels: Avoid reusing stack-local variables for callbacks
d40b52a 
Previously, stack-local variables hold the callbacks. This
worked as long as the call stack was not overridden by other
function calls or as long as the server closed the channel
very early. In other cases, the delayed close in libssh
could result in calling the callbacks even after we believed
the channel was freed, causing invalid memory access and crashes.

Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
c8236e9 
Removes the failed marker as it works now, as well as the forked fixture
as it makes it hard to debug possible issues.

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
channels: Avoid reusing stack-local variables for callbacks
9ea5497 
Previously, stack-local variables hold the callbacks. This
worked as long as the call stack was not overridden by other
function calls or as long as the server closed the channel
very early. In other cases, the delayed close in libssh
could result in calling the callbacks even after we believed
the channel was freed, causing invalid memory access and crashes.

Fixes ansible#57

Signed-off-by: Jakub Jelen <[email protected]>
Jakuje added a commit to Jakuje/pylibssh that referenced this issue Nov 20, 2024
@Jakuje
tests: Adjusted reproducer for ansible#57
61da542 
Removes the failed marker as it works now.

Signed-off-by: Jakub Jelen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ganeshrn ganeshrn

Labels
bug Something isn't working
Projects
None yet
Milestone
Jan 2021
2 participants
@webknjaz @ganeshrn