estuary-cdk: additional SIGQUIT logging & patching requests.Session.send for Airbyte imports
#2380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…connectors There have been instances where imported Airbyte connectors just hang without any apparent network traffic. Debugging this has been difficult, so this commit adds some additional logging to help troubleshoot what coroutines exist when a SIGQUIT is injected.
The `requests` package does not have a default timeout for its `send` method. I did not see any code in version 0.52.10 of the `airbyte_cdk` (the version almost all of the imported connectors use) that passes a timeout argument, so I'm patching the `send` method to use a default timeout if one isn't provided. This is to help troubleshoot & hopefully avoid imported Airbyte connectors that infrequently hang without any network traffic.
Alex-Bair
force-pushed
the
bair/cdk-sigquit-logging-and-requests-patch
branch
from
b03b02c to
4e17524
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
There have been tricky, intermittent cases of imported Airbyte connectors hanging with essentially no network traffic. I've added some additional logging when a
SIGQUITis injected & in theCaptureShim's methods to help debug when these imports hang.I also have a theory that since Airbyte imports use the
requestspackage to make synchronous HTTP requests, therequestspackage doesn't have any default timeouts, and I can't find anywhere in theairbyte_cdkwhere a timeout is specified forrequests, that setting a timeout forrequestsmight prevent Airbyte imports from hanging until the connector is restarted after 24 hours. I've maderequests_session_send_patch.pyto patch in a default timeout, and imported it in all Airbyte imports. I'm not certain the missing timeouts are the problem, but it's currently the best, actionable theory I have.We've only observed
source-zendesk-supportandsource-jira-nativehanging like this so far, but I chose to patchrequests.Session.sendfor all currently imported connectors to avoid playing whack-a-mole with imports that we haven't seen have this issue yet. If any imports still hang forDEFAULT_TIMEOUTseconds, then we'll know that the issue isn't due to a misingrequeststimeout.Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
Confirmed Airbyte imports can still make HTTP requests & receive responses. Confirmed that using
requeststo send an HTTP request to a server that accepts the request but doesn't send data back will timeout afterDEFAULT_TIMEOUTseconds.This change is