AWS Collect: Can't pickle coroutine

[edulop91]

Hello, hope all is well!

We recently started having some issues with the aws collect connector. I'll start looking into this in more detail tomorrow, but opening this early on the off-chance that someone has already ran into/solved this. Happening on both 1.9.4 and 1.9.5. Prevents SnowAlert from ingesting any aws_collect data.

stack trace

[sfc-gh-afedorov]

Thanks looks like it's a problem from —

session = await aio_sts_assume_role(
    src_role_arn=AUDIT_ASSUMER_ARN,
    dest_role_arn=master_reader_arn,
    dest_external_id=READER_EID,
)

And something related to the AWS credentials it's getting. Googling makes it seem related to — aio-libs/aiobotocore#663

But it seems a bit of a rabbit hole that might be better solved by storing credentials however ECS does it?

[sfc-gh-afedorov]

Are you running on EKS? Happy to help reproduce if it may be helpful, but probably won't be able to find time until next week.

[edulop91]

Thanks for the quick response! Yes, running on EKS with IRSA. Pulled on that thread a bit more and found:

• Aioboto3 not working on eks. terricain/aioboto3#189
• web credentials not supported aio-libs/aiobotocore#779

The solution might be bumping dependencies. Will try to spend some time tomorrow actually looking into this and seeing if a dep version bump might fix. I'll report back if I find anything interesting.

[sfc-gh-afedorov]

There's a co-dependency on boto3 and botocore through both aioboto3 and snowflake-connector-python which makes this particular bump either a painful manual endeavor or a fun CS challenge. Let me know you know a better solution or need help.

[edulop91]

These seem to be working out ok for me:

from setuptools import setup, find_packages

setup(
    name="snowalert-runners",
    version='1.0',
    packages=find_packages(),
    include_package_data=True,
    install_requires=[
        'aiohttp[speedups]',
        'aioboto3==8.0.5',
        'fire==0.1.3',
        'jira==2.0.0',
        'PyYAML==5.3.1',
        'xmltodict==0.12.0',
        'snowflake-connector-python==2.2.10',
        'snowflake-sqlalchemy==1.2.3',
        'pandas==1.0.4',
        'pybrake==0.4.0',
        'rpy2==3.2.0',
        'pytz==2018.9',
        'slackclient==1.3.1',
        'tzlocal==1.5.1',
        'azure-common==1.1.23',
        'azure-mgmt-compute==8.0.0',
        'azure-mgmt-network==5.0.0',
        'azure-mgmt-storage==4.1.0',
        'azure-mgmt-subscription==0.5.0',
        'azure-storage-blob==12.0.0',
        'azure-storage-common==2.1.0',
        'google-api-python-client==1.8.2',
        'pyTenable==1.1.1',
        'boto3==1.14.20',
        'botocore==1.15.32',
        'twilio==6.29.4',
        'simple_salesforce==0.74.3',
        'pdpyras==4.0',
        'duo_client==4.2.3',
        'cryptography==3.0.0',
        'requests==2.23.0',
    ],
)

Should I open against 1.9.6? Are there specific tests you recommend I run?

Currently running: python runners/connectors_runner.py "AWS_COLLECT_%%"