Fix all Amazon Provider MyPy errors

[potiuk]

Part of #19891

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

All green now!

[potiuk]

With this one (and some merged yesterday) will get to 29 MyPy errors left !

[potiuk]

Looks like going to be Green :)

[potiuk]

Hey @ashb - all should be good here. (and really close to finish MyPy).

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[JorgenG]

@potiuk This has broken the ability to run a glue job without specifying a script location on AWS Glue.

Previously it was allowed that this value was None, resulting in using the default script value configured in the glue service itself. Now, the only way to escape the local file upload is to start with s3://, which then obviously will break further down. Can you please revert this change for aws glue?

EDIT: What are sensible next steps? (Improving my language)

[potiuk]

@potiuk This has broken the ability to run a glue job without specifying a script location on AWS Glue.

Previously it was allowed that this value was None, resulting in using the default script value configured in the glue service itself. Now, the only way to escape the local file upload is to start with s3://, which then obviously will break further down. Can you please revert this change for aws glue?

Feel free to provide a good fix for it. I think you are the one who can test it eeasily, so likely it's best if you fix it. Can you please make a fix @JorgenG ?

[JorgenG]

@potiuk I think can jump on that early next week. I am not familiar with the MyPy tooling though. I assume the origin of this change is due to the parameter not being optional? Would I then add back in a "violation" of your MyPy fixes? 🤔

[potiuk]

@potiuk This has broken the ability to run a glue job without specifying a script location on AWS Glue.
Previously it was allowed that this value was None, resulting in using the default script value configured in the glue service itself. Now, the only way to escape the local file upload is to start with s3://, which then obviously will break further down. Can you please revert this change for aws glue?

Feel free to provide a good fix for it. I think you are the one who can test it eeasily, so likely it's best if you fix it. Can you please make a fix @JorgenG ?

Alternatively you can open an issue, and we can mark it as good first issue if it will be justified, and hopefully it will get fixed - but yo need to describe in detail the scenario - i think the one liner description above is not enough of the doscription what exactly got broken.

[JorgenG]

@potiuk Totally makes sense. I'll give it a go myself and if not I'll summarize as a better described issue. Thanks a lot for responding quickly! 👍

[potiuk]

@potiuk I think can jump on that early next week. I am not familiar with the MyPy tooling though. I assume the origin of this change is due to the parameter not being optional? Would I then add back in a "violation" of your MyPy fixes? thinking

No idea- maybe some of the typing in Amazon was wrong, I do not know which exact part of it you refer to so it's hard to say.

[potiuk]

@potiuk I think can jump on that early next week. I am not familiar with the MyPy tooling though. I assume the origin of this change is due to the parameter not being optional? Would I then add back in a "violation" of your MyPy fixes? thinking

Or maybe the way it worked before was accidental and not really supported/expected - in which case it is a no issue and it was simply wrongly used - maybe there is another "proper" way of doing it - this is what I am referring to as "you are the one that can fix it". While I think MyPy changes were neutral, it could be it is really a problem with underlying library behaviour when None is passed. That's why it needs a specific explanation and reproducible case in issue.

[potiuk]

Ah - I saw your detailed comment now - sure I will make a fix in a moment :)

[potiuk]

I did not realize you commented on the exact line - I only saw this comment and did not know what it referred to 🤣

[JorgenG]

Wow, thanks. That is very kind! Let me know if you want any more elaboration or investigation here. As you mention, I am not sure what is "correct" usage of this. However as an example from the boto3 start_job_run, https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.start_job_run,

There does not seem to be any requirement to provide this. I think script location is passed as an argument.

EDIT: Also as can be seen here; https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

If providing script location, it overrides what is set on the job. This wording IMO indicates that it should be possible to not provide it. 👍