Bug fix in AWS glue operator related to num_of_dpus #19787

[Ritika-Singhal]

This pull request is to fix an inconsistency and bug found during testing in #21348. This is an additional fix to the issue solved in #19787.

This pull request is in accordance with #21348 (comment)
In the AWSGlueJobOperator (file path: airflow/providers/amazon/aws/operators/glue.py), I have changed the default value of num_of_dpus from 6 to None.

Without this fix, it becomes inconsistent with the logic changed in the airflow/providers/amazon/aws/hooks/glue.py. In the AWSGlueJobOperator script, if the user does not want to specify the num_of_dpus, then also the __init__ function is default initializing it to 6 (which gives the error below). Currently, to get around it, the user has to specify num_of_dpus=None to run without error. This fix solves this issue.

File "/Users/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task result = execute_callable(context=context)
File "/Users/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 123, in execute create_job_kwargs=self.create_job_kwargs, 
File "/Users/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 74, in __init__ raise ValueError("Cannot specify num_of_dpus with custom WorkerType") 
ValueError: Cannot specify num_of_dpus with custom WorkerType [2022-02-05, 16:57:41 UTC] {local_task_job.py:154}
INFO - Task exited with return code 1

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.