Refacter docs fragments and synchronize modules and plugins (#1248)

Refacter docs fragments and synchronize modules and plugins

SUMMARY
Move some of our docs fragments about to better reflect what they're for and the fact that we now have our own "amazon.aws..." namespace we can use.

Synchronises the modules and plugins docs fragments (as far as possible)
Updates plugin docs fragments to reflect the previously announced env variable deprecations

Note: the amazon.aws.common.plugins fragment will necessitate some changes to the inventory and lookup plugins before use.  (See also #1225).  However, those changes will then bring them in line with the other modules
ISSUE TYPE

Docs Pull Request

COMPONENT NAME
plugins/docs_fragments
ADDITIONAL INFORMATION

Reviewed-by: Alina Buzachis <None>

changelogs/fragments/1248-docs.yml

@@ -0,0 +1,10 @@
+minor_changes:
+- docs_fragments - ``amazon.aws.boto3`` fragment now pulls the botocore version requirements from ``module_utils.botocore`` (https://github.com/ansible-collections/amazon.aws/pull/1248).
+- docs_fragments - common parameters for modules and plugins have been synchronised and moved to ``amazon.aws.common.modules`` and ``amazon.aws.common.plugins`` (https://github.com/ansible-collections/amazon.aws/pull/1248).
+- docs_fragments - region parameters for modules and plugins have been synchronised and moved to ``amazon.aws.region.modules`` and ``amazon.aws.region.plugins`` (https://github.com/ansible-collections/amazon.aws/pull/1248).
+
+deprecated_features:
+- docs_fragments - ``amazon.aws.aws`` docs fragment has been deprecated please use ``amazon.aws.common.modules`` instead (https://github.com/ansible-collections/amazon.aws/pull/1248).
+- docs_fragments - ``amazon.aws.ec2`` docs fragment has been deprecated please use ``amazon.aws.region.modules`` instead (https://github.com/ansible-collections/amazon.aws/pull/1248).
+- docs_fragments - ``amazon.aws.aws_credentials`` docs fragment has been deprecated please use ``amazon.aws.common.plugins`` instead (https://github.com/ansible-collections/amazon.aws/pull/1248).
+- docs_fragments - ``amazon.aws.aws_region`` docs fragment has been deprecated please use ``amazon.aws.region.plugins`` instead (https://github.com/ansible-collections/amazon.aws/pull/1248).

docs/docsite/rst/dev_guidelines.rst

@@ -297,10 +297,10 @@ Common Documentation Fragments for Connection Parameters
 There are four :ref:`common documentation fragments <module_docs_fragments>`
 that should be included into almost all AWS modules:
 
-* ``aws`` - contains the common boto3 connection parameters
-* ``ec2`` - contains the common region parameter required for many AWS modules
 * ``boto3`` - contains the minimum requirements for the collection
-* ``tags`` - contains the common tagging parameters used by many AWS modules
+* ``common.modules`` - contains the common boto3 connection parameters
+* ``region.modules`` - contains the common region parameter required for many AWS APIs
+* ``tags`` - contains the common tagging parameters
 
 These fragments should be used rather than re-documenting these properties to ensure consistency
 and that the more esoteric connection options are documented. For example:
@@ -311,9 +311,31 @@ and that the more esoteric connection options are documented. For example:
    module: my_module
    # some lines omitted here
    extends_documentation_fragment:
-       - amazon.aws.aws
-       - amazon.aws.ec2
        - amazon.aws.boto3
+       - amazon.aws.common.modules
+       - amazon.aws.region.modules
+   '''
+
+Other plugin types have a slightly different document fragment format, and should use
+the following fragments:
+
+* ``boto3`` - contains the minimum requirements for the collection
+* ``common.plugins`` - contains the common boto3 connection parameters
+* ``region.plugins`` - contains the common region parameter required for many AWS APIs
+* ``tags`` - contains the common tagging parameters
+
+These fragments should be used rather than re-documenting these properties to ensure consistency
+and that the more esoteric connection options are documented. For example:
+
+.. code-block:: python
+
+   DOCUMENTATION = '''
+   module: my_plugin
+   # some lines omitted here
+   extends_documentation_fragment:
+       - amazon.aws.boto3
+       - amazon.aws.common.plugins
+       - amazon.aws.region.plugins
    '''
 
 .. _ansible_collections.amazon.aws.docsite.dev_exceptions:

plugins/doc_fragments/aws.py

@@ -1,133 +1,20 @@
 # -*- coding: utf-8 -*-
-
-# Copyright: (c) 2014, Will Thames <[email protected]>
+# (c) 2022 Red Hat Inc.
+#
+# This file is part of Ansible
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
+from .common import ModuleDocFragment as CommonFragment
 
-class ModuleDocFragment(object):
+#
+# The amazon.aws.aws docs fragment has been deprecated,
+# please migrate to amazon.aws.common.modules.
+#
 
-    # AWS only documentation fragment
-    DOCUMENTATION = r'''
-options:
-  access_key:
-    description:
-      - AWS access key ID.
-      - See the AWS documentation for more information about access tokens
-        U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
-      - The C(AWS_ACCESS_KEY_ID), C(AWS_ACCESS_KEY) or C(EC2_ACCESS_KEY)
-        environment variables may also be used in decreasing order of
-        preference.
-      - The I(aws_access_key) and I(profile) options are mutually exclusive.
-      - The I(aws_access_key_id) alias was added in release 5.1.0 for
-        consistency with the AWS botocore SDK.
-      - The I(ec2_access_key) alias has been deprecated and will be removed in a
-        release after 2024-12-01.
-      - Support for the C(EC2_ACCESS_KEY) environment variable has been
-        deprecated and will be removed in a release after 2024-12-01.
-    type: str
-    aliases: ['aws_access_key_id', 'aws_access_key', 'ec2_access_key']
-  secret_key:
-    description:
-      - AWS secret access key.
-      - See the AWS documentation for more information about access tokens
-        U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
-      - The C(AWS_SECRET_ACCESS_KEY), C(AWS_SECRET_KEY), or C(EC2_SECRET_KEY)
-        environment variables may also be used in decreasing order of
-        preference.
-      - The I(secret_key) and I(profile) options are mutually exclusive.
-      - The I(aws_secret_access_key) alias was added in release 5.1.0 for
-        consistency with the AWS botocore SDK.
-      - The I(ec2_secret_key) alias has been deprecated and will be removed in a
-        release after 2024-12-01.
-      - Support for the C(EC2_SECRET_KEY) environment variable has been
-        deprecated and will be removed in a release after 2024-12-01.
-    type: str
-    aliases: ['aws_secret_access_key', 'aws_secret_key', 'ec2_secret_key']
-  session_token:
-    description:
-      - AWS STS session token for use with temporary credentials.
-      - See the AWS documentation for more information about access tokens
-        U(https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
-      - The C(AWS_SESSION_TOKEN), C(AWS_SECURITY_TOKEN) or C(EC2_SECURITY_TOKEN)
-        environment variables may also be used in decreasing order of preference.
-      - The I(security_token) and I(profile) options are mutually exclusive.
-      - Aliases I(aws_session_token) and I(session_token) were added in release
-        3.2.0, with the parameter being renamed from I(security_token) to
-        I(session_token) in release 6.0.0.
-      - The I(security_token), I(aws_security_token), and I(access_token)
-        aliases have been deprecated and will be removed in a release after
-        2024-12-01.
-      - Support for the C(EC2_SECRET_KEY) and C(AWS_SECURITY_TOKEN) environment
-        variables has been deprecated and will be removed in a release after
-        2024-12-01.
-    type: str
-    aliases: ['aws_session_token', 'security_token', 'aws_security_token', 'access_token']
-  profile:
-    description:
-      - A named AWS profile to use for authentication.
-      - See the AWS documentation for more information about named profiles
-        U(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
-      - The C(AWS_PROFILE) environment variable may also be used.
-      - The I(profile) option is mutually exclusive with the I(aws_access_key),
-        I(aws_secret_key) and I(security_token) options.
-    type: str
-    aliases: ['aws_profile']
 
-  endpoint_url:
-    description:
-      - URL to connect to instead of the default AWS endpoints.  While this
-        can be used to connection to other AWS-compatible services the
-        amazon.aws and community.aws collections are only tested against
-        AWS.
-      - The  C(AWS_URL) or C(EC2_URL) environment variables may also be used,
-        in decreasing order of preference.
-      - The I(ec2_url) and I(s3_url) aliases have been deprecated and will be
-        removed in a release after 2024-12-01.
-      - Support for the C(EC2_URL) environment variable has been deprecated and
-        will be removed in a release after 2024-12-01.
-    type: str
-    aliases: ['ec2_url', 'aws_endpoint_url', 's3_url' ]
-  aws_ca_bundle:
-    description:
-      - The location of a CA Bundle to use when validating SSL certificates.
-      - The C(AWS_CA_BUNDLE) environment variable may also be used.
-    type: path
-  validate_certs:
-    description:
-      - When set to C(false), SSL certificates will not be validated for
-        communication with the AWS APIs.
-      - Setting I(validate_certs=false) is strongly discouraged, as an
-        alternative, consider setting I(aws_ca_bundle) instead.
-    type: bool
-    default: true
-  aws_config:
-    description:
-      - A dictionary to modify the botocore configuration.
-      - Parameters can be found in the AWS documentation
-        U(https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore.config.Config).
-    type: dict
-  debug_botocore_endpoint_logs:
-    description:
-      - Use a C(botocore.endpoint) logger to parse the unique (rather than total)
-        C("resource:action") API calls made during a task, outputing the set to
-        the resource_actions key in the task results. Use the
-        C(aws_resource_action) callback to output to total list made during
-        a playbook.
-      - The C(ANSIBLE_DEBUG_BOTOCORE_LOGS) environment variable may also be used.
-    type: bool
-    default: false
-notes:
-  - B(Caution:) For modules, environment variables and configuration files are
-    read from the Ansible 'host' context and not the 'controller' context.
-    As such, files may need to be explicitly copied to the 'host'.  For lookup
-    and connection plugins, environment variables and configuration files are
-    read from the Ansible 'controller' context and not the 'host' context.
-  - The AWS SDK (boto3) that Ansible uses may also read defaults for credentials
-    and other settings, such as the region, from its configuration files in the
-    Ansible 'host' context (typically C(~/.aws/credentials)).
-    See U(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
-    for more information.
-'''
+class ModuleDocFragment(object):
+    def __init__(self):
+        self.DOCUMENTATION = CommonFragment.MODULES

plugins/doc_fragments/aws_credentials.py

@@ -6,11 +6,16 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
+#
+# The amazon.aws.aws_credentials docs fragment has been deprecated,
+# please migrate to amazon.aws.common.plugins.
+#
+
 
 class ModuleDocFragment(object):
 
     # Plugin options for AWS credentials
-    DOCUMENTATION = r'''
+    DOCUMENTATION = r"""
 options:
   aws_profile:
     description: The AWS profile
@@ -25,6 +30,11 @@ class ModuleDocFragment(object):
     aliases: [ aws_access_key_id ]
     env:
       - name: EC2_ACCESS_KEY
+        deprecated:
+          removed_at_date: '2024-12-01'
+          collection_name: amazon.aws
+          why: 'EC2 in the name implied it was limited to EC2 resources.  However, it is used for all connections.'
+          alternatives: AWS_ACCESS_KEY_ID
       - name: AWS_ACCESS_KEY
       - name: AWS_ACCESS_KEY_ID
   aws_secret_key:
@@ -33,13 +43,28 @@ class ModuleDocFragment(object):
     aliases: [ aws_secret_access_key ]
     env:
       - name: EC2_SECRET_KEY
+        deprecated:
+          removed_at_date: '2024-12-01'
+          collection_name: amazon.aws
+          why: 'EC2 in the name implied it was limited to EC2 resources.  However, it is used for all connections.'
+          alternatives: AWS_SECRET_ACCESS_KEY
       - name: AWS_SECRET_KEY
       - name: AWS_SECRET_ACCESS_KEY
   aws_security_token:
     description: The AWS security token if using temporary access and secret keys.
     type: str
     env:
       - name: EC2_SECURITY_TOKEN
+        deprecated:
+          removed_at_date: '2024-12-01'
+          collection_name: amazon.aws
+          why: 'EC2 in the name implied it was limited to EC2 resources.  However, it is used for all connections.'
+          alternatives: AWS_SESSION_TOKEN
       - name: AWS_SESSION_TOKEN
       - name: AWS_SECURITY_TOKEN
-'''
+        deprecated:
+          removed_at_date: '2024-12-01'
+          collection_name: amazon.aws
+          why: 'AWS_SECURITY_TOKEN was used for compatability with the original boto SDK, support for which has been dropped'
+          alternatives: AWS_SESSION_TOKEN
+"""

plugins/doc_fragments/aws_region.py

@@ -1,21 +1,21 @@
 # -*- coding: utf-8 -*-
 
-# Copyright: (c) 2017,  Ansible Project
+# (c) 2022 Red Hat Inc.
+#
+# This file is part of Ansible
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
+from .region import ModuleDocFragment as RegionFragment
+
+#
+# The amazon.aws.aws_region docs fragment has been deprecated,
+# please migrate to amazon.aws.region.plugins.
+#
 
-class ModuleDocFragment(object):
 
-    # Plugin option for AWS region
-    DOCUMENTATION = r'''
-options:
-  region:
-    description: The region for which to create the connection.
-    type: str
-    env:
-      - name: EC2_REGION
-      - name: AWS_REGION
-'''
+class ModuleDocFragment(object):
+    def __init__(self):
+        self.DOCUMENTATION = RegionFragment.PLUGINS

plugins/doc_fragments/boto3.py

@@ -1,19 +1,28 @@
 # -*- coding: utf-8 -*-
 
-# Copyright: (c) 2022,  Ansible Project
+# Copyright: (c) 2022, Ansible Project
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
+from ansible_collections.amazon.aws.plugins.module_utils import botocore as botocore_utils
+
 
 class ModuleDocFragment(object):
 
-    # Minimum requirements for the collection
-    DOCUMENTATION = r'''
-options: {}
+    # Modules and Plugins can (currently) use the same fragment
+    def __init__(self):
+
+        # Minimum requirements for the collection
+        requirements = f"""
+options: {{}}
 requirements:
   - python >= 3.6
-  - boto3 >= 1.18.0
-  - botocore >= 1.21.0
-'''
+  - boto3 >= {botocore_utils.MINIMUM_BOTO3_VERSION}
+  - botocore >= {botocore_utils.MINIMUM_BOTOCORE_VERSION}
+"""
+
+        self.DOCUMENTATION = requirements
+        self.MODULES = requirements
+        self.PLUGINS = requirements