refactor(ingest): move Airflow into datahub_provider module

[hsheth2]

Airflow users expect stuff to be in datahub_provider.{hooks,operators,lineage}.
Maintains backwards compatibility by re-exporting from the datahub.integrations.airflow files.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[sunkickr]

How do you feel about renaming DataHubAirflowLineageBackend to DataHubLineageBackend this would fall more inline with how other airflow modules are named

[hsheth2]

@sunkickr makes sense - just made the change

[sunkickr]

Just like my comment on tests, example airflow dags can also be included in the datahub_provider like in the sample_repo here: https://github.com/astronomer/airflow-provider-sample/tree/main/sample_provider/example_dags. This way all airflow stuff can live in the provider and the registry will be able to build a page for each example dag in the example_dags directory like here: https://registry.astronomer-stage.io/providers/google/example-dags/example-automl-nl-text-classification

[sunkickr]

You may want to store tests for operators, hooks, and the lineagebackend in datahub_provider like done in the sample repo: https://github.com/astronomer/airflow-provider-sample/tree/main/tests. This way modules from datahub_provider won't need to be imported into datahub.

[hsheth2]

Not sure I'm following the suggestion - the tests in both the sample repo and in this one are stored separate from the main source directories.

The datahub imports of datahub_provider are primarily for backwards compatibility, so I'm pretty ok with leaving those in the code.

[sunkickr]

That makes sense was just suggesting because module tests are usually packaged with modules.

[hsheth2]

@sunkickr - I'd like to keep the examples/airflow directory outside of src, since it will be better for discoverability and everything under the src directory gets bundled and packaged with the pip package, which I don't want for the examples.

I have gone through and updated all the imports to use datahub_provider.

[sunkickr]

@hsheth2 - Maybe we could just duplicate the example dags repo into the provider package so they are easily discoverable in both the datahub repo and the registry. Example dags located in the datahub_provider/example_dags/ directory will be searchable in the registry. Also we are planning lots of features for example dags so provider packages that have them will get more exposure by having more searchable/interactive content. Right now if a user is looking at a datahub module in the registry, example dags that use that module will appear with it if they are included in the package.

[sunkickr]

Everything looks good to me structurally

[shirshanka]

LGTM!