Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flytekitplugin pandera update: use entrypoint and structured dataset #2821

Merged
merged 11 commits into from
Oct 17, 2024

Conversation

cosmicBboy
Copy link
Contributor

Tracking issue

flyteorg/flyte#5851

Why are the changes needed?

The pandera flytekit plugin is out-dated and needs to be maintained.

What changes were proposed in this pull request?

  • Adds an entrypoint config in setup.py for auto-registration of the type transformer
  • Rename schema.py -> pandas_transformer.py
  • Create a basics deck renderer PandasReportRenderer
  • Add configuration ValidationConfig with on_error option. This is so that on_error="warn" can create a Flyte deck that's actually rendered in the UI. Ideally on_error="raise" will also render a deck, but right now there's a limitation where decks aren't rendered with failed tasks.

How was this patch tested?

Update the unit tests, run with pytest test in the plugin directory.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
@cosmicBboy cosmicBboy changed the title Flytekitplugin pandera fix Flytekitplugin pandera update: use entrypoint and structured dataset Oct 16, 2024
Copy link

codecov bot commented Oct 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.22%. Comparing base (cd8216a) to head (5ae037a).
Report is 23 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2821      +/-   ##
==========================================
+ Coverage   76.82%   77.22%   +0.40%     
==========================================
  Files         196      246      +50     
  Lines       20301    22269    +1968     
  Branches     2610     2609       -1     
==========================================
+ Hits        15596    17198    +1602     
- Misses       4004     4317     +313     
- Partials      701      754      +53     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
@cosmicBboy cosmicBboy merged commit 8da1329 into master Oct 17, 2024
106 checks passed
@cosmicBboy cosmicBboy deleted the flytekitplugin-pandera-fix branch October 17, 2024 15:29
kumare3 pushed a commit that referenced this pull request Nov 8, 2024
…2821)

* fix pandera plugin error

Signed-off-by: Niels Bantilan <[email protected]>

* add validation config

Signed-off-by: Niels Bantilan <[email protected]>

* use structured dataset transformer

Signed-off-by: Niels Bantilan <[email protected]>

* refactor get_literal_type

Signed-off-by: Niels Bantilan <[email protected]>

* cleanup

Signed-off-by: Niels Bantilan <[email protected]>

* add unit tests

Signed-off-by: Niels Bantilan <[email protected]>

* remove unused import

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

* use logger, remove unneeded typing for <py38

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants