Do not lazy-load pyspark.ml #2184
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eduardo Apolinario <[email protected]>
eapolinario
requested review from
wild-endeavor,
kumare3,
pingsutw and
cosmicBboy
as code owners
dosubot
bot
added
the
size:XS
wild-endeavor
previously approved these changes
Feb 10, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2184 +/- ##
==========================================
+ Coverage 85.67% 85.69% +0.02%
==========================================
Files 314 314
Lines 24005 23958 -47
Branches 3627 3622 -5
==========================================
- Hits 20566 20532 -34
+ Misses 2801 2787 -14
- Partials 638 639 +1 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Eduardo Apolinario <[email protected]>
wild-endeavor
approved these changes
Feb 12, 2024
fiedlerNr9
pushed a commit
that referenced
this pull request
Jul 25, 2024
* Do not lazy-load pyspark.ml Signed-off-by: Eduardo Apolinario <[email protected]> * Lint Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jan Fiedler <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracking issue
flyteorg/flyte#4877
Why are the changes needed?
Spark tasks are currently broken on version 1.10.3 and 1.10.2 of the flytekit spark plugin.
What changes were proposed in this pull request?
Do not lazy-load the
pyspark.mlmodule.How was this patch tested?
Built an image that used that and tested on a real Flyte cluster using a slightly modified version of https://github.com/flyteorg/flytesnacks/blob/master/examples/k8s_spark_plugin/k8s_spark_plugin/pyspark_pi.py.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link