Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Python Package Installation Process on M1 Macs #740

Conversation

TeoZosa
Copy link
Contributor

@TeoZosa TeoZosa commented Nov 5, 2021

TL;DR

Updates the pyarrow dependency to 6.0.0 for easier flytekit installation on M1 macs.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

Previously, installation on M1 macs required pyarrow to be built from source since no prebuilt arm64 wheels were available for the 3.0.0 version. This build, in turn, depended on the 3.0.0 version of the Apache Arrow library (pyarrow's build system had breaking changes between 3.0.0 and 4.0.0). Apache Arrow can be installed via brew, but only the 6.0.0 version is currently available. This meant that users had to also build the Apache Arrow library from source (and remember to set the correct LDFLAGS and CFLAGS, etc.) prior to building pyarrow.

Since pyarrow version 6.0.0 includes prebuilt wheels for the arm64 architecture, simply upgrading the pyarrow dependency version to 6.0.0 elides the above steps, yielding a much more pleasant flytekit installation experience.

Tracking Issue

NA

Follow-up issue

NA

@welcome
Copy link

welcome bot commented Nov 5, 2021

Thank you for opening this pull request! 🙌
These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge. - Sign off your commits (Reference: DCO Guide).

@TeoZosa TeoZosa force-pushed the feat/bump-pyarrow-to-6.0.0-for-m1-mac-compatibility branch from c2ae8a9 to 797fcce Compare November 5, 2021 01:26
@kumare3
Copy link
Contributor

kumare3 commented Nov 5, 2021

Interesting this seems like an infra issue? cc @evalsocket / @samhita-alla / @wild-endeavor does anyone of you know

@TeoZosa TeoZosa force-pushed the feat/bump-pyarrow-to-6.0.0-for-m1-mac-compatibility branch from 797fcce to 58cbaa7 Compare November 5, 2021 17:18
@TeoZosa TeoZosa force-pushed the feat/bump-pyarrow-to-6.0.0-for-m1-mac-compatibility branch from 58cbaa7 to c05b03c Compare November 5, 2021 17:33
@eapolinario
Copy link
Collaborator

The failures have to do with conflicting requirements around pyarrow. From the CI logs:

2021-11-05T18:49:10.7895944Z [1m[31mE           #11 19.48 The conflict is caused by:
2021-11-05T18:49:10.7896480Z [0m
2021-11-05T18:49:10.7897116Z [1m[31mE           #11 19.48     The user requested pyarrow==6.0.0
2021-11-05T18:49:10.7897697Z [0m
2021-11-05T18:49:10.7898370Z [1m[31mE           #11 19.48     flytekit 0.23.0 depends on pyarrow<4.0.0 and >=2.0.0

Once #741 is merged we should merge master and retry.

@wild-endeavor
Copy link
Contributor

#741 has been merged btw. let's close this PR...

@TeoZosa
Copy link
Contributor Author

TeoZosa commented Nov 8, 2021

#741 has been merged btw. let's close this PR...

Ah! Did we not want to rerun this PR to bump pyarrow from 3.0.0 to 6.0.0? #741 only removed the max version threshold, but pyarrow is still pinned at 3.0.0.

@TeoZosa
Copy link
Contributor Author

TeoZosa commented Nov 8, 2021

*Also FYI, just rebased onto the latest commit from master so this branch is in sync with the latest changes.

@TeoZosa
Copy link
Contributor Author

TeoZosa commented Nov 8, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants