Skip to content

Commit

Permalink
Update plugins requirements files versions (#1365)
Browse files Browse the repository at this point in the history 
* Update plugins requirements files versions

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix deck-standard test

Signed-off-by: Eduardo Apolinario <[email protected]>

* Install all flytekit

Signed-off-by: Eduardo Apolinario <[email protected]>

* Regenerate papermill dev-requirements.txt

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove dev-requirements from papermill plugin

Signed-off-by: Eduardo Apolinario <[email protected]>

* Install flytekit from the repo.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION env var in the case of onnx plugins tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove dev-requirements from fsspec plugin

Signed-off-by: Eduardo Apolinario <[email protected]>

* Special-case whylogs plugin, remove dev-requirements, and install flytekit from repo in all plugins

Signed-off-by: Eduardo Apolinario <[email protected]>

* Regenerate plugins requirements

Signed-off-by: Eduardo Apolinario <[email protected]>

* Skip onnx-tensorflow tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Revert "Regenerate plugins requirements"

This reverts commit 39ab92b.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Install flytekit via the old means in plugins tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Do not skip onnx tests.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove local flytekit install from papermill requirements

Signed-off-by: Eduardo Apolinario <[email protected]>

* Regenerate whylogs requirements

Signed-off-by: Eduardo Apolinario <[email protected]>

* Comment out onnx-tensorflow from pythonbuild gh action and rename setup to furloughed_setup.py

Signed-off-by: Eduardo Apolinario <[email protected]>

* pyspark==3.3.0 in papermill requirements

Signed-off-by: Eduardo Apolinario <[email protected]>

* Restore dev-requirements files in papermill plugin

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
  • Loading branch information
@eapolinario
eapolinario and eapolinario authored Dec 13, 2022
1 parent c3d54b1 commit 8181742
Show file tree
Hide file tree
Showing 63 changed files with 2,271 additions and 1,811 deletions.
14 changes: 11 additions & 3 deletions .github/workflows/pythonbuild.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,9 @@ jobs:
- flytekit-modin
- flytekit-onnx-pytorch
- flytekit-onnx-scikitlearn
- flytekit-onnx-tensorflow
# onnxx-tensorflow needs a version of tensorflow that does not work with protobuf>4.
# The issue is being tracked on the tensorflow side in https://github.com/tensorflow/tensorflow/issues/53234#issuecomment-1330111693
# flytekit-onnx-tensorflow
- flytekit-pandera
- flytekit-papermill
- flytekit-polars
Expand Down Expand Up @@ -130,12 +132,18 @@ jobs:
cd plugins/${{ matrix.plugin-names }}
pip install -r requirements.txt
if [ -f dev-requirements.txt ]; then pip install -r dev-requirements.txt; fi
pip install --no-deps -U https://github.com/flyteorg/flytekit/archive/${{ github.sha }}.zip#egg=flytekit
pip install -U https://github.com/flyteorg/flytekit/archive/${{ github.sha }}.zip#egg=flytekit
pip freeze
- name: Test with coverage
run: |
cd plugins/${{ matrix.plugin-names }}
coverage run -m pytest tests
# onnx plugins does not support protobuf>4 yet (in fact it is tensorflow that
# does not support that yet). More details in https://github.com/onnx/onnx/issues/4239.
if [[ ${{ matrix.plugin-names }} == *"onnx"* || ${{ matrix.plugin-names }} == "flytekit-whylogs" ]]; then
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python coverage run -m pytest tests
else
coverage run -m pytest tests
fi
lint:
runs-on: ubuntu-latest
steps:
Expand Down
107 changes: 54 additions & 53 deletions plugins/flytekit-aws-athena/requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,34 +1,34 @@
#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile requirements.in
#
-e file:.#egg=flytekitplugins-athena
# via -r requirements.in
arrow==1.2.2
arrow==1.2.3
# via jinja2-time
binaryornot==0.4.4
# via cookiecutter
certifi==2022.6.15
certifi==2022.12.7
# via requests
cffi==1.15.1
# via cryptography
chardet==5.0.0
chardet==5.1.0
# via binaryornot
charset-normalizer==2.1.0
charset-normalizer==2.1.1
# via requests
click==8.1.3
# via
# cookiecutter
# flytekit
cloudpickle==2.1.0
cloudpickle==2.2.0
# via flytekit
cookiecutter==2.1.1
# via flytekit
croniter==1.3.5
croniter==1.3.8
# via flytekit
cryptography==37.0.4
cryptography==38.0.4
# via
# pyopenssl
# secretstorage
Expand All @@ -40,33 +40,34 @@ deprecated==1.2.13
# via flytekit
diskcache==5.4.0
# via flytekit
docker==5.0.3
docker==6.0.1
# via flytekit
docker-image-py==0.1.12
# via flytekit
docstring-parser==0.14.1
docstring-parser==0.15
# via flytekit
flyteidl==1.1.8
flyteidl==1.3.0
# via flytekit
flytekit==1.1.0
flytekit==1.3.0b2
# via flytekitplugins-athena
googleapis-common-protos==1.56.3
googleapis-common-protos==1.57.0
# via
# flyteidl
# grpcio-status
grpcio==1.47.0
grpcio==1.51.1
# via
# flytekit
# grpcio-status
grpcio-status==1.47.0
grpcio-status==1.51.1
# via flytekit
idna==3.3
idna==3.4
# via requests
importlib-metadata==4.12.0
importlib-metadata==5.1.0
# via
# click
# flytekit
# keyring
jaraco-classes==3.2.3
# via keyring
jeepney==0.8.0
# via
# keyring
Expand All @@ -77,11 +78,13 @@ jinja2==3.1.2
# jinja2-time
jinja2-time==0.2.0
# via cookiecutter
keyring==23.6.0
joblib==1.2.0
# via flytekit
keyring==23.11.0
# via flytekit
markupsafe==2.1.1
# via jinja2
marshmallow==3.17.0
marshmallow==3.19.0
# via
# dataclasses-json
# marshmallow-enum
Expand All @@ -90,105 +93,103 @@ marshmallow-enum==1.5.1
# via dataclasses-json
marshmallow-jsonschema==0.13.0
# via flytekit
more-itertools==9.0.0
# via jaraco-classes
mypy-extensions==0.4.3
# via typing-inspect
natsort==8.1.0
natsort==8.2.0
# via flytekit
numpy==1.21.6
numpy==1.23.5
# via
# flytekit
# pandas
# pyarrow
packaging==21.3
# via marshmallow
pandas==1.3.5
packaging==22.0
# via
# docker
# marshmallow
pandas==1.5.2
# via flytekit
protobuf==3.20.2
protobuf==4.21.11
# via
# flyteidl
# flytekit
# googleapis-common-protos
# grpcio-status
# protoc-gen-swagger
protoc-gen-swagger==0.1.0
# via flyteidl
py==1.11.0
# via retry
pyarrow==6.0.1
pyarrow==10.0.1
# via flytekit
pycparser==2.21
# via cffi
pyopenssl==22.0.0
pyopenssl==22.1.0
# via flytekit
pyparsing==3.0.9
# via packaging
python-dateutil==2.8.2
# via
# arrow
# croniter
# flytekit
# pandas
python-json-logger==2.0.2
python-json-logger==2.0.4
# via flytekit
python-slugify==6.1.2
python-slugify==7.0.0
# via cookiecutter
pytimeparse==1.1.8
# via flytekit
pytz==2022.1
pytz==2022.6
# via
# flytekit
# pandas
pyyaml==6.0
# via
# cookiecutter
# flytekit
regex==2022.6.2
regex==2022.10.31
# via docker-image-py
requests==2.28.1
# via
# cookiecutter
# docker
# flytekit
# responses
responses==0.21.0
responses==0.22.0
# via flytekit
retry==0.9.2
# via flytekit
secretstorage==3.3.2
secretstorage==3.3.3
# via keyring
singledispatchmethod==1.0
# via flytekit
six==1.16.0
# via
# grpcio
# python-dateutil
# via python-dateutil
sortedcontainers==2.4.0
# via flytekit
statsd==3.3.0
# via flytekit
text-unidecode==1.3
# via python-slugify
typing-extensions==4.3.0
toml==0.10.2
# via responses
types-toml==0.10.8.1
# via responses
typing-extensions==4.4.0
# via
# arrow
# flytekit
# importlib-metadata
# responses
# typing-inspect
typing-inspect==0.7.1
typing-inspect==0.8.0
# via dataclasses-json
urllib3==1.26.9
urllib3==1.26.13
# via
# docker
# flytekit
# requests
# responses
websocket-client==1.3.3
websocket-client==1.4.2
# via docker
wheel==0.38.0
wheel==0.38.4
# via flytekit
wrapt==1.14.1
# via
# deprecated
# flytekit
zipp==3.8.0
zipp==3.11.0
# via importlib-metadata
2 changes: 1 addition & 1 deletion plugins/flytekit-aws-athena/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

microlib_name = f"flytekitplugins-{PLUGIN_NAME}"

plugin_requires = ["flytekit>=1.1.0b0,<2.0.0"]
plugin_requires = ["flytekit>=1.3.0b2,<2.0.0"]

__version__ = "0.0.0+develop"

Expand Down
Loading

0 comments on commit 8181742

Please sign in to comment.