Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cookie cutter initial template #738

Merged
merged 23 commits into from
Nov 9, 2021
Merged

Cookie cutter initial template #738

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
fd5dea3
Add cookiecutter to requirements and pyflyte init
eapolinario Oct 28, 2021
79da424
Invoke cookiecutter with the overridden configuration.
eapolinario Oct 28, 2021
38b0d88
Comment the use of directory in the call to cookiecutter.
eapolinario Oct 28, 2021
688f214
Use the original repo and do not clobber existing files+directories
eapolinario Nov 4, 2021
d2130cc
Add example sub-command
eapolinario Nov 4, 2021
79e9c1d
make requirements
eapolinario Nov 4, 2021
ab4c987
Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…
eapolinario Nov 5, 2021
94f22cf
Fix test_dataclass_transformer test
eapolinario Nov 5, 2021
b441cf6
Constrain mashmallow-jsonschema temporarily
eapolinario Nov 5, 2021
e1a6a4d
Add --template flag and project-name argument to command
eapolinario Nov 5, 2021
e5ad645
Revert changes to dataclass transformer tests
eapolinario Nov 5, 2021
bf7f22d
Constrain marshmallow-jsonschema in dev-requirements
eapolinario Nov 5, 2021
25832e2
Revert "Constrain marshmallow-jsonschema in dev-requirements"
eapolinario Nov 5, 2021
25aa23c
Fix typo in Makefile
eapolinario Nov 5, 2021
f547ed9
Fix regeneration of spark2 requirements file
eapolinario Nov 5, 2021
3f91dca
Use click for prompts
eapolinario Nov 5, 2021
e8ea83a
Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…
eapolinario Nov 5, 2021
b725095
Changes default workflow name to `my_wf`
eapolinario Nov 5, 2021
f492bb5
Remove the two inputs from simple-example
eapolinario Nov 5, 2021
b0fd8b4
Add more descriptive help for the template flag
eapolinario Nov 5, 2021
0420410
Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…
eapolinario Nov 8, 2021
3efecec
Set cookiecutter in setup.py
eapolinario Nov 8, 2021
bf60412
Default top level directory to flyte (as opposed to myapp)
eapolinario Nov 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 19 additions & 15 deletions dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,13 @@ click==7.1.2
# via
# -c requirements.txt
# flytekit
cloudpickle==2.0.0
# via
# -c requirements.txt
# flytekit
codespell==2.1.0
# via -r dev-requirements.in
coverage[toml]==6.0.2
coverage[toml]==6.1.1
# via -r dev-requirements.in
croniter==1.0.15
# via
Expand Down Expand Up @@ -95,17 +99,17 @@ docstring-parser==0.12
# via
# -c requirements.txt
# flytekit
filelock==3.3.1
filelock==3.3.2
# via virtualenv
flyteidl==0.21.6
flyteidl==0.21.8
# via
# -c requirements.txt
# flytekit
grpcio==1.41.0
grpcio==1.41.1
# via
# -c requirements.txt
# flytekit
identify==2.3.0
identify==2.3.3
# via pre-commit
idna==3.3
# via
Expand Down Expand Up @@ -150,7 +154,7 @@ marshmallow-enum==1.5.1
# via
# -c requirements.txt
# dataclasses-json
marshmallow-jsonschema==0.12.0
marshmallow-jsonschema==0.13.0
# via
# -c requirements.txt
# flytekit
Expand All @@ -163,7 +167,7 @@ mypy-extensions==0.4.3
# -c requirements.txt
# mypy
# typing-inspect
natsort==7.1.1
natsort==8.0.0
# via
# -c requirements.txt
# flytekit
Expand All @@ -174,7 +178,7 @@ numpy==1.21.3
# -c requirements.txt
# pandas
# pyarrow
packaging==21.0
packaging==21.2
# via
# -c requirements.txt
# pytest
Expand All @@ -194,12 +198,12 @@ pluggy==1.0.0
# via pytest
pre-commit==2.15.0
# via -r dev-requirements.in
protobuf==3.19.0
protobuf==3.19.1
# via
# -c requirements.txt
# flyteidl
# flytekit
py==1.10.0
py==1.11.0
# via
# -c requirements.txt
# pytest
Expand Down Expand Up @@ -259,7 +263,7 @@ pyyaml==5.4.1
# -c requirements.txt
# docker-compose
# pre-commit
regex==2021.10.21
regex==2021.11.2
# via
# -c requirements.txt
# docker-image-py
Expand All @@ -270,7 +274,7 @@ requests==2.26.0
# docker-compose
# flytekit
# responses
responses==0.14.0
responses==0.15.0
# via
# -c requirements.txt
# flytekit
Expand Down Expand Up @@ -310,7 +314,7 @@ toml==0.10.2
# mypy
# pre-commit
# pytest
tomli==1.2.1
tomli==1.2.2
# via
# -c requirements.txt
# coverage
Expand All @@ -329,7 +333,7 @@ urllib3==1.26.7
# flytekit
# requests
# responses
virtualenv==20.8.1
virtualenv==20.10.0
# via pre-commit
websocket-client==0.59.0
# via
Expand All @@ -339,7 +343,7 @@ wheel==0.37.0
# via
# -c requirements.txt
# flytekit
wrapt==1.13.2
wrapt==1.13.3
# via
# -c requirements.txt
# deprecated
Expand Down
52 changes: 29 additions & 23 deletions doc-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ alabaster==0.7.12
# via sphinx
ansiwrap==0.8.4
# via papermill
astroid==2.8.3
astroid==2.8.4
# via sphinx-autoapi
attrs==21.2.0
# via jsonschema
Expand All @@ -25,13 +25,13 @@ beautifulsoup4==4.10.0
# furo
# sphinx-code-include
# sphinx-material
black==21.9b0
black==21.10b0
# via papermill
bleach==4.1.0
# via nbconvert
boto3==1.19.0
boto3==1.19.10
# via sagemaker-training
botocore==1.22.0
botocore==1.22.11
# via
# boto3
# s3transfer
Expand All @@ -52,6 +52,8 @@ click==7.1.2
# flytekit
# hmsclient
# papermill
cloudpickle==2.0.0
# via flytekit
croniter==1.0.15
# via flytekit
cryptography==35.0.0
Expand Down Expand Up @@ -84,15 +86,15 @@ entrypoints==0.3
# jupyter-client
# nbconvert
# papermill
flyteidl==0.21.6
flyteidl==0.21.8
# via flytekit
furo @ git+git://github.com/flyteorg/furo@main
# via -r doc-requirements.in
gevent==21.8.0
# via sagemaker-training
greenlet==1.1.2
# via gevent
grpcio==1.41.0
grpcio==1.41.1
# via
# -r doc-requirements.in
# flytekit
Expand All @@ -104,11 +106,13 @@ imagesize==1.2.0
# via sphinx
importlib-metadata==4.8.1
# via keyring
importlib-resources==5.4.0
# via jsonschema
inotify_simple==1.2.1
# via sagemaker-training
ipykernel==5.5.6
# via flytekit
ipython==7.28.0
ipython==7.29.0
# via ipykernel
ipython-genutils==0.2.0
# via
Expand All @@ -129,13 +133,13 @@ jmespath==0.10.0
# via
# boto3
# botocore
jsonschema==4.1.2
jsonschema==4.2.0
# via nbformat
jupyter-client==7.0.6
# via
# ipykernel
# nbclient
jupyter-core==4.8.1
jupyter-core==4.9.1
# via
# jupyter-client
# nbconvert
Expand All @@ -148,7 +152,7 @@ keyring==23.2.1
# via flytekit
lazy-object-proxy==1.6.0
# via astroid
lxml==4.6.3
lxml==4.6.4
# via sphinx-material
markupsafe==2.0.1
# via jinja2
Expand All @@ -159,7 +163,7 @@ marshmallow==3.14.0
# marshmallow-jsonschema
marshmallow-enum==1.5.1
# via dataclasses-json
marshmallow-jsonschema==0.12.0
marshmallow-jsonschema==0.13.0
# via flytekit
matplotlib-inline==0.1.3
# via ipython
Expand All @@ -169,7 +173,7 @@ mypy-extensions==0.4.3
# via
# black
# typing-inspect
natsort==7.1.1
natsort==8.0.0
# via flytekit
nbclient==0.5.4
# via
Expand All @@ -193,7 +197,7 @@ numpy==1.21.3
# pyarrow
# sagemaker-training
# scipy
packaging==21.0
packaging==21.2
# via
# bleach
# sphinx
Expand All @@ -215,9 +219,9 @@ pickleshare==0.7.5
# via ipython
platformdirs==2.4.0
# via black
prompt-toolkit==3.0.20
prompt-toolkit==3.0.22
# via ipython
protobuf==3.19.0
protobuf==3.19.1
# via
# flyteidl
# flytekit
Expand All @@ -227,7 +231,7 @@ psutil==5.8.0
# via sagemaker-training
ptyprocess==0.7.0
# via pexpect
py==1.10.0
py==1.11.0
# via retry
py4j==0.10.9.2
# via pyspark
Expand Down Expand Up @@ -274,7 +278,7 @@ pyyaml==6.0
# sphinx-autoapi
pyzmq==22.3.0
# via jupyter-client
regex==2021.10.21
regex==2021.11.2
# via
# black
# docker-image-py
Expand All @@ -284,7 +288,7 @@ requests==2.26.0
# papermill
# responses
# sphinx
responses==0.14.0
responses==0.15.0
# via flytekit
retry==0.9.2
# via flytekit
Expand Down Expand Up @@ -315,7 +319,7 @@ snowballstemmer==2.1.0
# via sphinx
sortedcontainers==2.4.0
# via flytekit
soupsieve==2.2.1
soupsieve==2.3
# via beautifulsoup4
sphinx==4.2.0
# via
Expand Down Expand Up @@ -369,15 +373,15 @@ textwrap3==0.9.2
# via ansiwrap
thrift==0.15.0
# via hmsclient
tomli==1.2.1
tomli==1.2.2
# via black
tornado==6.1
# via
# ipykernel
# jupyter-client
tqdm==4.62.3
# via papermill
traitlets==5.1.0
traitlets==5.1.1
# via
# ipykernel
# ipython
Expand Down Expand Up @@ -412,13 +416,15 @@ werkzeug==2.0.2
# via sagemaker-training
wheel==0.37.0
# via flytekit
wrapt==1.13.2
wrapt==1.13.3
# via
# astroid
# deprecated
# flytekit
zipp==3.6.0
# via importlib-metadata
# via
# importlib-metadata
# importlib-resources
zope.event==4.5.0
# via gevent
zope.interface==5.4.0
Expand Down
60 changes: 60 additions & 0 deletions flytekit/clis/sdk_in_container/init.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import click
from cookiecutter.main import cookiecutter
from cookiecutter.prompt import read_user_variable


@click.group("init")
def init():
"""
Create flyte-ready projects.
"""
# TODO: Add `interactive` boolean flag to allow for programmatic access to this subcommand.
pass


@click.command("example")
def generate_simple_example():
eapolinario marked this conversation as resolved.
Show resolved Hide resolved
"""
This command creates a directory containing the minimal code necessary to start a Flyte-ready project.
The generated directory structure for the default values resemble:
flyte_example
├── Dockerfile
├── flyte.config
├── LICENSE
├── myapp
│   ├── __init__.py
│   └── workflows
│   ├── example.py
│   └── __init__.py
├── README.md
└── requirements.txt
"""
print("What is the name of your project? This is the name of the directory that will be created.")
project_name = read_user_variable("project_name", "flyte_example")
print("What should we call your application? This serves as the top level package where your workflows will live.")
app = read_user_variable("app", "myapp")
print("What should be the name of your example workflow?")
workflow_name = read_user_variable("workflow", "workflow_example")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just use click for consistency http://pallets-click.readthedocs.io/en/stable/prompts.html

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using cookiecutter just for the templating feels less cluttered. Thanks for the suggestion!


config = {
"project_name": project_name,
"app": app,
"workflow": workflow_name,
}
cookiecutter(
"https://github.com/flyteorg/flytekit-python-template.git",
# TODO: remove this once we make the transition to cookie-cutter official.
checkout="cookie-cutter",
no_input=True,
# We do not want to clobber existing files/directories.
overwrite_if_exists=False,
extra_context=config,
# By specifying directory we can have multiple templates in the same repository,
# as described in https://cookiecutter.readthedocs.io/en/1.7.2/advanced/directories.html.
# The idea is to extend the number of templates, each in their own subdirectory, for example
# a tensorflow-based example.
directory="simple-example",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I mean. This is hard coded. Or are you thinking of parameterizing this? I was suggesting that you could create
pyflyte init tensorflow pyflyte init xyz
Or otherwise we should call it
pyflyte init <project-name> --template=tensorflow
and then --template can be defaulted to simple-example?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of parametrizing this going forward, but I really liked the suggestion of making project-name part of the invocation. I'll go ahead and implement both suggestions.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also updated the PR description.

)


init.add_command(generate_simple_example)
Loading