Cookie cutter initial template #738

eapolinario · 2021-11-04T23:46:35Z

TL;DR

Add a command to generate the boilerplate for flyte-ready repos.

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

This PR defines a new pyflyte command called init that takes a template and a project name as arguments. These are used to call the template engine provided by cookiecutter.

In terms of where templates are stored, we defined a cookiecutter template in flyteorg/flytekit-python-template#16. Note how we have the ability to keep multiple templates in the same repo, the idea being that we're going to expose them using different templates under the pyflyte init command.

The pyflyte init command defaults to "simple-example" cookiecutter template and this template prompts the user for 2 additional inputs:

app: which is the top-level directory under project-name that will store the actual workflow
workflow: the name of the example workflow.

All inputs have default values.

Also worth noting that the command does not clobber existing user code, i.e. the command fails if the user attempts to overwrite a directory by re-running the command.

Here's how the help provided by click looks like:

❯ pyflyte init --help
...
Usage: pyflyte init [OPTIONS] PROJECT_NAME

  Create flyte-ready projects.

Options:
  --template TEXT  cookiecutter template to be used
  --help           Show this message and exit.

Tracking Issue

flyteorg/flyte#1799

Follow-up issue

NA

Signed-off-by: Eduardo Apolinario <[email protected]>

…late

Signed-off-by: Eduardo Apolinario <[email protected]>

flytekit/clis/sdk_in_container/init.py

eapolinario · 2021-11-05T01:55:13Z

Unit test is failing. Fix in #739.

kumare3 · 2021-11-05T04:07:13Z

flytekit/clis/sdk_in_container/init.py

+        # as described in https://cookiecutter.readthedocs.io/en/1.7.2/advanced/directories.html.
+        # The idea is to extend the number of templates, each in their own subdirectory, for example
+        # a tensorflow-based example.
+        directory="simple-example",


This is what I mean. This is hard coded. Or are you thinking of parameterizing this? I was suggesting that you could create
pyflyte init tensorflow pyflyte init xyz
Or otherwise we should call it
pyflyte init <project-name> --template=tensorflow
and then --template can be defaulted to simple-example?

I was thinking of parametrizing this going forward, but I really liked the suggestion of making project-name part of the invocation. I'll go ahead and implement both suggestions.

I also updated the PR description.

Signed-off-by: Eduardo Apolinario <[email protected]>

This reverts commit bf7f22d.

Signed-off-by: Eduardo Apolinario <[email protected]>

codecov · 2021-11-05T07:19:16Z

Codecov Report

Merging #738 (bf60412) into master (c4f33c6) will decrease coverage by 0.00%.
The diff coverage is 72.72%.

@@            Coverage Diff             @@
##           master     #738      +/-   ##
==========================================
- Coverage   85.61%   85.60%   -0.01%     
==========================================
  Files         341      342       +1     
  Lines       28915    28926      +11     
  Branches     2381     2381              
==========================================
+ Hits        24755    24763       +8     
- Misses       3527     3530       +3     
  Partials      633      633

Impacted Files	Coverage Δ
flytekit/clis/sdk_in_container/init.py	`66.66% <66.66%> (ø)`
flytekit/clis/sdk_in_container/pyflyte.py	`83.05% <100.00%> (+0.59%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4f33c6...bf60412. Read the comment docs.

kumare3 · 2021-11-05T15:07:15Z

flytekit/clis/sdk_in_container/init.py

+    )
+    app = read_user_variable("app", "myapp")
+    click.echo("What should be the name of your example workflow?")
+    workflow_name = read_user_variable("workflow", "workflow_example")


Should we just use click for consistency http://pallets-click.readthedocs.io/en/stable/prompts.html

using cookiecutter just for the templating feels less cluttered. Thanks for the suggestion!

Signed-off-by: Eduardo Apolinario <[email protected]>

…late Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario · 2021-11-05T20:54:56Z

Only the code coverage tasks are failing.

kumare3 · 2021-11-05T21:26:16Z

flytekit/clis/sdk_in_container/init.py

+
+
+@click.command("init")
+@click.option("--template", default="simple-example", help="cookiecutter template to be used")


Suggested change

@click.option("--template", default="simple-example", help="cookiecutter template to be used")

@click.option("--template", default="simple-example", help="cookiecutter template folder name to be used in the repo - https://github.com/flyteorg/flytekit-python-template.git")

Also eventually I think the repo itself should be parameterizable

this idea of exposing the repo is interesting. So your expectation is that people will have their own private (or public forks) of the repo? In any case, if the need arises we can definitely go that way, but not in this PR, ok?

kumare3 · 2021-11-05T21:26:29Z

Small comment then +1

Signed-off-by: Eduardo Apolinario <[email protected]>

…late Signed-off-by: Eduardo Apolinario <[email protected]>

requirements.in

Signed-off-by: Eduardo Apolinario <[email protected]>

flytekit/clis/sdk_in_container/init.py

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add cookiecutter to requirements and pyflyte init Signed-off-by: Eduardo Apolinario <[email protected]> * Invoke cookiecutter with the overridden configuration. Signed-off-by: Eduardo Apolinario <[email protected]> * Comment the use of directory in the call to cookiecutter. Signed-off-by: Eduardo Apolinario <[email protected]> * Use the original repo and do not clobber existing files+directories Signed-off-by: Eduardo Apolinario <[email protected]> * Add example sub-command Signed-off-by: Eduardo Apolinario <[email protected]> * make requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Fix test_dataclass_transformer test Signed-off-by: Eduardo Apolinario <[email protected]> * Constrain mashmallow-jsonschema temporarily Signed-off-by: Eduardo Apolinario <[email protected]> * Add --template flag and project-name argument to command Signed-off-by: Eduardo Apolinario <[email protected]> * Revert changes to dataclass transformer tests Signed-off-by: Eduardo Apolinario <[email protected]> * Constrain marshmallow-jsonschema in dev-requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Constrain marshmallow-jsonschema in dev-requirements" This reverts commit bf7f22d. * Fix typo in Makefile Signed-off-by: Eduardo Apolinario <[email protected]> * Fix regeneration of spark2 requirements file Signed-off-by: Eduardo Apolinario <[email protected]> * Use click for prompts Signed-off-by: Eduardo Apolinario <[email protected]> * Changes default workflow name to `my_wf` Signed-off-by: Eduardo Apolinario <[email protected]> * Remove the two inputs from simple-example Signed-off-by: Eduardo Apolinario <[email protected]> * Add more descriptive help for the template flag Signed-off-by: Eduardo Apolinario <[email protected]> * Set cookiecutter in setup.py Signed-off-by: Eduardo Apolinario <[email protected]> * Default top level directory to flyte (as opposed to myapp) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Robert Everson <[email protected]>

eapolinario added 5 commits October 27, 2021 19:08

Add cookiecutter to requirements and pyflyte init

fd5dea3

Signed-off-by: Eduardo Apolinario <[email protected]>

Invoke cookiecutter with the overridden configuration.

79da424

Signed-off-by: Eduardo Apolinario <[email protected]>

Comment the use of directory in the call to cookiecutter.

38b0d88

Signed-off-by: Eduardo Apolinario <[email protected]>

Use the original repo and do not clobber existing files+directories

688f214

Signed-off-by: Eduardo Apolinario <[email protected]>

Add example sub-command

d2130cc

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario requested review from kumare3 and wild-endeavor as code owners November 4, 2021 23:46

eapolinario added 3 commits November 4, 2021 16:54

make requirements

79e9c1d

Signed-off-by: Eduardo Apolinario <[email protected]>

Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…

ab4c987

…late

Fix test_dataclass_transformer test

94f22cf

Signed-off-by: Eduardo Apolinario <[email protected]>

kumare3 reviewed Nov 5, 2021

View reviewed changes

flytekit/clis/sdk_in_container/init.py Outdated Show resolved Hide resolved

kumare3 reviewed Nov 5, 2021

View reviewed changes

eapolinario added 7 commits November 4, 2021 23:28

Constrain mashmallow-jsonschema temporarily

b441cf6

Signed-off-by: Eduardo Apolinario <[email protected]>

Add --template flag and project-name argument to command

e1a6a4d

Signed-off-by: Eduardo Apolinario <[email protected]>

Revert changes to dataclass transformer tests

e5ad645

Signed-off-by: Eduardo Apolinario <[email protected]>

Constrain marshmallow-jsonschema in dev-requirements

bf7f22d

Signed-off-by: Eduardo Apolinario <[email protected]>

Revert "Constrain marshmallow-jsonschema in dev-requirements"

25832e2

This reverts commit bf7f22d.

Fix typo in Makefile

25aa23c

Signed-off-by: Eduardo Apolinario <[email protected]>

Fix regeneration of spark2 requirements file

f547ed9

Signed-off-by: Eduardo Apolinario <[email protected]>

kumare3 reviewed Nov 5, 2021

View reviewed changes

Use click for prompts

3f91dca

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario force-pushed the cookie-cutter-initial-template branch from d606b40 to 3f91dca Compare November 5, 2021 16:03

eapolinario added 3 commits November 5, 2021 11:10

Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…

e8ea83a

…late Signed-off-by: Eduardo Apolinario <[email protected]>

Changes default workflow name to my_wf

b725095

Signed-off-by: Eduardo Apolinario <[email protected]>

Remove the two inputs from simple-example

f492bb5

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario mentioned this pull request Nov 5, 2021

Update the Getting started docs to use pyflyte init flyteorg/flyte#1801

Merged

kumare3 reviewed Nov 5, 2021

View reviewed changes

Add more descriptive help for the template flag

b0fd8b4

Signed-off-by: Eduardo Apolinario <[email protected]>

kumare3 previously approved these changes Nov 8, 2021

View reviewed changes

Merge remote-tracking branch 'origin' into cookie-cutter-initial-temp…

0420410

…late Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario dismissed kumare3’s stale review via 0420410 November 8, 2021 21:16

wild-endeavor reviewed Nov 8, 2021

View reviewed changes

requirements.in Outdated Show resolved Hide resolved

Set cookiecutter in setup.py

3efecec

Signed-off-by: Eduardo Apolinario <[email protected]>

kumare3 reviewed Nov 9, 2021

View reviewed changes

flytekit/clis/sdk_in_container/init.py Outdated Show resolved Hide resolved

Default top level directory to flyte (as opposed to myapp)

bf60412

Signed-off-by: Eduardo Apolinario <[email protected]>

wild-endeavor approved these changes Nov 9, 2021

View reviewed changes

eapolinario merged commit d846ed5 into master Nov 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cookie cutter initial template #738

Cookie cutter initial template #738

eapolinario commented Nov 4, 2021 •

edited

Loading

eapolinario commented Nov 5, 2021

kumare3 Nov 5, 2021

eapolinario Nov 5, 2021

eapolinario Nov 5, 2021

codecov bot commented Nov 5, 2021 •

edited

Loading

kumare3 Nov 5, 2021

eapolinario Nov 5, 2021

eapolinario commented Nov 5, 2021

kumare3 Nov 5, 2021

eapolinario Nov 5, 2021

kumare3 commented Nov 5, 2021



		@click.command("init")
		@click.option("--template", default="simple-example", help="cookiecutter template to be used")

Cookie cutter initial template #738

Cookie cutter initial template #738

Conversation

eapolinario commented Nov 4, 2021 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

eapolinario commented Nov 5, 2021

kumare3 Nov 5, 2021

Choose a reason for hiding this comment

eapolinario Nov 5, 2021

Choose a reason for hiding this comment

eapolinario Nov 5, 2021

Choose a reason for hiding this comment

codecov bot commented Nov 5, 2021 • edited Loading

Codecov Report

kumare3 Nov 5, 2021

Choose a reason for hiding this comment

eapolinario Nov 5, 2021

Choose a reason for hiding this comment

eapolinario commented Nov 5, 2021

kumare3 Nov 5, 2021

Choose a reason for hiding this comment

eapolinario Nov 5, 2021

Choose a reason for hiding this comment

kumare3 commented Nov 5, 2021

eapolinario commented Nov 4, 2021 •

edited

Loading

codecov bot commented Nov 5, 2021 •

edited

Loading