Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Housekeeping] [flytekit] Improve handling in pyflyte register/package #2638

Closed
2 tasks done
wild-endeavor opened this issue Jun 28, 2022 · 1 comment
Closed
2 tasks done
Labels
enhancement New feature or request flytekit FlyteKit Python related issue
Milestone

Comments

@wild-endeavor
Copy link
Contributor

wild-endeavor commented Jun 28, 2022

Housekeeping items

Error messages

Both pyflyte register and package do not have very good error messages when you just run them.

 $ pyflyte register
Traceback (most recent call last):
  File "/Users/ytong/envs/flytekit/bin/pyflyte", line 33, in <module>
    sys.exit(load_entry_point('flytekit', 'console_scripts', 'pyflyte')())
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/ytong/envs/flytekit/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/clis/sdk_in_container/register.py", line 139, in register
    zip_file = fast_package(detected_root, output)
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/tools/fast_registration.py", line 30, in fast_package
    ignore = IgnoreGroup(source, [GitIgnore, DockerIgnore, StandardIgnore])
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/tools/ignore.py", line 110, in __init__
    self.ignores = [ignore(root) for ignore in ignores]
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/tools/ignore.py", line 110, in <listcomp>
    self.ignores = [ignore(root) for ignore in ignores]
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/tools/ignore.py", line 74, in __init__
    self.pm = self._parse()
  File "/Users/ytong/go/src/github.com/flyteorg/flytekit/flytekit/tools/ignore.py", line 78, in _parse
    dockerignore = os.path.join(self.root, ".dockerignore")
  File "/usr/local/Cellar/[email protected]/3.8.13_1/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
$ pyflyte package
No packages to scan for flyte entities. Aborting!

We should make it so that the help message shows up along with a nice error when there is an error. See pyflyte run which handles it better.

Temp files

Both pyflyte package --fast and register (and I think run too) will produce a .tgz file in the local directory. This is part of the "fast registration" process (read more about that here). If the user doesn't delete this though, then the next time, the tgz file itself gets bundled with the second .tgz file, thus doubling the size of the file every time. This is just a bit annoying - we should be creating a temporary (but persistent) folder for these somewhere else on the FS. That way, if the user wants to look at it after the fact, they can, but if they don't, they don't need to be bothered with the cleanup and the OS will just remove them eventually.

Misc

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@wild-endeavor wild-endeavor added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Jun 28, 2022
@wild-endeavor wild-endeavor changed the title [Housekeeping] [flytekit] [Housekeeping] [flytekit] Improve handling in pyflyte register/package Jun 28, 2022
@wild-endeavor wild-endeavor added flytekit FlyteKit Python related issue and removed untriaged This issues has not yet been looked at by the Maintainers labels Jun 28, 2022
@wild-endeavor
Copy link
Contributor Author

@madhur-tandon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request flytekit FlyteKit Python related issue
Projects
None yet
Development

No branches or pull requests

1 participant