Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare monorepo import other repos #14

Merged
merged 1,996 commits into from
Sep 7, 2023

Conversation

eapolinario
Copy link
Owner

Tracking issue

Describe your changes

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Screenshots

Note to reviewers

katrogan and others added 30 commits December 12, 2022 14:24
Signed-off-by: Katrina Rogan <[email protected]>

Signed-off-by: Katrina Rogan <[email protected]>
* fixed nil panic on gate node sleep check

Signed-off-by: Dan Rammer <[email protected]>

* lint issue

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Dan Rammer <[email protected]>
…teorg#495)

* adding CheckAndFetchInputsForExecution on single task executions

Signed-off-by: Dan Rammer <[email protected]>

* fixed unit test

Signed-off-by: Dan Rammer <[email protected]>

* fixed typo

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Dan Rammer <[email protected]>
* Update kafka config

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* update cnofig

Signed-off-by: Kevin Su <[email protected]>

* Update pkg/runtime/interfaces/application_configuration.go

Co-authored-by: Dan Rammer <[email protected]>
Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Dan Rammer <[email protected]>
* expose and use kubeclient configs if available

Signed-off-by: Babis Kiosidis <[email protected]>

* omit empty kubeclientconfig

Signed-off-by: Babis Kiosidis <[email protected]>

* setting configuration on all kubeclients

Signed-off-by: Daniel Rammer <[email protected]>

* addressing PR renaming comments

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Babis Kiosidis <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Dan Rammer <[email protected]>
Co-authored-by: Babis Kiosidis <[email protected]>
* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <[email protected]>

* Merge pod template spec with pod spec in separate func

Signed-off-by: Fabio Grätz <[email protected]>

* Apply pod template to pytorch job pod spec

Signed-off-by: Fabio Grätz <[email protected]>

* Handle nil podspecs before merging

Signed-off-by: Fabio Grätz <[email protected]>

* Pass both default and primare container name to MergePodSpecs

Signed-off-by: Fabio Grätz <[email protected]>

* Move podSpec.DeepCopy into MergePodSpecs

Signed-off-by: Fabio Grätz <[email protected]>

* Add tests

Signed-off-by: Fabio Grätz <[email protected]>

* Merge pod template into tfjob and mpijob

Signed-off-by: Fabio Grätz <[email protected]>

* Lint

Signed-off-by: Fabio Grätz <[email protected]>

* Correct usage of default and primate container (template) name

Signed-off-by: Fabio Grätz <[email protected]>

* Override mpi default container name

Signed-off-by: Fabio Grätz <[email protected]>

* Carry over ObjectMeta from pod template

Signed-off-by: Fabio Grätz <[email protected]>

* Remove old `TestBuildPodWithSpec` test

Signed-off-by: Fabio Grätz <[email protected]>

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
* update idl

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* update loader

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* Updare plugin

Signed-off-by: Kevin Su <[email protected]>

* Updare plugin

Signed-off-by: Kevin Su <[email protected]>

* Updare plugin

Signed-off-by: Kevin Su <[email protected]>

* Updare plugin

Signed-off-by: Kevin Su <[email protected]>

* Updare plugin

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* update

Signed-off-by: Kevin Su <[email protected]>

* updateidl

Signed-off-by: Kevin Su <[email protected]>

* go import

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
…eorg#303)

* PodSpec NodeSelectors overwrite the default k8s plugin settings

Signed-off-by: Dan Rammer <[email protected]>

* added unit test

Signed-off-by: Dan Rammer <[email protected]>

* better node selector value

Signed-off-by: Dan Rammer <[email protected]>

* fixed issue with unionmaps where k8s plugin overrides labels

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Dan Rammer <[email protected]>
…lyteorg#511)

* fixing issue where k8s plugin default labels override those set on pod

Signed-off-by: Dan Rammer <[email protected]>

* updated flyteplugins dependency

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Dan Rammer <[email protected]>
* Doc Hub Proposal

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* fix test error

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* update database schema

Signed-off-by: Kevin Su <[email protected]>

* update

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* More tests

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* list description entity

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

* register docs when creating task

Signed-off-by: Kevin Su <[email protected]>

* update go.sum

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* Add short description to workflow

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* more test

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* Lint

Signed-off-by: Kevin Su <[email protected]>

* update to one transation

Signed-off-by: Kevin Su <[email protected]>

* Fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* update comment

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* Bump idl

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* Add identifier

Signed-off-by: Kevin Su <[email protected]>

* update idl

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* merged master

Signed-off-by: Kevin Su <[email protected]>

* Address comment

Signed-off-by: Kevin Su <[email protected]>

* Address comment

Signed-off-by: Kevin Su <[email protected]>

* update migrations.go

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

* Merged master

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
…yteorg#512)

* added random print statements

Signed-off-by: Dan Rammer <[email protected]>

* random print statement

Signed-off-by: Dan Rammer <[email protected]>

* infering gate node approve condition input and output variables from upstream nodes

Signed-off-by: Dan Rammer <[email protected]>

* removed print statements

Signed-off-by: Dan Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Dan Rammer <[email protected]>

Signed-off-by: Dan Rammer <[email protected]>
Signed-off-by: Ankit Goyal <[email protected]>

Signed-off-by: Ankit Goyal <[email protected]>
* First working version with DaskCluster

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update plugin

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add container customization

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add correct `getTaskPhase`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Refactor dask.go

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use new dask operator which includes a status

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add first tests and use data from flyteidl

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Refactor tests

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add support for custom namespace

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add support for passing on annotations

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add support for env vars

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add default container logic to job runner

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add TestGetTaskPhaseDask

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add logs to task info

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use tagged version of dask go operator

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix linting issues

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Refactor `ToK8sResourceRequirements`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use platform resources by default

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix incorrect resources

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Remove namespace

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Don't restart job runner and scheduler

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Run `go mod tidy` after rebase

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Run formatter

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update to new `flyteidl`

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Add support for interruptible workers

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update flytekit to 1.4.0

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix linting errors

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update `flyteidl` to 1.3.2

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Reorder imports

Signed-off-by: Bernhard Stadlbauer <[email protected]>

Signed-off-by: Bernhard Stadlbauer <[email protected]>
* Add dask plugin

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Use `ToK8sResourceRequirements` from flyteplugins

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Run go mod tidy after merge

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Update flyteplugins to 1.0.28

Signed-off-by: Bernhard Stadlbauer <[email protected]>

* Fix linting error

Signed-off-by: Bernhard Stadlbauer <[email protected]>

Signed-off-by: Bernhard Stadlbauer <[email protected]>
* Add tensorflow to loader.go

Signed-off-by: Kevin Su <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: byronhsu <[email protected]>

Signed-off-by: byronhsu <[email protected]>
Signed-off-by: Flyte-Bot <[email protected]>

Signed-off-by: Flyte-Bot <[email protected]>
Co-authored-by: flyte-bot <[email protected]>
katrogan and others added 29 commits August 10, 2023 09:14
* Adding a predredirect hook plugin

Signed-off-by: pmahindrakar-oss <[email protected]>

* Add test logs

Signed-off-by: pmahindrakar-oss <[email protected]>

* test logs

Signed-off-by: pmahindrakar-oss <[email protected]>

* fix

Signed-off-by: pmahindrakar-oss <[email protected]>

* Reading identity token for getting subject

Signed-off-by: pmahindrakar-oss <[email protected]>

* reverting

Signed-off-by: pmahindrakar-oss <[email protected]>

* Adding PreRedirectHookError

Signed-off-by: pmahindrakar-oss <[email protected]>

* Add some more tests

Signed-off-by: pmahindrakar-oss <[email protected]>

* lint fixes

Signed-off-by: pmahindrakar-oss <[email protected]>

* removed log line

Signed-off-by: pmahindrakar-oss <[email protected]>

---------

Signed-off-by: pmahindrakar-oss <[email protected]>
* Fix double http in the Spark Driver UI Link

Signed-off-by: mucahit-kantepe <[email protected]>

* fix test

Signed-off-by: mucahit-kantepe <[email protected]>

* Apply suggestions from code review

Co-authored-by: Ketan Umare <[email protected]>
Signed-off-by: Mücahit Kantepe <[email protected]>

* Update spark.go

Signed-off-by: Mücahit Kantepe <[email protected]>

* fmt

Signed-off-by: mucahit-kantepe <[email protected]>

* updated to append https if neither http or https prefixes spark uri

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: mucahit-kantepe <[email protected]>
Signed-off-by: Mücahit Kantepe <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: mucahit-kantepe <[email protected]>
Co-authored-by: Mücahit Kantepe <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Build single-binary using local components
…y_owner-everywhere

Use github.repository_owner everywhere
…into prepare-monorepo--import-other-repos

Signed-off-by: Eduardo Apolinario <[email protected]>
…nto prepare-monorepo--import-other-repos

Signed-off-by: Eduardo Apolinario <[email protected]>
… into prepare-monorepo--import-other-repos

Signed-off-by: Eduardo Apolinario <[email protected]>
… into prepare-monorepo--import-other-repos

Signed-off-by: Eduardo Apolinario <[email protected]>
…to prepare-monorepo--import-other-repos

Signed-off-by: Eduardo Apolinario <[email protected]>
@eapolinario eapolinario merged commit 6ed8167 into master Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.