job priority #1466

golanha · 2021-12-02T14:44:42Z

This change is

* Add algorithm-optimizer module * Add optimizer to api server * no lint * Add Test * remove mem * increase memory * optimizer schema * validations * sampler and validation on store * correct test * handle optunaboard * optimizer pod deploymnet fixes * lock file * update consts * volume mounts in spec * missing argument in volume mounts * storage as datasource * env variable set * set env variable * env set correction * rest api for optunaboard and algorithm-operator handle optunaboard * deepscan * boardURL * rename optimizer folder to hyperparams-tuner * optimizer to hyperparams-tuner * optimizer to hyperparams-tuner * get optunaboard version from values * lint * change validation * CR * Consts typo * hyperparams not mandatory incase of grid sample * test failed * CR

* Add algorithm-optimizer module * Add optimizer to api server * no lint * Add Test * remove mem * increase memory * optimizer schema * validations * sampler and validation on store * correct test * handle optunaboard * optimizer pod deploymnet fixes * lock file * update consts * volume mounts in spec * missing argument in volume mounts * storage as datasource * env variable set * set env variable * env set correction * rest api for optunaboard and algorithm-operator handle optunaboard * deepscan * boardURL * rename optimizer folder to hyperparams-tuner * optimizer to hyperparams-tuner * optimizer to hyperparams-tuner * get optunaboard version from values * lint * change validation * CR * Consts typo * hyperparams not mandatory incase of grid sample * test failed * CR .... bump version [skip ci]

golanha · 2021-12-05T14:19:27Z

/deploy

golanha · 2021-12-05T15:20:49Z

/deploy

golanha · 2021-12-06T12:58:07Z

/deploy

golanha · 2021-12-15T15:03:46Z

/deploy

golanha · 2021-12-15T17:54:42Z

/deploy

golanha

Reviewable status: 80 of 103 files reviewed, 9 unresolved discussions (waiting on @nassiharel and @yehiyam)

core/pipeline-driver-queue/lib/service/preferred-jobs.js, line 10 at r14 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

no async

Done.

Code quote:

 async deletePreferredJobs

core/pipeline-driver-queue/lib/service/preferred-jobs.js, line 44 at r14 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

This code is a little cumbersome.
Try to split to functions and add some comments

Done.

core/pipeline-driver-queue/lib/validation/preference.js, line 20 at r14 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

what if we will add two more variables?

if ((tag && jobId) || (tag && pipeline) || (pipeline && jobId)
|| (pipeline && jobId)|| (pipeline && jobId)|| (pipeline && jobId)
|| (pipeline && jobId)|| (pipeline && jobId)|| (pipeline && jobId)|| (pipeline && jobId)
) {

Done.

nassiharel

Reviewed 3 of 14 files at r10, 5 of 6 files at r13, 2 of 2 files at r14, 5 of 10 files at r15, all commit messages.
Reviewable status: 94 of 103 files reviewed, 3 unresolved discussions (waiting on @yehiyam)

nassiharel · 2021-12-22T09:31:11Z

core/pipeline-driver-queue/api/rest-api/routes/preferred.js

+const routes = () => {
+    const router = RestServer.router();
+
+    router.get('/', async (req, res) => {


this is not async

nassiharel · 2021-12-22T09:31:21Z

core/pipeline-driver-queue/api/rest-api/routes/preferred.js

+        const response = preferredService.getPreferredJobsList();
+        res.json(response);
+    });
+    router.post('/', async (req, res) => {


this is not async

nassiharel · 2021-12-22T09:34:40Z

core/pipeline-driver-queue/api/rest-api/routes/preferred.js

+        const response = preferredService.getPreferredJobsList();
+        res.json(response);
+    });
+    router.post('/', async (req, res) => {


This is not a best practice, it is confusing...
The user can set the same jobs ids in addedJobs and removedJobs...
I think you need to add router.delete with a body.

nassiharel · 2021-12-22T09:35:24Z

core/pipeline-driver-queue/config/main/config.base.js

+    poweredBy: 'HKube Pipeline Driver queue',
+    bodySizeLimit: process.env.BODY_SIZE_LIMIT || '50mb'
+};
+config.swagger = {


Why you need this?

nassiharel · 2021-12-22T09:36:59Z

core/pipeline-driver-queue/lib/validation/preference.js

+            preferenceRequest
+        );
+        const { position, query } = preferenceRequest.addedJobs;
+        if ((position === 'first' || position === 'last')) {


if ((position === 'first' || position === 'last') && query) {

core/pipeline-driver-queue/lib/validation/preference.js

core/pipeline-driver-queue/package.json

golanha

Reviewable status: 87 of 104 files reviewed, 9 unresolved discussions (waiting on @nassiharel and @yehiyam)

core/pipeline-driver-queue/package.json, line 32 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

Do you need performance-now?

I didn't add it. It is currently used in scoreing.js

core/pipeline-driver-queue/config/main/config.base.js, line 18 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

Why you need this?

You are right, I didn't use it.

core/pipeline-driver-queue/lib/validation/preference.js, line 13 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

if ((position === 'first' || position === 'last') && query) {

Done.

core/pipeline-driver-queue/lib/validation/preference.js, line 26 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

You can say

 if (numberOfFields === 0) {
      throw new InvalidDataError('Query must contain one of jobId ,tag ,pipelineName');
}

Done.

core/pipeline-driver-queue/api/rest-api/routes/preferred.js, line 7 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

this is not async

Done.

core/pipeline-driver-queue/api/rest-api/routes/preferred.js, line 11 at r17 (raw file):

Previously, NassiHarel (Nassi Harel) wrote…

This is not a best practice, it is confusing...
The user can set the same jobs ids in addedJobs and removedJobs...
I think you need to add router.delete with a body.

Done.

golanha · 2021-12-23T11:19:19Z

/deploy

nassiharel

Reviewable status: 87 of 104 files reviewed, 3 unresolved discussions (waiting on @nassiharel and @yehiyam)

golanha · 2021-12-26T08:33:36Z

/deploy

golanha · 2021-12-26T09:42:09Z

/deploy

* job priority * Optimizer (#1430) * Add algorithm-optimizer module * Add optimizer to api server * no lint * Add Test * remove mem * increase memory * optimizer schema * validations * sampler and validation on store * correct test * handle optunaboard * optimizer pod deploymnet fixes * lock file * update consts * volume mounts in spec * missing argument in volume mounts * storage as datasource * env variable set * set env variable * env set correction * rest api for optunaboard and algorithm-operator handle optunaboard * deepscan * boardURL * rename optimizer folder to hyperparams-tuner * optimizer to hyperparams-tuner * optimizer to hyperparams-tuner * get optunaboard version from values * lint * change validation * CR * Consts typo * hyperparams not mandatory incase of grid sample * test failed * CR * Optimizer (#1430) * Add algorithm-optimizer module * Add optimizer to api server * no lint * Add Test * remove mem * increase memory * optimizer schema * validations * sampler and validation on store * correct test * handle optunaboard * optimizer pod deploymnet fixes * lock file * update consts * volume mounts in spec * missing argument in volume mounts * storage as datasource * env variable set * set env variable * env set correction * rest api for optunaboard and algorithm-operator handle optunaboard * deepscan * boardURL * rename optimizer folder to hyperparams-tuner * optimizer to hyperparams-tuner * optimizer to hyperparams-tuner * get optunaboard version from values * lint * change validation * CR * Consts typo * hyperparams not mandatory incase of grid sample * test failed * CR .... bump version [skip ci] * Optimizer (#1430) * Add algorithm-optimizer module * Add optimizer to api server * no lint * Add Test * remove mem * increase memory * optimizer schema * validations * sampler and validation on store * correct test * handle optunaboard * optimizer pod deploymnet fixes * lock file * update consts * volume mounts in spec * missing argument in volume mounts * storage as datasource * env variable set * set env variable * env set correction * rest api for optunaboard and algorithm-operator handle optunaboard * deepscan * boardURL * rename optimizer folder to hyperparams-tuner * optimizer to hyperparams-tuner * optimizer to hyperparams-tuner * get optunaboard version from values * lint * change validation * CR * Consts typo * hyperparams not mandatory incase of grid sample * test failed * CR .... bump version [skip ci] * job priority * deep scan * exclude failing test * tests * update db package version * package-lock * expose rest, use 2 queues * update preference api * remove redundant file * error handling * load data and tests * CR * CR * api update * upgrade consts * test accoring to api changes * ingress to driver-queue * swagger ui path to driver-queue/preferred * CR * deepscan * fix api test * CR * add delete test * CR * event only when realy dequeueing * driver-queue -> queue * dequeued length * no metrics updates for preferred queue Co-authored-by: GitHub Action <[email protected]> .... bump version [skip ci]

golanha and others added 6 commits December 2, 2021 16:05

job priority

622a480

job priority

00d5d18

Merge branch 'master' into prioritized_jobs

5037e47

github-actions bot added HKube: API-Server HKube: Datasources HKube: GC-Service HKube: OpenAPI-Spec HKube: Pipeline Driver Queue labels Dec 2, 2021

golanha added 5 commits December 2, 2021 17:20

deep scan

fbaee4d

exclude failing test

8202259

tests

1aed123

update db package version

3fbd089

package-lock

dcd0e18

golanha requested a review from nassiharel December 5, 2021 14:06

hkube-ci had a problem deploying to dev December 5, 2021 14:19 Failure

hkube-ci had a problem deploying to dev December 5, 2021 15:21 Failure

hkube-ci temporarily deployed to dev December 6, 2021 12:58 Inactive

golanha added 3 commits December 14, 2021 08:00

expose rest, use 2 queues

9e56e5c

update preference api

f463a2a

remove redundant file

500afce

hkube-ci temporarily deployed to dev December 15, 2021 15:04 Inactive

error handling

2b7bd10

golanha added 2 commits December 21, 2021 13:46

CR

45125a0

deepscan

cf80e23

golanha commented Dec 21, 2021

View reviewed changes

golanha requested a review from nassiharel December 21, 2021 11:52

nassiharel reviewed Dec 21, 2021

View reviewed changes

fix api test

9c22ec9

golanha requested a review from nassiharel December 22, 2021 09:15

nassiharel reviewed Dec 22, 2021

View reviewed changes

CR

4ffbaaa

golanha commented Dec 23, 2021

View reviewed changes

golanha requested a review from nassiharel December 23, 2021 07:09

hkube-ci temporarily deployed to dev December 23, 2021 11:19 Inactive

add delete test

e2f3b2d

nassiharel previously approved these changes Dec 23, 2021

View reviewed changes

CR

4418884

golanha dismissed nassiharel’s stale review via 4418884 December 23, 2021 14:23

Merge branch 'master' into prioritized_jobs

13c44db

hkube-ci temporarily deployed to dev December 26, 2021 08:34 Inactive

golanha added 3 commits December 26, 2021 11:04

event only when realy dequeueing

7b7a83b

driver-queue -> queue

4ce36df

dequeued length

e6a32b7

hkube-ci temporarily deployed to dev December 26, 2021 09:42 Inactive

no metrics updates for preferred queue

e6f4fb8

golanha merged commit 4f35822 into master Dec 27, 2021

golanha deleted the prioritized_jobs branch December 27, 2021 07:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

job priority #1466

job priority #1466

golanha commented Dec 2, 2021 •

edited by maty21

Loading

golanha commented Dec 5, 2021

golanha commented Dec 5, 2021

golanha commented Dec 6, 2021

golanha commented Dec 15, 2021

golanha commented Dec 15, 2021

golanha left a comment

nassiharel left a comment

nassiharel Dec 22, 2021

nassiharel Dec 22, 2021

nassiharel Dec 22, 2021

nassiharel Dec 22, 2021

nassiharel Dec 22, 2021

golanha left a comment

golanha commented Dec 23, 2021

nassiharel left a comment

golanha commented Dec 26, 2021

golanha commented Dec 26, 2021

job priority #1466

job priority #1466

Conversation

golanha commented Dec 2, 2021 • edited by maty21 Loading

golanha commented Dec 5, 2021

golanha commented Dec 5, 2021

golanha commented Dec 6, 2021

golanha commented Dec 15, 2021

golanha commented Dec 15, 2021

golanha left a comment

Choose a reason for hiding this comment

nassiharel left a comment

Choose a reason for hiding this comment

nassiharel Dec 22, 2021

Choose a reason for hiding this comment

nassiharel Dec 22, 2021

Choose a reason for hiding this comment

nassiharel Dec 22, 2021

Choose a reason for hiding this comment

nassiharel Dec 22, 2021

Choose a reason for hiding this comment

nassiharel Dec 22, 2021

Choose a reason for hiding this comment

golanha left a comment

Choose a reason for hiding this comment

golanha commented Dec 23, 2021

nassiharel left a comment

Choose a reason for hiding this comment

golanha commented Dec 26, 2021

golanha commented Dec 26, 2021

golanha commented Dec 2, 2021 •

edited by maty21

Loading