Don't error when attempting to trigger schedules for inactive projects

[katrogan]

Tracking issue

Why are the changes needed?

Currently the scheduler will fail its scheduling loop when any execution fails to trigger
see here:

flyte/flyteadmin/scheduler/core/gocron_scheduler.go

Line 189 in a13a63d

err := g.CatchUpSingleSchedule(ctx, job.schedule, *fromTime, until)

which is called in the primary Run method here:

flyte/flyteadmin/scheduler/schedule_executor.go

Lines 82 to 89 in a13a63d

	af := futures.NewAsyncFuture(ctx, func(ctx context.Context) (interface{}, error) {
	return gcronScheduler.CatchupAll(ctx, currTime), nil
	})
	isCatchupSuccess, err := af.Get(ctx)
	if err != nil {
	logger.Errorf(ctx, "failed to get future value for catchup due to %v", err)
	return err
	}

which results in a fatal log here:

flyte/flyteadmin/scheduler/start.go

Line 58 in a13a63d

logger.Fatalf(ctx, "Flyte native scheduler failed to start due to %v", err)

which terminates the scheduler.

What changes were proposed in this pull request?

This changes introduces a well structured error message type when a project already exists, and configures the scheduler to move on gracefully when an active scheduler is triggered for an archived project.

This mimics the existing behavior for the scheduler encountering an edge case when it attempts to trigger a formerly active schedule: https://github.com/flyteorg/flyte/blob/master/flyteadmin/scheduler/executor/executor_impl.go#L94

How was this patch tested?

I ran flyte sandbox locally and activated a once minute launch plan. I archived the parent project (flytesnacks) and observed the scheduler continued to run.

{"json":{"src":"execution_manager.go:833"},"level":"debug","msg":"Failed to validate ExecutionCreateRequest {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:0x1400033c008} sizeCache:0 unknownFields:[] Project:flytesnacks Domain:development Name:f182916c04b5dc387000 Spec:launch_plan:{resource_type:LAUNCH_PLAN project:\"flytesnacks\" domain:\"development\" name:\"int_doubler_wf_scheduled\" version:\"kxtp6OI3ahFkpOj50oiQ2g\"} metadata:{mode:SCHEDULED scheduled_at:{seconds:1723467046}} Inputs:<nil> Org:} with err project [flytesnacks] is not active","ts":"2024-08-12T14:50:46+02:00"}
{"json":{"src":"auth_interceptor.go:155"},"level":"debug","msg":"Request failed due to [rpc error: code = InvalidArgument desc = project [flytesnacks] is not active]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2024-08-12T14:50:46+02:00"}
{"json":{"routine":"jobfunc-8988759870306525270","src":"executor_impl.go:118"},"level":"debug","msg":"project flytesnacks is inactive, ignoring schedule create failure for {BaseModel:{ID:2 CreatedAt:2024-08-12 14:47:44.00598 +0200 CEST UpdatedAt:2024-08-12 14:47:44.00598 +0200 CEST DeletedAt:<nil>} SchedulableEntityKey:{Project:flytesnacks Domain:development Name:int_doubler_wf_scheduled Version:kxtp6OI3ahFkpOj50oiQ2g} CronExpression: FixedRateValue:1 Unit:MINUTE KickoffTimeInputArg: Active:0x1400539c6d0}","ts":"2024-08-12T14:50:46+02:00"}
{"json":{"routine":"jobfunc-8988759870306525270","src":"executor_impl.go:129"},"level":"info","msg":"successfully fired the request for schedule {BaseModel:{ID:2 CreatedAt:2024-08-12 14:47:44.00598 +0200 CEST UpdatedAt:2024-08-12 14:47:44.00598 +0200 CEST DeletedAt:<nil>} SchedulableEntityKey:{Project:flytesnacks Domain:development Name:int_doubler_wf_scheduled Version:kxtp6OI3ahFkpOj50oiQ2g} CronExpression: FixedRateValue:1 Unit:MINUTE KickoffTimeInputArg: Active:0x1400539c6d0} for time 2024-08-12 14:50:46 +0200 CEST","ts":"2024-08-12T14:50:46+02:00"}

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 45 lines in your changes missing coverage. Please review.

Project coverage is 36.17%. Comparing base (9cfb4e3) to head (38fba77).
Report is 130 commits behind head on master.

Files with missing lines	Patch %	Lines
flyteidl/gen/pb-go/flyteidl/admin/project.pb.go	2.63%	37 Missing ⚠️
flyteadmin/scheduler/executor/executor_impl.go	58.33%	3 Missing and 2 partials ⚠️
flyteadmin/pkg/errors/errors.go	66.66%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5649      +/-   ##
==========================================
- Coverage   36.18%   36.17%   -0.01%     
==========================================
  Files        1302     1302              
  Lines      109428   109484      +56     
==========================================
+ Hits        39594    39608      +14     
- Misses      65699    65738      +39     
- Partials     4135     4138       +3     

Flag	Coverage Δ
unittests-datacatalog	51.37% <ø> (ø)
unittests-flyteadmin	55.33% <63.63%> (+<0.01%)	⬆️
unittests-flytecopilot	12.17% <ø> (ø)
unittests-flytectl	62.32% <ø> (+0.04%)	⬆️
unittests-flyteidl	7.08% <2.63%> (-0.01%)	⬇️
unittests-flyteplugins	53.31% <ø> (ø)
unittests-flytepropeller	41.74% <ø> (ø)
unittests-flytestdlib	55.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eapolinario]

Looks great, just a couple of nits.

[katrogan]

PTAL @eapolinario