Built-in ergo scheduler functionality

[rickyviank]

Overview

The current ergo components can be invoked by other components through rabbitmq amqp protocol by providing event-based parameters like pubtopic, subtopic, heartbeat, etc. However, current ergo components are not equipped with the ability to invoke itself via time-based parameter such as a pre-configured schedule. Hence, this feature request focus on exposing a new config field in ergo component yaml file that will allow ergo developer to specify the invoke method of time-based parameters and/or event-based parameters.

With time-based parameters, ergo developer should be able to specify the rate at which they wish the components is invoked in cron syntax. In the case where the component is only configured with time-based parameters, the component will run according to the specified rate in the config, and cannot be triggered by other components. However, if the components have both time-based parameters and event-based parameters, components should be triggered by other components through the primary exchange or by the specified time.

Stakeholders

This feature is requested by DPI as it helps accelerate the transition from old arch (apipersist) to new arch (qed) while remaining agile. Other stakeholders that will required schedule-based invocations of components will also greatly benefit from this feature. In order to achieve this milestone, DPI will require solid support from the INFRA team to build this feature in ergo.

Rationale

The work around that DPI did to get the api connectors invoked hourly is by specifying the cron parameters in a serverless.yaml file, whereby serverless will create an AWS lambda with the cron job, and the lambda will publish an event to AWS event bridge, which is then routed to our components that are packaged in AWS ECS, resulting the components to be invoked.

The most obvious benefit of having scheduler built into ergo is to enable the independence of time-based components from AWS services which has a different deployment process as compared to other event-based components. with 1-click deployment coming soon, making changes to components parameters (be it time-based or event-based) can and should be as seamless as going to the component ergo config yaml file, and edit the parameters, instead of editing the serverless.yaml file, which seemingly feel unrelated to the components.

With serverless.yaml configuration, ergo developer also cannot do local testing of their time-based components with ease. Hence, most of the time, they deploy their components, edit the serverless.yaml and hope for the best. This greatly slow down our development cycle, and made it hard to catch runtime errors.

There are multiple components currently in DPI that require time-based parameters:

• Scheduler
• Data collector
• Outage detector
• and more coming up.

Useful links

• https://blog.rabbitmq.com/posts/2015/04/scheduling-messages-with-rabbitmq/