Identify next steps until we can run this

[bradenmacdonald]

Identify what remaining features/support are required before we can start using this for hosting instance(s) in production at one of the participating providers.

• What is our definition of "production ready"?
• e.g. do we need autoscaling support before we can use this "in production"? do we need CI doing helm chart tests? Do we need better monitoring?
• How does this integrate with existing tooling the provider has (e.g. Grove)?

[bradenmacdonald]

@gabor-boros please investigate this for us, or assign to someone from our team.

[gabor-boros]

@bradenmacdonald I was lagging behind with OpenCraft-related commitments that I had to sort out first. Unfortunately, I had no time to get back to this.

Also, today, it turned out that I'd have to be a core contributor to be able to review tasks here. I guess that stands for working on issues too, though I have/had concerns with from Serenity point of view at the moment. Ref: #25 (comment)

Taking these into account, I may not be the best to work on this at the moment. Or should I start work on this without being a CC, @antoviaque ?

[felipemontoya]

I don't think you need to be a core contributor to review. Only to be assigned as a reviewer. All and any feedback you can give at any time will be appreciated.

[MoisesGSalas]

At the moment we have merged functionality for a shared elasticsearch, but we still need openedx/edx-search#130 to be merged if we want to avoid forcing a new image build with a non-standard dependency.

We are going to keep discussing this topic at eduNEXT internally.

[antoviaque]

@gabor-boros Yup, as noted in other related ticket, you don't need to be a core contributor to do the review itself, only to be able to give a final 👍 and merge.

[MoisesGSalas]

I did a very small review and I came to the following conclusion:

If we consider the simplest most barebones production installation that we would feel comfortably running we would consider that the project would needs to offer at least this basic features:

• A central load balancer with automatic SSL certificates.
• Autoscaling of both pods and nodes.
• Some level of monitoring tools.

Of these features, the first one is already implemented, and the other two are being tracked in #2 and #3. In addition, we also implemented a shared elasticsearch for the whole cluster, but it depends on changes on external packages/services that has not been merged/released (edx-search#130 and cs_comments_service#404).

We also have some housekeeping to do, namely renaming every mention of tutor-multi to harmony, hosting the chart as a chart repository and releasing a new version.

I think that would be the bare minimum for one of the providers to actually start using the tool. Nevertheless, we could also improve our current documentation to ease the adoption of the project from people that have not been part of it since the beginning.

In summary:

• A central load balancer with automatic SSL certificates.
• Autoscaling of both pods and nodes.
• Some level of monitoring tools.
• Rename tutor-multi references
• Defining the Open edX release this is compatible with (seems like we are aiming for Palm).
• Host the repository.
• Make a new release of the chart.

[gabor-boros]

As discussed I did the review as well. I completely agree what @MoisesGSalas said above. Without proper monitoring and auto-scaling, I wouldn't consider anything "production ready". Besides that, I would add the following:

• Setting node affinity for resources defined by Tutor (lms, cms, workers, redis, etc)
• Some way to size volumes attached to instances, elastic search, redis, etc
• Some way to run periodic/cron jobs

Refining "Some level of monitoring tools", I would say that for production-grade cluster we would need the folllowing (maybe not every bit is needed for every provider, but could be turned on/off upon need):

• Alertmanager
• Prometheus
• Grafana
• Log collection and aggregation (i.e. Opensearch + Filebeat)

Also, if we want to make sure the chart is production ready, we should have some sort of automation for

• Chart testing
• Chart packaging
• Chart releasing
• README formatting and markdown validation

Lastly, this is a gray area in terms of responsibilities/project boundary but the ability to support backward compatible edX installations. At the moment (IIRC) Tutor is not supporting past edX installations. The latest versions of Tutor only supports the latest versions of edX instances and no backporting is supported (though I may remember wrongly). From a production point of view, this could be quickly become a difficulty, especially if a provider have to deal with instances that cannot be immediately updated to new versions.

[felipemontoya]

@bradenmacdonald @jfavellar90 @gabor-boros @MoisesGSalas @cmltaWt0 tagging you here to please take an async look to figure out what should be the next step for this project.

[bradenmacdonald]

In terms of us deploying this in production, we have scheduled a discovery task for this for our team (SE-5971) to work out the plan, but not yet sure on when we can fit it into a sprint.

[bradenmacdonald]

OpenCraft is now using Harmony in prod, so I'm going to close this :)