Noctua in AWS

[kltm]

Project link

https://github.com/orgs/geneontology/projects/161

Project description

Summary

Developers at all levels should be able to test code, put out new code releases, and update data in Noctua in AWS using our documentation.

The bulk of the work is likely to be in https://github.com/geneontology/noctua_app_stack; some may also be in https://github.com/geneontology/devops-documentation, or even https://github.com/geneontology/noctua.

The features that we want to hit over our current setup:

• Use of shared infrastructure (AWS)
• Use of shared devops software
• HTTPS for Noctua
• Improved Java version for Noctua (for minerva updates)

These should mostly be "done" already from previous work, but require testing and more documentation.
Additional targets we'll need to hit:

• (Optionally) export of models from AWS image to GitHub or S3.
• Create a "development" and "production" template set for configurations

Background Details

With the help of @abessiari, we built up a set of DevOps tools and scripts to setup and service our major components, namely the GO API, the RDF endpoint, and AmiGO/GOlr. This development work wrapped up in January, with the Noctua devops partially tested. We want to make sure that this new infrastructure is fully documented to meet the following criteria and use cases

• Is usable for developers who have minimal experience with DevOps
• Introduces the broad concepts of our DevOps
  • Terraform and workspaces (via S3 buckets)
    • Our naming conventions
  • How we refresh and test our services for a data release
    • Services not updated, but new one out with fresh data and old one destroyed
    • New instance up at patterned URL; test; switch Cloudflare/DNS; old instance down
• Given the context above, can be used by a developer to
  • Deploy to a fresh instance at a new random domain for testing/development of software
  • Deploy a fresh instance to switch over to on a data release
  • TODO: Deploy a fresh instance with security updates

(Note that we also should have functional Docker images for local development, but that may be outside the scope of these Terraform-related docs.)

This documentation, out of necessity, may have to touch on topics like the release pipeline, credential distribution, DNS/Cloudflare, and the cert server.

PI

Chris

Product owner (PO)

Seth?

Technical lead (TL)

Anushya

Other personnel (OP)

Marginal: Sierra, Dustin

Technical specs

N/A

Other comments

This project is being derived from the Noctua portion of a larger AWS-ification project:
https://github.com/orgs/geneontology/projects/146
#85

[kltm]

Noting for @mugitty and @pgaudet that I'm in the process of spinning this into its own project. @mugitty and myself can meet to add more granular tickets. We can do a little of this tomorrow at the project meeting.