RTX Code Release (PI-3): Enhancements to Replace and Route to support enterprise hydrofabric and new NWPS api capabilities in near-real time

[taddyb33]

The FIM-S team at RTX has been diligently working on enhancements to the existing Github solutions. This PR is the delivery of said enhancements, and documentation, for OWP approval. In short, the following enhancements were made:

• Usage of T-Route through an API (RTX Code Release (PI-3): Enhancements to T-Route so it can be used for OWP's Replace and Route t-route#837) rather than WRF-Hydro
• Using the api.water.noaa.gov to read RFC forecasts
• Using v20.1 of the enterprise hydrofabric rather than RouteLink.nc files
• Using an event-driven architecture and docker compose rather than AWS Lambda and EC2

Below we will go into more details of our changes:

Figure 1: A diagram of the new RTX RnR Architecture

Additions

• Changed the RnR architecture to be an event-driven API

  • Rather than being started through a python cmd line call, we are utilizing many docker containers managed by docker compose
  • The full pipeline is managed by a FastAPI app and can be triggered by running the /api/v1/publish/start endpoint from localhost:8000/docs or through a CURL command
  • This architecture uses RabbitMQ and Redis caching to ensure all forecasts are properly handled as floods, non-floods, errors, or forecasts that have been routed (no action).
  • A consumer python function handles individual RFC flooding events, formats output, calls T-Route, and post-processes output

• Added IaC to call RnR through docker compose and syncs outputs to an S3 bucket

  • see Source/RnR/terraform/README.md for the full docs

• Added T-Route as a service

  • Run the following command to pull the container: docker pull ghcr.io/taddyb33/t-route-dev:0.0.2
  • See RTX Code Release (PI-3): Enhancements to T-Route so it can be used for OWP's Replace and Route t-route#837 for more information and code

• Created a new Hfsubset tool for generating slices of v20.1 of the enterprise hydrofabric

  • Run the following command to pull the container: docker pull ghcr.io/taddyb33/hfsubset-legacy:0.0.4
  • See https://github.com/taddyb33/hfsubset-legacy for more information and code

• Added a mock database to locally test Replace and Route without the need for a production server

  • Dockerfiles and scripts have been included for generating a mock DB
  • The mock DB can also be pulled using: docker pull ghcr.io/taddyb33/hydrovis/mock_database:0.0.1

• Added jupyter notebooks to view historical RnR runs and verify correctness

  • See Source/RnR/notebooks/test_historical_LID.ipynb

• Created a front-end service to view netcdf files and plots for successful RnR runs on RFC points.

• Added a full docs/ suite to ensure code is documented

• Added docstrings to functions and type hints to ensure code is readable and can be followed

Removals

• See Changes

Changes

• the existing owp-viz-replace-route-tar.gzip file has been moved to Source/RnR/nwm_3.0/ to ensure no code is lost

Testing

PyTests were generated for the following tests in the Source/RnR/tests folder and can be run using pytest -s from the Source/RnR folder:

1. Testing the data consumer
2. Testing hfsubset capabilities
3. Testing how we are retrieving NWPS API data
4. Testing how can are publishing RFC forecasts
5. Testing the redis cache
6. Testing how we ping our mock database
7. Testing the full RnR service

These tests can be run either while docker compose is running, or if it is dead.

Screenshots

Figure 2: A geospatial plot of where flow is (green) and where flow is not (grey) between two RFC points on the v20.1 enterprise hydrofabric

Notes

• This code has been Demo'd to OWP and team on August 29th, 2024.
• A MAJORITY of the inserted code is from included Bootstrap 5 CSS files for our Front-end service

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

• Windows
• Linux
• Browser

Accessibility

• Keyboard friendly
• Screen reader friendly

Other

• Is useable without CSS
• Is useable without JS
• Flexible from small to large screens
• No linting errors or warnings
• JavaScript tests are passing

[taddyb33]

Edit: added note about front-end services and how the majority of code insertions are bootstrap css files