Google Floods Part 1: Creating gauge points and tooltip graphs; COUNTRY=cambodia #1328

lowriech · 2024-08-08T22:11:17Z

Description

This fulfills step 1 and 2 of 3 for #1317 (see plan).

In this PR, we're fetching flood status by gauge from Google's Floods API for the entire country (10-100 gauges per country). We're displaying these gauges as points on Prism. They're color coded by flood status and provide past and future flooding in a tooltip when clicked.

Forecasts for each gauge were added here: #1330 (merged into this branch for review)

How to test the feature:

Open the Mozambique or cambodia deployment
Enable the Google Flood layer
Click on a gauge point

Checklist - did you ...

Test your changes with

REACT_APP_COUNTRY=rbd yarn start
REACT_APP_COUNTRY=cambodia yarn start
REACT_APP_COUNTRY=mozambique yarn start
Add / update necessary tests?
Add / update outdated documentation?

Screenshot/video of feature:

382087325-f448a602-a406-4f64-b6c5-861a8bc25c49

github-actions · 2024-08-08T22:18:28Z

Build succeeded and deployed at https://prism-1328.surge.sh
(hash aac80e7 deployed at 2025-01-08T22:11:02)

ericboucher · 2024-08-09T20:30:39Z

api/app/main.py

@@ -412,3 +413,10 @@ def post_raster_geotiff(raster_geotiff: RasterGeotiffModel):
    return JSONResponse(
        content={"download_url": presigned_download_url}, status_code=200
    )
+
+
+@app.get("/google-floods-gauges")


If we're gonna add a bunch of integrations, it will be worthwile to think about a standardized approach and common architecture. And maybe expose them through a sub endpoint of the API to keep things cleaner. wdyt @lowriech ?

I think that's a good question, @ericboucher. Do you see other APIs we're already leveraging fitting into this integration category (kobo, hdc stats, others)? Or are you thinking just for new integrations?

I'm wary of premature abstraction and trying to determine if I think now is the right time to align on this standardized approach.

It seems like low-hanging fruit to abstract once we decide on a pattern. So yes, agree we should think about it, but not sure about the velocity of adding entirely new providers. Would vote to wait until we see more pattern around this.

frontend/src/config/mozambique/layers.json

gislawill · 2024-08-12T22:10:28Z

@lowriech and @ericboucher, there are a couple outstanding issues (noted in the PR description) but this ready for initial review

gislawill · 2024-08-12T22:20:23Z

frontend/src/components/MapView/utils.ts

+      if (
+        itemProps.visibility === FeatureInfoVisibility.IfDefined &&
+        !properties[item]
+      ) {
+        return obj;
+      }


We have some feature properties that may or may not be defined by the google api (for example: site name, river name). This allows us to hide the property from the pop up if it's not defined

gislawill · 2024-08-12T22:22:53Z

api/app/googleflood.py

+            "river": (
+                data["river"] if "river" in data and len(data["river"]) > 1 else None
+            ),


"river" can come in as a string with just 1 space (" "). This ensures there's a real value

frontend/src/config/mozambique/layers.json

gislawill · 2024-08-16T15:39:28Z

Note: I'm adding pytest-recording to store our google flood responses in local tests and use them in CI/CD. That will fix the failing API test in this PR

…api key to docker compose

…AM/prism-app into feature/google-floods/1317

gislawill · 2024-11-18T16:10:06Z

Thanks @wadhwamatic, is this good to go as far as you're concerned?

Some thoughts on potential remaining steps:

How does the copy look?
Is the layer set up optimal in each deployment (should this be in it's own layer group as currently implemented or put in an existing layer group)
We'll need translations once the copy and layer organization is complete

I'll note for context that the 502 errors from Google in RBD is not an issue on this branch. It seems to only crop up in #1332 (where I've added a cache-based workaround)

wadhwamatic · 2024-12-17T23:01:16Z

@ericboucher - can you make a final pass on this PR? It's gtg from my side. Let's please remove the google flood layers from prism.json in the demo countries before merging into main though but bring it back subsequently in #1332

ericboucher · 2024-12-19T10:03:03Z

api/app/googleflood.py

+            ],
+        },
+        "properties": {
+            "gaugeId": data["gaugeId"],


How confident are we that all the keys will be present? Should ze use a getter with a default None to be safe?
Or make sure that we have proper error handling for missing keys.

We're pretty confident that they'll be present but there's no downside to the default none to be safe. I've updated to add a getter for non-essential values and error handling for missing essential values

ericboucher · 2024-12-19T10:06:27Z

api/app/utils.py

@@ -38,3 +40,29 @@ def filter(self, record):
            self.warning_count += 1

        return False
+
+
+def make_request_with_retries(


Do we need something specific, or could we rely on the Retry util directly?
from requests.adapters import Retry

The retry utility has some useful functionality that this approach didn't have (back-off timing and status filtering for retries specifically). I've updated this function to wrap the retries utility. This make_request_with_retries function is still useful to reduce redundancy

ericboucher · 2024-12-19T10:11:20Z

frontend/src/config/rbd/layers.json

@@ -6531,5 +6531,77 @@
      }
    ],
    "legend_text": "Joint Food Security & Nutrition Hotspot Analysis conducted by WFP, UNICEF, and partners based on SMART surveys and Cadre Harmonisé (CH) & Integrated Food Security Phase Classification (IPC) analyses"
+  },
+  "google_flood_status_at_gauges": {


Should this layer be shared ans then just override the necessary param?

I haven't implemented a shared layer before and I'm still trying to understand how it works. I see in the README:

Since many layers are common across multiple countries, we created shared configuration files that any deployment can access. These layers are generated by WFP globally and made available through the Humanitarian Data Cube.

Is the HDC part meaningful? Based on a quick scan, the current shared layers do appear to all be coming from HDC but I don't see why they need to (technologically speaking that is).

Also, @ericboucher, could you point me to examples of shared layers with overrides? I see examples of shared layers with duplicates in countries but I'm not seeing overrides.

@ericboucher, adding this layer to our shared layer config caused it to load available dates for every deployment — regardless of whether or not the deployment used this layer. I think it's better to only add this layer to deployments that are using it until we're more selective about which layers we're loading data for

ericboucher · 2024-12-19T10:12:39Z

frontend/src/utils/fetch-with-timeout.ts

@@ -8,7 +8,7 @@ interface FetchWithTimeoutOptions extends RequestInit {

 export const ANALYSIS_REQUEST_TIMEOUT = 60000;

-const DEFAULT_REQUEST_TIMEOUT = 15000;
+const DEFAULT_REQUEST_TIMEOUT = 30000;


Still needed? Is it to account for the retries?

It's still needed. Not necessarily for retries but for loading all of the data needed for RBD. The server needs to make a lot of separate requests to get all of the various country data and, even with parallelization, it can take about 20 seconds.

@gislawill - I'm still seeing "Impossible to get point data dates for forecast_river_discharge_at_gauges" error on the surge deployment and locally. Any thoughts on this?

@wadhwamatic there are two issues right now:

The current deployment of our production PRISM server no longer has the functionality from this branch

This is resulting in 404 errors because the route to get the flood dates isn't found on the server

@ericboucher, any objection to me running a deployment from this branch to production? It will have the latest on main merged in

~~The google floods API seems to be down again. I'm seeing 502 "Bad gateway" errors when hitting these endpoints locally. I'll reach out to Aviel about it.~~

This is the known issue with RBD that is fixed by the caching in Google Floods Part 3: Adding Inundation Maps; COUNTRY=mozambique #1332

I'll remove the changes to RBD from this PR accordingly

@wadhwamatic redeploying the server fixed it

ericboucher · 2024-12-19T10:16:45Z

frontend/src/components/Common/Chart/index.tsx

@@ -208,8 +223,21 @@ const Chart = memo(
          fill: false,
        }));
      }
+      if (data.GoogleFloodConfig) {


This PR is introducing a lot of extra complexity to this file. I'll open an issue recommending to split it by functions/services

ericboucher

Nice work overall. Let's make sure to QA all different tooltips inc. EWS.
I left a few comments in line but this looks gtm overall !

…st_with_retries

gislawill · 2025-01-06T00:30:24Z

Nice work overall. Let's make sure to QA all different tooltips inc. EWS. I left a few comments in line but this looks gtm overall !

Confirmed EWS tooltips look good

ericboucher · 2025-01-15T15:57:55Z

@gislawill are we good to merge?

gislawill · 2025-01-15T16:02:51Z

I believe we are good to go. I'd love to have @wadhwamatic's approval before merging.

wadhwamatic · 2025-01-15T18:57:51Z

sorry @gislawill, missed the fix for the png download. yes, gtg now. thanks!

Adding Google Floods Backend

c5c1215

lowriech marked this pull request as draft August 8, 2024 22:11

ericboucher reviewed Aug 9, 2024

View reviewed changes

gislawill added 2 commits August 11, 2024 22:26

Viewing the gauges on the frontend, adding error handling to backend

b9722d3

Update comment

bc92d38

gislawill reviewed Aug 12, 2024

View reviewed changes

frontend/src/config/mozambique/layers.json Outdated Show resolved Hide resolved

Updates to support tooltip content

08a4499

gislawill changed the title ~~Adding Google Floods Backend~~ Adding Google Floods Backend + Creating a Gauge Points Aug 12, 2024

gislawill added 3 commits August 12, 2024 14:05

copy update

0653f1e

Refactor title state management

7cd866d

Fixing and adding server tests

94e1588

gislawill changed the title ~~Adding Google Floods Backend + Creating a Gauge Points~~ Adding Google Floods Backend + Creating a Gauge Points; COUNTRY=mozambique Aug 12, 2024

gislawill reviewed Aug 12, 2024

View reviewed changes

ericboucher reviewed Aug 13, 2024

View reviewed changes

frontend/src/config/mozambique/layers.json Outdated Show resolved Hide resolved

ericboucher and others added 4 commits August 13, 2024 14:19

update set_env

6ac3aad

Merge branch 'master' into feature/google-floods/1317

eb29b6e

update snapshots

6be26d7

Merge branch 'master' into feature/google-floods/1317

2fe481a

gislawill linked an issue Aug 19, 2024 that may be closed by this pull request

[Feature Request]: Google Flood Hub integration #1317

Closed

gislawill added 3 commits August 20, 2024 17:29

Adding support for multi country deployments and adding google flood …

8e84cbb

…api key to docker compose

Use interpolation for popup titles

efdf2c9

Adding pytest-recording

8fa327c

gislawill marked this pull request as ready for review August 21, 2024 01:21

ericboucher added 3 commits August 21, 2024 10:38

Merge branch 'master' into feature/google-floods/1317

f5ffbf6

Update test_google_floods_api.py

438ee04

Merge branch 'feature/google-floods/1317' of https://github.com/WFP-V…

e369704

…AM/prism-app into feature/google-floods/1317

gislawill mentioned this pull request Nov 25, 2024

Google Floods Part 3: Adding Inundation Maps; COUNTRY=mozambique #1332

Open

wadhwamatic added 3 commits December 3, 2024 21:56

update text for river discharge layer

455a69e

Merge branch 'master' into feature/google-floods/1317

733efcf

Merge branch 'master' into feature/google-floods/1317

5b4e17d

wadhwamatic added 2 commits December 18, 2024 08:07

Merge branch 'master' into feature/google-floods/1317

8cf36fd

Merge branch 'master' into feature/google-floods/1317

51defcf

ericboucher reviewed Dec 19, 2024

View reviewed changes

ericboucher approved these changes Dec 19, 2024

View reviewed changes

ericboucher mentioned this pull request Dec 19, 2024

Cleanup / Split Chart/index file #1399

Open

gislawill added 3 commits December 30, 2024 11:38

Error handling for gauge response. Adding retry utility to make_reque…

9aa99ed

…st_with_retries

swap getattr with get

00cc9bd

reverting rbd updates

898fc9c

gislawill and others added 6 commits January 6, 2025 20:45

Update readme

9a92c3f

Removing configuration from mozambique

8ca7304

update snapshot

d004fbe

Fix for chart png download

ce929e4

fix lint

fbc942c

Merge branch 'master' into feature/google-floods/1317

aac80e7

wadhwamatic merged commit e85b377 into master Jan 15, 2025
6 checks passed

wadhwamatic deleted the feature/google-floods/1317 branch January 15, 2025 21:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google Floods Part 1: Creating gauge points and tooltip graphs; COUNTRY=cambodia #1328

Google Floods Part 1: Creating gauge points and tooltip graphs; COUNTRY=cambodia #1328

lowriech commented Aug 8, 2024 •

edited by gislawill

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

ericboucher Aug 9, 2024

gislawill Aug 12, 2024

lowriech Aug 13, 2024

gislawill commented Aug 12, 2024

gislawill Aug 12, 2024

gislawill Aug 12, 2024

gislawill commented Aug 16, 2024

gislawill commented Nov 18, 2024

wadhwamatic commented Dec 17, 2024

ericboucher Dec 19, 2024

gislawill Dec 30, 2024

ericboucher Dec 19, 2024 •

edited

Loading

gislawill Dec 30, 2024

ericboucher Dec 19, 2024

gislawill Dec 30, 2024

gislawill Jan 7, 2025

ericboucher Dec 19, 2024

gislawill Dec 30, 2024

wadhwamatic Jan 2, 2025

gislawill Jan 6, 2025 •

edited

Loading

gislawill Jan 7, 2025

ericboucher Dec 19, 2024

ericboucher left a comment

gislawill commented Jan 6, 2025

ericboucher commented Jan 15, 2025

gislawill commented Jan 15, 2025

wadhwamatic commented Jan 15, 2025

Google Floods Part 1: Creating gauge points and tooltip graphs; COUNTRY=cambodia #1328

Google Floods Part 1: Creating gauge points and tooltip graphs; COUNTRY=cambodia #1328

Conversation

lowriech commented Aug 8, 2024 • edited by gislawill Loading

Description

How to test the feature:

Checklist - did you ...

Screenshot/video of feature:

github-actions bot commented Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill commented Aug 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill commented Aug 16, 2024

gislawill commented Nov 18, 2024

wadhwamatic commented Dec 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericboucher Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericboucher left a comment

Choose a reason for hiding this comment

gislawill commented Jan 6, 2025

ericboucher commented Jan 15, 2025

gislawill commented Jan 15, 2025

wadhwamatic commented Jan 15, 2025

lowriech commented Aug 8, 2024 •

edited by gislawill

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

ericboucher Dec 19, 2024 •

edited

Loading

gislawill Jan 6, 2025 •

edited

Loading