Provide message return value data

[birdychang]

It was reported that there's no message return value in our datasets.

Although there's an effort previously to extract raw return value data, the task was never executed and the design is not user friendly.

After some internal discussions, the team decided to add both raw and parsed return value to the receipts table.

Tasks:

• update Lily code to include new columns (feat: add new columns for receipts table lily#1248)
• cut a new release and deploy to production (Update CHANGELOG.md lily#1250, v0.17.0)
• verify new timescaleDB data
• make schema changes in BigQuery
• test with one snapshot and verify backfilled BigQuery data
• backfill receipts table in BigQuery since 3/14
• backfill receipts table in BigQuery from 2023/01/01 to 2023/03/14
• backfill receipts table in BigQuery from 2022/01/01 to 2022/12/31

[charliealgernon]

Following up from this Slack thread. Are we still on track to complete the backfill in a week or so (given that it was scoped for 2 weeks a week ago)?

[Terryhung]

hi @charliealgernon,
Since 3/14, we've backfilled the receipt return and return_parsed data.

[charliealgernon]

What about before 3/14?

[Terryhung]

@charliealgernon This week, we will be backfilling data from January 1, 2023, to March 14, 2023. Subsequently, we plan to backfill the data for the year 2022 in the upcoming weeks.

[charliealgernon]

@Terryhung Thanks for the update. Good to hear that by EOW we'll have all of 2023 (ytd). Do you have an ETA for 2022?

[Terryhung]

@charliealgernon, After our assessment, it can be completed by the weekend after next.
FYI, The data from 2023 is available for use now.

[Terryhung]

@charliealgernon FYI, the data from 2022 is available for use now!