Finished Tap Refactor, upgraded all streams to v2 #77
Conversation
Pull changes from Stitch See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!32
release/37 changes, squashed See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!37
…e/38' [ECDDC-575] Refactor Clients stream See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!40
…ream' # Conflicts: # tap_mambu/sync.py # tap_mambu/tap_mambu_refactor/main.py
…e/38' [ECDDC-555] Refactor centres stream See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!39
…ream' into 'release/38' [ECDDC-604] Added field "assigned_branch_key" to clients stream See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!42
… into 'release/38' [ECDDC-607] Changed readme file to correct tap_config.json into config.json See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!44
…elease/38' [ECDDC-601] Add code scanning with sonar See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!43
…_move-unit-tests_release-38
…/41' [ECDDC-656] Refactor tap core code See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!94
[ECDDC-654] Added the Tap Tester into the CI pipeline See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!86
…/41' [ECDDC-656] Renamed unittests folder and fixed an import bug See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!95
# Conflicts: # tap_mambu/tap_mambu_refactor/tap_processors/processor.py
# Conflicts: # tap_mambu/helpers/schemas/loan_transactions.json # tap_mambu/tap_processors/processor.py
Merge release/40 into release/41 See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!100
|
Hi @DownstreamDataTeam, thanks for your contribution! In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. |
…e/41' [ECDDC-685] Adjusted v1 stream schemas See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!103
# Conflicts: # tap_mambu/tap_processors/audit_trail_processor.py # tap_mambu/tap_processors/processor.py
Merge master to release 41 See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!105
| last_stream = None | ||
| if stream_name in selected_streams: | ||
| return True, last_stream | ||
| return False, last_stream |
There was a problem hiding this comment.
The get_selected_streams in the Catalog module will do a similar task to this, except it doesn't skip preceding streams if they're selected. Instead, it will shuffle the list of selected streams based on the currently syncing stream before returning the ordered set. https://github.com/singer-io/singer-python/blob/master/singer/catalog.py#L151
We've found that in practice, it's better to attempt all streams when resuming rather than wait for the next scheduled extraction, since sometimes folks have their frequency set at many hours or a whole day.
There was a problem hiding this comment.
We'll consider replacing this code with the included function
There was a problem hiding this comment.
In my inspection of the JSON schema files today, I noticed one small thing that would be very hard to correct after release. It seems that the fields booking_date and creation_date are no longer marked with a "format": "date-time" Is this something you'd want to fix before I ship it? It might also be good to take some time to review for other date-times that need the format.
@DownstreamDataTeam Sorry, that is for the |
…ime' into 'release/41' [ECDDC-719] Added format date-time to interest accrual breakdown See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!110
|
You did it @DownstreamDataTeam! Thank you for signing the Singer Contribution License Agreement. |
Description of change
Manual QA steps
Risks
Rollback steps