Hotfix: Changed deposit_accounts sorting field #91
Conversation
…ease/40' [ECDDC-653] Refactored main.py and unit tests See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!72
…_breakdown_stream # Conflicts: # tap_mambu/tap_mambu_refactor/main.py
…to 'release/40' [ECDDC-646] New Sonar config version See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!65
[ECDDC-591] Fixed issue with users deduplication key See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!78
…' into 'release/40' [ECDDC-603] Added interest accrual breakdown stream See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!74
…se/40' [ECDDC-649] Added task_link_key field to tasks stream See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!75
[ECDDC-652] Adjusted Snyk dev test See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!67
…ease/40' [ECDDC-653] Finished implementing catalog automatic fields checker test See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!77
…b.com:mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu into feature/ECDDC-657_Refactor-audit-trail-stream
…nto 'release/45' [ECDDC-729] Refactored offset and bookmark multithreaded generators See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!120
… 'release/45' [ECDDC-716] Deposit accounts missing records See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!123
… to one using a custom dict class "HashableDict" which implements a hash and unique keys for each dict (generated from the json dump data)
[ECDDC-726] Resolve issues in the PR See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!131
… compute a hash for
[ECDDC-726] Replaced all json.dumps with tuple conversions, as they are faster to compute a hash for See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!132
…'release/45' [ECDDC-749] Added python_requires to setup.py See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!136
…e/45' [ECDDC-752] Fixed deposit_accounts missing records See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!138
|
Hi @DownstreamDataTeam, thanks for your contribution! In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. |
There was a problem hiding this comment.
This looks good. There are also other generators that have the endpoint_sorting_criteria set to id. I see LoanTransactionsGenerator, InterestAccrualBreakdownGenerator , DepositTransactionsGenerator, GlJournalEntriesGenerator, and LoanAccountsGenerator. Is it possible they could have the same issues in the future since they also sort by id?
They shouldn't cause an issue, because the filter criteria is set on |
|
You did it @DownstreamDataTeam! Thank you for signing the Singer Contribution License Agreement. |
Description of change
Changed the sorting criteria from
idtolastModifiedDatebecause of how theMultithreadedBookmarkGeneratorworks: changes the filter at every n records.If the sort is on
id, there's a high chance that a record with thelastModifiedDateclose to present to be in that batch, thus the extraction continues from that date and misses older records that were not extracted.Manual QA steps
Risks
Rollback steps