Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hotfix: Changed deposit_accounts sorting field #91

Merged
merged 343 commits into from
Sep 12, 2022
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
343 commits
Select commit Hold shift + click to select a range
0411ab9
Added a new stream: interest_accrual_breakdown
Mar 23, 2022
6fab293
Removed unused import temporarily
Mar 23, 2022
e4ca63f
Added the tests for the new stream
Mar 23, 2022
2b0250e
Added task_link_key field to tasks stream
Mar 23, 2022
fdcf8f4
New Sonar config version
Mar 23, 2022
55cc5f2
Merge branch 'feature/ECDDC-653_implement-tests-for-fields' into 'rel…
Mar 23, 2022
73f490e
Merge branch 'release/40' into feature/ECDDC-603_add_interest_accrual…
Mar 23, 2022
8fa5d81
Merge branch 'feature/ECDDC-646_Integrate-sonar-into-pipeline-fix' in…
Mar 23, 2022
1835243
Solved a merge conflict issue
Mar 23, 2022
653af8b
Finished implementing catalog automatic fields checker test
Mar 23, 2022
2566966
Fixed the tests after merging the release branch
Mar 23, 2022
de90318
Fixed issue with users deduplication key
Mar 23, 2022
144b06b
Merge branch 'feature/ECDDC-591_refactor_users_stream' into 'release/40'
Mar 24, 2022
9e5bc08
Merge branch 'release/40' into feature/ECDDC-653_implement-tests-for-…
Mar 24, 2022
38d3e82
Merge branch 'feature/ECDDC-603_add_interest_accrual_breakdown_stream…
Mar 24, 2022
5c127d1
Merge branch 'feature/ECDDC-649_add_missing_tasks_fields' into 'relea…
Mar 24, 2022
2760547
Merge branch 'feature/ECDDC-652_adjust_snyk_dev_test' into 'release/40'
Mar 24, 2022
4e19f66
Merge branch 'feature/ECDDC-653_implement-tests-for-fields' into 'rel…
Mar 25, 2022
0d99a79
Added tap tester job to ci pipeline
Mar 25, 2022
3992aa7
Refactored activities stream
Mar 28, 2022
7962223
Added unit tests for activities generator
Mar 28, 2022
0033c64
Updated tap tester job
Mar 25, 2022
4e540c5
squash! Updated tap tester job
Mar 28, 2022
a14e736
Updated tap tester job
Mar 28, 2022
b8d8d86
Updated tap tester job
Mar 28, 2022
fa1f6bb
Updated tap tester job
Mar 28, 2022
fdaf693
Updated tap tester job
Mar 28, 2022
ecaac46
Finished audit trail refactor, needs polishing and unit tests
Mar 28, 2022
b69651d
Finished audit trail refactor, needs polishing and unit tests
Mar 28, 2022
8a009f1
Merge branch 'feature/ECDDC-657_Refactor-audit-trail-stream' of gitla…
Mar 28, 2022
7fd5664
Updated tap tester job
Mar 28, 2022
e0c517a
Updated tap tester job
Mar 28, 2022
fa6e874
Updated tap tester job
Mar 28, 2022
84009ce
Updated tap tester job
Mar 28, 2022
9be401f
Finished refactoring of Audit Trail stream
Mar 28, 2022
0e51892
Implemented unit tests for audit trail stream
Mar 29, 2022
352a0d8
Update test_processors.py
Mar 29, 2022
6c6eeb6
Added missing fields to loan_transaction's 'affected_amounts' object
Mar 29, 2022
3ce36aa
Added all the other missing fields
Mar 29, 2022
6914b9c
Fixed field name
Mar 29, 2022
9f68728
Merge branch 'feature/ECDDC-667_Add-affected-amount-missing-fields' i…
Mar 29, 2022
7bb9829
Merge branch 'feature/ECDDC-658_Refactor-activities-stream' into 'rel…
Mar 29, 2022
ef0c811
Merge branch 'release/40' into 'feature/ECDDC-657_Refactor-audit-trai…
Mar 29, 2022
355479b
Updated tap tester job
Mar 29, 2022
68a8645
Updated tap tester job
Mar 29, 2022
8e6affb
Merge branch 'feature/ECDDC-657_Refactor-audit-trail-stream' into 're…
Mar 30, 2022
92fbea6
Merge branch 'master' into 'release/40'
Mar 30, 2022
1c85c0c
Refactored endpoint_deduplication in Base Processor, moving it to a s…
Mar 31, 2022
6ad9210
Finished refactoring of loan_products stream, waiting on schema
Mar 31, 2022
e197d1e
Added old field back, so that the change is backwards compatible
Apr 1, 2022
96e9481
Finished refactoring of deposit_products stream, waiting on schema
Apr 1, 2022
f6beb79
Merge branch 'hotfix/ECDDC-674_add-tax-on-interest-from-arrears-amoun…
Apr 1, 2022
b96009f
Removed sorting from deposit products as it's not present in the v2
Apr 1, 2022
354ea21
Added loan_products stream to sync.py
Apr 1, 2022
5fb1614
Added deposit products to sync.py
Apr 1, 2022
66cbf87
Finished refactor of custom_field_sets, waiting for schema script
Apr 1, 2022
6096429
Added some custom exceptions to the Tap, and throwing/catching them i…
Apr 4, 2022
e8e8ffb
Merge branch 'feature/ECDDC-672_Rethink-choose-next-record' into 'rel…
Apr 5, 2022
de3c109
Added work so far, need to fetch changes from release/41
Apr 5, 2022
89e3b16
Merge branch 'release/41' into feature/ECDDC-543_Refactor-gl-accounts…
Apr 5, 2022
de4cd27
Added the script that generates streams schema
Apr 5, 2022
4e87c4a
Final changes to gl_accounts stream
Apr 6, 2022
f87eefb
Fixed some edge cases
Apr 6, 2022
bb24a4e
Added custom exceptions and missing stream
Apr 6, 2022
af3af34
Updated loan_products schema
Apr 6, 2022
c5f63c2
Updated deposit_products schema
Apr 6, 2022
b5798a5
Removed problematic child stream
Apr 6, 2022
05e3cdf
Updated custom_field_sets schema
Apr 7, 2022
aeff576
Updated gl_accounts schema
Apr 7, 2022
195a996
Ordered alphabetically the fields and added custom_fields field to sp…
Apr 7, 2022
a0e68d8
Reorganised the script into multiple files
Apr 7, 2022
9d7ed9f
Refactored gl_journal_entries, including unit tests and schema update
Apr 7, 2022
c917c44
Merge branch 'release/41' into feature/ECDDC-588_Refactor-loan-produc…
Apr 7, 2022
6bbd8af
Merge branch 'release/41' into feature/ECDDC-583_refactor-deposit-pro…
Apr 7, 2022
f41b618
Merge branch 'release/41' into feature/ECDDC-582_refactor-custom-fiel…
Apr 7, 2022
ea05702
Fixed unit test
Apr 7, 2022
140a6f1
Fixed unit tests
Apr 7, 2022
47a5c4d
Finished refactor for release/41 streams, need to merge with other br…
Apr 8, 2022
5a45c39
Merge branch 'feature/ECDDC-588_Refactor-loan-products-stream' into f…
Apr 8, 2022
d6ef0d6
Merge branch 'feature/ECDDC-583_refactor-deposit-products-stream' int…
Apr 8, 2022
535d87c
Merge branch 'feature/ECDDC-582_refactor-custom-field-sets-stream' in…
Apr 8, 2022
f2f90bb
Merge branch 'feature/ECDDC-543_Refactor-gl-accounts-stream' into fea…
Apr 8, 2022
01d5d82
Merge branch 'feature/ECDDC-430_Refactor-gl-journal-entries-stream' i…
Apr 8, 2022
9ccc181
Refactored imports
Apr 8, 2022
28a18d7
Merge remote-tracking branch 'origin/feature/ECDDC-673_schema_fetcher…
Apr 8, 2022
5816240
Fixed sync isinstance (replaced with issubclass)
Apr 8, 2022
970262f
Fixed issue with generator values
Apr 11, 2022
d6483f3
moved custom_fields field to the properties
Apr 11, 2022
a76de55
Merge remote-tracking branch 'origin/feature/ECDDC-673_schema_fetcher…
Apr 11, 2022
b6737f3
pascal to snake convertor - added small hack for ID word
Apr 11, 2022
9b924a9
Merge remote-tracking branch 'origin/feature/ECDDC-673_schema_fetcher…
Apr 11, 2022
2f40b34
added custom_fields field to all streams
Apr 12, 2022
2d829ec
Merge remote-tracking branch 'origin/feature/ECDDC-673_schema_fetcher…
Apr 12, 2022
3a67298
updated the schemas
Apr 12, 2022
6ecc4db
Merge branch 'hotfix/ECDDC-678_Fix-processor-choose-next-record' into…
Apr 13, 2022
113abfd
Merge branch 'feature/ECDDC-430_Refactor-gl-journal-entries-stream' i…
Apr 13, 2022
46a7369
Merge branch 'feature/ECDDC-582_refactor-custom-field-sets-stream' in…
Apr 13, 2022
fbb2b0b
Merge branch 'feature/ECDDC-588_Refactor-loan-products-stream' into '…
Apr 13, 2022
6fc823c
Merge branch 'release/41' into feature/ECDDC-543_Refactor-gl-accounts…
Apr 13, 2022
0532972
Merge branch 'release/41' into feature/ECDDC-583_refactor-deposit-pro…
Apr 13, 2022
510e3b5
Merge branch 'feature/ECDDC-673_schema_fetcher' into 'release/41'
Apr 13, 2022
24bcaf1
Merge branch 'feature/ECDDC-543_Refactor-gl-accounts-stream' into 're…
Apr 13, 2022
47643a7
Merge branch 'release/41' into feature/ECDDC-583_refactor-deposit-pro…
Apr 13, 2022
6fa7496
Merge branch 'feature/ECDDC-583_refactor-deposit-products-stream' int…
Apr 13, 2022
65a3bb8
Merge branch 'release/41' into feature/ECDDC-656_Refactor-Tap-core-code
Apr 13, 2022
93291c2
Merge branch 'release/41' into 'feature/ECDDC-677_schema_checker'
Apr 13, 2022
7d4124f
fixed merge conflicts issue
Apr 13, 2022
14088c5
Merge branch 'feature/ECDDC-677_schema_checker' into 'release/41'
Apr 13, 2022
59f9d3f
Merge branch 'feature/ECDDC-656_Refactor-Tap-core-code' into 'release…
Apr 13, 2022
e7ebdab
Merge branch 'feature/ECDDC-654_tap_tester_to_ci' into 'release/41'
Apr 13, 2022
3d65a3c
adjusted an import to match the new structure
Apr 14, 2022
821070e
renamed mambu unittests folder
Apr 14, 2022
457321b
updated ci file to use the new tests dir name
Apr 14, 2022
132f7a3
added fixture jsons
Apr 14, 2022
2526d90
First draft of the PerformanceMetrics module
Apr 14, 2022
d12a9b3
Finished Performance Metrics baseline implementation
Apr 14, 2022
8ae91d0
Added records/s (per extraction, not per request) and total extractio…
Apr 14, 2022
0e8f5d3
Merge branch 'feature/ECDDC-656_Refactor-Tap-core-code' into 'release…
Apr 15, 2022
b634430
Improved performance metrics (more for the Multithreaded version than…
Apr 18, 2022
15c6d6f
Implemented multithreaded Tap PoC
Apr 18, 2022
e769d59
Merge branch 'master' into master-into-release-40
Apr 18, 2022
48e5017
Merge branch 'master-into-release-40' into release-40-into-release-41
Apr 18, 2022
de1f8ca
Merge branch 'release-40-into-release-41' into 'release/41'
Apr 19, 2022
7e8edf2
FInal implementation of MultithreadedGenerator PoC
Apr 20, 2022
0888dbd
Added matplotlib to setup.py inside extras (under the name "mambu-per…
Apr 20, 2022
963b031
Merge branch 'feature/ECDDC-655_Performance-metrics' into feature/ECD…
Apr 20, 2022
425dc64
Fixed error if no generator or processor metrics are collected
Apr 20, 2022
a479564
Merge branch 'feature/ECDDC-655_Performance-metrics' into feature/ECD…
Apr 20, 2022
8c20863
Added new fields to loan_repayments schema
Apr 20, 2022
1383366
Updated loan_repayments schema
Apr 20, 2022
064ea16
Added new fields to activities schema + sorted the fields
Apr 20, 2022
3d0be1d
Separated metrics plotting from metrics collecting
Apr 21, 2022
ee4f344
Merge branch 'feature/ECDDC-655_Performance-metrics' into feature/ECD…
Apr 21, 2022
386bf94
Merge branch 'feature/ECDDC-655_Performance-metrics' into 'release/42'
Apr 26, 2022
aa1547a
Updated the schemas and fixed typos
Apr 26, 2022
9e6becb
Added format attribute to date fields
Apr 26, 2022
94cd72a
Commented changes to client, as they are not needed until we use the …
Apr 26, 2022
8c196a0
Merge branch 'feature/ECDDC-685_check_v1_stream_schemas' into 'releas…
Apr 26, 2022
6e3c525
Merge branch 'feature/ECDDC-682_Multithreading-poc-implementation' in…
Apr 27, 2022
0155856
Merge branch 'master' into merge_master_to_release_41
Apr 27, 2022
0e5a085
Merge branch 'merge_master_to_release_41' into 'release/41'
Apr 27, 2022
4475ab8
added default and prefix to the user agent
Apr 28, 2022
115026f
Proposed changes as part of refactoring and performance improvements
May 2, 2022
c355ede
Removed bookmark helper as it may not be the best approach.
May 2, 2022
6bd7f5c
More changes in order to get Multithreading to a functional point
May 2, 2022
0f383dc
Small changes, implemented overlap_window correctly
May 3, 2022
adf929d
Added multithreaded refactor to clients stream, removed from activities
May 3, 2022
05cd8bb
Fixed transform_json not being used
May 3, 2022
158b706
Fixed error correction algorithm
May 3, 2022
51a1275
Added tests for sync_all_streams function
May 3, 2022
65afa95
Added tests for MambuClient
May 3, 2022
322defc
Added more tests
May 4, 2022
8178902
Merged 2 tests and covered extra cases on other tests
May 5, 2022
8515dac
Added tests for raise_for_error function
May 5, 2022
ff6d767
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
May 6, 2022
1adf86e
Improved extraction of child streams by extracting children for all r…
May 6, 2022
68169ef
Added offset by bookmark functionality
May 9, 2022
2c27a68
Overwrite the prepare_batch_params
May 9, 2022
1d257f7
Changed multithreading back to activities
May 9, 2022
2c1bccc
Merge commit '2c27a68beffc29f8a31eaa532d96b764d455e0b4' into feature/…
May 9, 2022
5b8ec73
Working Prototype
May 9, 2022
0692675
Refactored datetime_utils, included pytz for timezone information
May 10, 2022
462dec7
Merge branch 'feature/ECDDC-698_unit_tests_improvement' into 'release…
May 11, 2022
9d24a46
Merge branch 'release/43' into feature/ECDDC-697_user_agent_prefix
May 11, 2022
d2e86af
Added the new user agent to the tests
May 11, 2022
6b4151a
Merge branch 'feature/ECDDC-697_user_agent_prefix' into 'release/43'
May 11, 2022
6bde3e9
Changed sorting criteria to use 'id' instead of 'lastModifiedDate'
May 12, 2022
8bfe36b
Updated deposit_accounts unit test too
May 12, 2022
9fba90f
Changed sorting by id for deposit_transactions
May 12, 2022
9817037
Added format date-time to interest accrual breakdown
May 13, 2022
1c39ddd
Merge branch 'hotfix/ECDDC-719_interest-accrual-missing-format-date-t…
May 13, 2022
c43821c
Fixed import errors and unit tests (only the timezone problem, still …
May 13, 2022
e2b4cba
Fixed wrong transform operation order on raw batches of records
May 13, 2022
919997b
Fixed iter() call on generator, replaced with .__iter__() direct call…
May 13, 2022
c512fda
Fixed unit tests
May 13, 2022
a7c2b11
Fixed typo in generator, implemented multithreading for clients, comm…
May 17, 2022
9312761
Added multithreading to gl_journal_entries stream
May 17, 2022
2d82b84
Merge branch 'master' into release-42-merge
May 18, 2022
fbfe86e
Merge branch 'release-42-merge' into 'release/42'
May 18, 2022
e758ac7
Added MultithreadedOffsetGenerator
May 19, 2022
460a737
Added multithreading to installments
May 19, 2022
ec3a97c
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
May 19, 2022
85848e6
Finished all multithreading refactors planned for my side
May 19, 2022
ec0a119
Bugfix for multithreaded offset generator
May 24, 2022
7ecf18b
Added multithreading to users
May 24, 2022
bf2db16
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
May 24, 2022
7af4d18
Removed extra code
May 24, 2022
e69469c
Removed unused import
May 24, 2022
d232d10
Fixed the unittests
May 24, 2022
e363b0e
Changed sorting for interest accrual breakdown
May 25, 2022
67a71cf
Merge branch 'feature/ECDDC-707_multithreading_to_other_streams' into…
May 25, 2022
49a3984
Merge commit 'd232d101149ccf34c737d8c5ac43616b6a0563d1' into feature/…
May 26, 2022
010d196
Fixed all multithreading issues, need to address batch limits
May 30, 2022
9b02db1
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
May 31, 2022
7c96b4a
Added unit tests for multithreading requests and bookmark generator
Jun 7, 2022
cedb8f4
Added unit test for offset generator
Jun 7, 2022
771a1d5
Renamed a file
Jun 7, 2022
1130fcb
Small test code refactoring
Jun 7, 2022
30e0a6c
Improved a test
Jun 7, 2022
2d40177
Final version of the multithreading, needs some adjusting after testing
Jun 9, 2022
f193d66
Added more tests
Jun 9, 2022
fa79283
Made all dates translate to full date+time, and then truncate them, s…
Jun 10, 2022
f60fcd6
Fixed all date issues and unit tests
Jun 10, 2022
3e76e06
Also fixed deposit transactions generator (was set not to use multith…
Jun 10, 2022
7213d9b
Added logging back to requests, removed leftover file from another task
Jun 10, 2022
d0df2e1
Final self-review
Jun 10, 2022
e0b9970
Added pytz back
Jun 10, 2022
ec5faea
Replaced the 2 logging messages for a request with a single message
Jun 15, 2022
96a3def
Fixed groups unit test, and made interest accrual breakdown use day-b…
Jun 15, 2022
9bd16c2
Merge branch 'release/45' into feature/ECDDC-707_Performance-improvem…
Jun 15, 2022
ed8523f
Fixed unit tests (mocking multithreading shutdown)
Jun 15, 2022
0c758d2
Removed the todo comment
Jun 15, 2022
d4bbdcc
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
Jun 15, 2022
e2e0e37
Fixed merge conflicts
Jun 16, 2022
60242e4
Merge branch 'release/41' into merge-41-into-42
Jun 20, 2022
4110fd8
Merge branch 'merge-41-into-42' into 'release/42'
Jun 20, 2022
d14076e
Merge branch 'feature/ECDDC-707_Performance-improvements-and-refactor…
Jun 20, 2022
55bcd34
Merge branch 'release/45' into feature/ECDDC-695_Implement-multithrea…
Jun 21, 2022
6e0c43e
Merge branch 'feature/ECDDC-695_Implement-multithreaded-child-streams…
Jun 22, 2022
c2550fc
Merge branch 'release/42' into merge-master-into-45
Jun 22, 2022
bdcf451
Merge branch 'master' into merge-master-into-45
Jun 22, 2022
8a4d930
Merge branch 'merge-master-into-45' into 'release/45'
Jun 23, 2022
ac2e360
Merge branch 'feature/ECDDC-727_multithreading_unittests' into 'relea…
Jun 23, 2022
f09d70b
Refactored offset and bookmark multithreaded generators
Jun 27, 2022
1c53b7e
More refactoring. Deleted multithreading POC
Jun 29, 2022
143b2d2
Modified batch_limit value and client default page size
Jun 29, 2022
0423c5f
Adjusted the unit tests using the new refactored multithreaded genera…
Jun 29, 2022
5d96b64
Finished multithreaded bookmark generator unit tests
Jun 30, 2022
d2cc9cd
Finished multithreaded offset generator unit tests
Jun 30, 2022
d4f752a
Moved max_threads to init_config and kept the batch_limit the same fo…
Jul 4, 2022
643274b
Small adjustments to the multithreading unit tests
Jul 4, 2022
b661649
Adjusted the last_batch_set value from the test_preprocess_batches_flow
Jul 4, 2022
06a5f97
Moved DEFAULT_PAGE_SIZE const into the constants file
Jul 4, 2022
2c1f709
Merge branch 'feature/ECDDC-729_refactor_multithreading_generators' i…
Jul 6, 2022
8e75959
Merge branch 'hotfix/ECDDC-716_deposit-accounts-missing-records' into…
Jul 7, 2022
a18ae11
Corrected some issues and messages
Aug 4, 2022
b641c35
Fixed another comment
Aug 4, 2022
984929c
Changed json.dumps implementation for converting dict lists into sets…
Aug 5, 2022
baa1275
Updated unit tests to reflect changes to code
Aug 8, 2022
c4d6bd1
Updated singer-python to 5.12.2
Aug 8, 2022
2ad5cbe
Merge branch 'hotfix/ECDDC-726_fix-pull-request' into 'release/45'
Aug 10, 2022
7ca2af9
Replaced all json.dumps with tuple conversions, as they are faster to…
Aug 11, 2022
d2644bd
Merge branch 'hotfix/ECDDC-726_fix-pull-request' into 'release/45'
Aug 11, 2022
d23da21
Added python_requires to setup.py
Aug 19, 2022
7ea4a0b
Merge branch 'master' into hotfix/ECDDC-749_add-python-requires-to-se…
Aug 19, 2022
3a55775
Merge branch 'hotfix/ECDDC-749_add-python-requires-to-setup-py' into …
Aug 19, 2022
2278b31
changed sorting from id to lastModifiedDate
Sep 1, 2022
96d229d
fixed an unit test
Sep 1, 2022
49450c3
Merge branch 'hotfix/ECDDC-752_deposit_accounts_sorting' into 'releas…
Sep 1, 2022
87e0abc
Merge branch 'master' into release/45
DownstreamDataTeam Sep 12, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ def test_deposit_accounts_generator():
assert generator.endpoint_path == "deposits:search"
assert generator.endpoint_bookmark_field == "lastModifiedDate"
assert generator.endpoint_sorting_criteria == {
"field": "id",
"field": "lastModifiedDate",
"order": "ASC"
}
assert generator.endpoint_filter_criteria == [
Expand Down
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
author='[email protected]',
classifiers=['Programming Language :: Python :: 3 :: Only'],
py_modules=['tap_mambu'],
python_requires='>=3.9',
install_requires=[
'backoff==1.8.0',
'requests==2.23.0',
Expand Down
2 changes: 1 addition & 1 deletion tap_mambu/tap_generators/deposit_accounts_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ def _init_endpoint_config(self):
super(DepositAccountsGenerator, self)._init_endpoint_config()
self.endpoint_path = "deposits:search"
self.endpoint_sorting_criteria = {
"field": "id",
"field": "lastModifiedDate",
"order": "ASC"
}
self.endpoint_filter_criteria = [
Expand Down