Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timezone improvement #85

Merged
merged 355 commits into from
Sep 27, 2022
Merged

Conversation

DownstreamDataTeam
Copy link
Contributor

@DownstreamDataTeam DownstreamDataTeam commented Jul 21, 2022

Description of change

  • Implemented better datetime utils (taking into account timezones, and converting all data to UTC for processing), so we can eliminate all timezone related issues
  • Implemented function to get tenant timezone (so calculations to “local time” are done using tenant TZ, not local computer TZ)
  • Added improved datetime utils to all streams
  • Made the following streams use datetime instead of just date:
    • clients
    • communications
    • deposit_transactions
    • loan_accounts
    • loan_transactions
  • Refactored processors to clean up the code
  • Fixed issue where loan_accounts processor was not collecting performance metrics

Rollback steps

  • revert this branch

Radu Marinoiu and others added 30 commits March 24, 2022 09:31
[ECDDC-591] Fixed issue with users deduplication key

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!78
…' into 'release/40'

[ECDDC-603] Added interest accrual breakdown stream

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!74
…se/40'

[ECDDC-649] Added task_link_key field to tasks stream

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!75
[ECDDC-652] Adjusted Snyk dev test

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!67
…ease/40'

[ECDDC-653] Finished implementing catalog automatic fields checker test

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!77
…b.com:mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu into feature/ECDDC-657_Refactor-audit-trail-stream
…nto 'release/40'

[ECDDC-667] Added missing fields to loan_transaction's 'affected_amounts' object

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!81
…ease/40'

[ECDDC-658] Refactor activities stream

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!79
Radu Marinoiu added 6 commits July 19, 2022 15:15
… into 'release/48'

[ECDDC-724] Refactor timezone information system

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!125
…m-tap-processor' into 'release/48'

[ECDDC-730] Decoupled Multithreading processor from TapProcessor

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!126
…counts-processor' into 'release/48'

[ECDDC-737] Fixed Deduplication Processor PerformanceMetrics Bug

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!127
@singer-bot
Copy link

Hi @DownstreamDataTeam, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

Radu Marinoiu and others added 9 commits July 22, 2022 15:29
… into 'release/48'

[ECDDC-724] Fix for timezone system

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!128
[ECDDC-732] Merge master into release/48

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!135
# Conflicts:
#	mambu_tests/conftest.py
#	mambu_tests/helpers.py
#	mambu_tests/multithreading/test_multithreaded_bookmark_daybyday_generator.py
#	mambu_tests/multithreading/test_multithreaded_bookmark_generator.py
#	mambu_tests/tap_generators/test_activities_generator.py
#	mambu_tests/tap_generators/test_groups_generator.py
#	mambu_tests/test_client.py
#	mambu_tests/test_sync.py
#	tap_mambu/helpers/multithreaded_requests.py
#	tap_mambu/sync.py
#	tap_mambu/tap_generators/activities_generator.py
#	tap_mambu/tap_generators/audit_trail_generator.py
#	tap_mambu/tap_generators/clients_generator.py
#	tap_mambu/tap_generators/communications_generator.py
#	tap_mambu/tap_generators/deposit_accounts_generator.py
#	tap_mambu/tap_generators/deposit_transactions_generator.py
#	tap_mambu/tap_generators/gl_journal_entries_generator.py
#	tap_mambu/tap_generators/groups_generator.py
#	tap_mambu/tap_generators/installments_generator.py
#	tap_mambu/tap_generators/interest_accrual_breakdown_generator.py
#	tap_mambu/tap_generators/loan_transactions_generator.py
#	tap_mambu/tap_generators/multithreaded_bookmark_generator.py
#	tap_mambu/tap_processors/deduplication_processor.py
#	tap_mambu/tap_processors/multithreaded_parent_processor.py
#	tap_mambu/tap_processors/processor.py
Solved conflicts between release/48 and master branch

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!137
@DownstreamDataTeam DownstreamDataTeam changed the title Release/48 Timezone improvement Aug 31, 2022
@DownstreamDataTeam DownstreamDataTeam marked this pull request as ready for review August 31, 2022 10:48
Alexandru Rosca added 4 commits September 14, 2022 19:21
Sync Master with release/48

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!139
[ECDDC-756] Bugfix: Added missing date formats

See merge request mambucom/product/ecosystem/mambu-marketplace/connectors/singer/tap-mambu!140


def datetime_to_local_str(dttm: datetime) -> str:
return dttm.astimezone(_timezone).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will leave the local time in the format 2022-09-20T14:10:24.865664Z. This returns the correct local time, but the Z at the end implies UTC time. Adding %z to the end of the time format will add the offset to the UTC time, which seems to be the intended return value. The ISO-8601 format would be %Y-%m-%dT%H:%M:%S.%f%z This would make the return value 2022-09-20T14:10:24.865664-0400 when using US/Eastern as the timezone.

@bryantgray bryantgray merged commit d119019 into singer-io:master Sep 27, 2022
@bryantgray bryantgray mentioned this pull request Sep 27, 2022
@singer-bot
Copy link

You did it @DownstreamDataTeam!

Thank you for signing the Singer Contribution License Agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants