470 IngestionMediator class

[jkwening]

Note

This is an overhaul of our ingestion process. This PR request is now stable and passed all included unit tests. services.py is now the point of access for user requests related ingestion workflow, weekly_update process, and ancillary features. The following modules are now obsolete:

• load_data.py: replaced by services.py

• functions.py: its methods have been refactored into Meta.py - class encapsulation of meta.csv

What's In This

Implemented mediator design pattern for ingestion objects and work flow. This required:

• creating an ingestion_mediator module: housing IngestionMediator class
• creating a base colleague class that will be inherited by all objects will communicate with the mediator
• refactored LoadData, Manifest, HISql, and GetApiData to all inherit from Colleague class and utilize IngestionMediator to coordinate activities.
  • these are now decoupled from one another and communicate via IngestionMediator and never directly with one another
• refactored and encapsulated meta.json related methods into Meta class within Meta.py module
• LoadData can now load data either from existing cleaned.psv files or by processing and cleaning raw_data.csv files.
• added 'dependency' field to manifest.csv that will track dependencies across unique_data_ids and if requested will automatically update and reload dependent unique data ids. This is the default behavior for weekly update procedure.

TODO

• refactor LoadData, Manifest, and Cleaner modules to decouple them
  from each other and have IngestionMediator coordinate their activities
• decouple cleaner workflow from LoadData workflow and coordinate activity into raw data processing work flow - end goal is for LoadData to only load successfully cleaned PSV files. For prescat raw data, S3 bucket update will trigger cleaner process, and reloading into database.
• implement solution and workflow for resolving double-adding projects issue based on
  'dependency' field in manifest Error logging and cron jobs #470
  • implement reverse dependency feature - for a given data id, check whether the data that it is dependent on is loaded in the db, if not, get that loaded into the db and recursively whatever this is dependent on prior to loading the originally requested data id low priority feature
• merge some of services.py code as methods in IngestionMediator class and refactor so it instantiates an instance of IngestionMediator allowing it to do regularly scheduled updates on the server and another method for on demand requests by user (update-only or rebuild - maintain current functionalities and expand if needed)
  • refactor weekly_update() in services
  • add methods for on demand requests by user in services.py
  • tie in send log file to admin feature
• clean up unnecessary codes that have been refactored or are now obsolete