Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise pytesting, update docker, add QC notebook #32

Merged
merged 11 commits into from
Jan 11, 2023
Merged

Revise pytesting, update docker, add QC notebook #32

merged 11 commits into from
Jan 11, 2023

Conversation

CBroz1
Copy link
Contributor

@CBroz1 CBroz1 commented Jan 4, 2023

See also element PR

Submission checklist

  • Reviewed all Files changed prior to selecting Create Pull Request.
  • Ran and passed all pytests, and ran the pipeline for features not covered by tests.
  • Ran pre-commit checks, including black, flake8, etc.
  • Updated the changelog and if making a new release, updated the version.py. - Will add 'release date' after review
  • Updated documentation, including concepts.md, tutorials.md, and relevant docstrings.
  • After merge, I will push a new tag to upstream (test* prefix if no new release).

Submission details

In this pull request, I make the following changes:

  • markdown lint: standardize docs - remove trailing spaces, line length (markdownlint.yaml, pre-commit-config.yaml, readme)
  • revise pytesting
    • add pyproject.toml - add default test path and configs to speed up test initialization
    • init -> conftest - following pytest documentation, startup fixtures go here. removes the need for imports in test docs
    • migrate standard funcs to element_interface (see PR #71)
    • add command-line options for verbosity and teardown
    • tests/*py remove imports per conftest change above and adjust variable names for clarity
    • conftest: add pytest fixtures
      • add fixture for element helper funcs
      • add func to loop populate command and parallel teardown func (populate_table,populate_clear)
      • add fixtures to populate motion correction, segmentation, etc post_curation
    • tests_populate - add tests for results (e.g., activity traces)
    • tests_support - add tests for element support funcs (e.g., get_root_data_dir)
  • requirements.txt - remove old git+ format leftover from prior to pypi publication
  • requirements_test -> requirements_dev: jupytext and pre-commit are developer dependencies. pytest/pytest-cov is also for developer testing
  • sessions.csv - rename acq_software per element PR #38
  • workflow structure - reorganize to avoid monkey patching
    • workflow/init: migrate db_prefix here to be available to both pipeline and reference
    • workflow/reference: migrate supplemental tables, Equipment -> Device
    • workflow/ingest: print->logger statements, Equipment -> device, parameterize print str

Coverage report

Overall, I improve coverage by ~40%

----- coverage: platform darwin, python 3.9.13-final-0 -----
Name                             Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------------------------------------
element_miniscope/__init__.py        0      0   100%
element_miniscope/miniscope.py     281     57    80%   84, 87, 244, 247-267, 274, 278, 300, 497-558, 592, 604, 768-821, 843, 948, 1000, 1070, 1165, 1199
element_miniscope/version.py         1      1     0%   2
--------------------------------------------------------------------------------------------------------
TOTAL                              282     58    79%

Missing

  • Single lines: primarily error statements (e.g., FileNotFound)
  • 247-267 : V3 data is not in sample dataset
  • 497-558 : task mode trigger is not tested - time consuming
  • 768-821 : nonrigid motion correction

Docker

I'm able to install docker if I hop in to the container, but my install scripts time out during docker compose. There are perhaps workarounds, but I leave this to future PRs. Instead, I leave warnings on the topic.

- markdownlint: remove trailing spaces
- pre-commit-config: remove trailing spaces
- requirements_dev: collapse across development/testing requirements
- tests/__init__ -> conftest per pytest documentation
- tests/*py: remove extra imports
- tests/*py: adjust variable names for clarity
- workflow/init: migrate db_prefix here to be available to both pipeline and reference
- workflow/reference: migrate supplemental tables, Equipment -> device
- workflow/ingest: print->logger statements, Equipment -> device, parameterize print str
- workflow/pipeline: Loc
- pyproject.toml: NEW. gives default params and path to pytest to speed up init
@CBroz1 CBroz1 changed the title See details Revise pytesting, update docker Jan 4, 2023
@CBroz1 CBroz1 self-assigned this Jan 4, 2023
@CBroz1 CBroz1 linked an issue Jan 4, 2023 that may be closed by this pull request
CBroz1 added 2 commits January 5, 2023 15:35
WIP notebooks and jupytext add to precommit
Dockerfile - add caiman install
populate.py -> process.py to reflect standard elsewhere
paths.py - add output typing suggestions
conftest pytest fixtures
  - set default teardown true
  - add fixture for element helper funcs
  - add func to loop populate command and parallel teardown func
  - add fixtures to populate motition correction, segmentation, etc
tests_populate - add tests for results (e.g., activity traces)
tests_support - add tests for element support funcs
@CBroz1 CBroz1 requested a review from kushalbakshi January 5, 2023 22:35
@CBroz1 CBroz1 marked this pull request as ready for review January 5, 2023 22:36
@CBroz1 CBroz1 removed their assignment Jan 5, 2023
@CBroz1 CBroz1 linked an issue Jan 9, 2023 that may be closed by this pull request
@CBroz1 CBroz1 changed the title Revise pytesting, update docker Revise pytesting, update docker, add QC notebook Jan 10, 2023
@kushalbakshi kushalbakshi merged commit d91409b into datajoint:main Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Docker compose revise volume mapping Add 04-automate Jupyter notebook
2 participants