Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[API] Add dicom upload #9154

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

Conversation

maximemulder
Copy link
Contributor

@maximemulder maximemulder commented Mar 20, 2024

Brief summary of changes

This is a port of #6161, which contains a description of the changes. Since the old PR has diverged quite a bit from the main branch, and that its author no longer works here, I found it easier to copy the changes on the current main branch and open a new PR rather than to rebase the PR. While porting, I only rewrote the old PR code whenever static analysis or practical stests showed that it was no longer in sync with the main branch (or when my IDE removed trailing spaces).

Testing instructions (if applicable)

  • Test the four new API routes, described in section 5 of the documentation of the API v0.0.4.

Notes for existing projects

This PR requires to run the following SQL patch SQL/New_patches/2024-03_19-MRIUploadServerProcessRel.sql.

@maximemulder
Copy link
Contributor Author

maximemulder commented Mar 20, 2024

I must say I am not sure what I am supposed to do regarding the failed tests on files that I did not modify in the PR. Any direction @driusan (or someone else) 🤔.

@kongtiaowang
Copy link
Contributor

kongtiaowang commented Apr 24, 2024

Line php/libraries/FilesDownloadHandler.php
71 Parameter #1 $value of function strval expects
bool|float|int|resource|string|null, mixed given.

Line php/libraries/SiteIDGenerator.php
81 Negated boolean expression is always false.
💡 Because the type is coming from a PHPDoc, you can turn off this
check by setting treatPhpDocTypesAsCertain: false in your
./test/phpstan-loris.neon.
297 Parameter #1 $value of function strval expects
bool|float|int|resource|string|null, mixed given.
377 Parameter #1 $value of function strval expects
bool|float|int|resource|string|null, array<int,
int|string>|string|null given.
387 Parameter #1 $value of function strval expects
bool|float|int|resource|string|null, array<int, int|string>|string
given.
400 Parameter #1 $value of function strval expects
bool|float|int|resource|string|null, array<int, int|string>|string
given.

@maximemulder
Copy link
Contributor Author

Hum, yeah, I already saw that.

I was just suprised to see that the errors are marked as originating from FilesDownloadHandler.php and SiteIDGenerator.php, which are not modified by this PR. I guess I must call these files from elsewhere and that the location reporting is just imprecise.

I'll tackle this tomorrow probably.

@maximemulder
Copy link
Contributor Author

I've looked at the static errors in more details, it appears to be mainly due to an upgrade of phpstan, which changes the typing of string conversions with strval and (string) from mixed -> string to (bool|float|int|resource|string|null) -> string. I still don't know why this PR triggers the error and not other ones (it did trigger on my main branch though), but at least the errors are clear.

However, I find the code that triggers the errors to be quite cryptic, with some lines that seem obviously wrong (at least according to typing). If someone who knows this code better than me could look into it, it would be best. In the meantime, I'm adding this to the next meeting.

@laemtl
Copy link
Contributor

laemtl commented May 28, 2024

I still don't know why this PR triggers the error and not other ones (it did trigger on my main branch though), but at least the errors are clear.

Probably because composer update was run, which triggered an update of the static checker.

@maximemulder
Copy link
Contributor Author

Ok it seems the linting errors are fixed, silenced. @laemtl had noticed that the errors were due to a composer update on my end. The errors should still appear in any PR that updates composer and should be fixed eventually. However, this can have its own issue and be another PR.

@maximemulder
Copy link
Contributor Author

maximemulder commented Aug 29, 2024

@ridz1208 Ok I finally finished updating and re-testing this PR. Generally it seems to work. However, if the pipeline fails during the latter stages, the DICOM archive previously inserted does not have its session initialized. This should be fixed with the new DICOM archive PR, but I don't expect it to be merged before at least a few weeks.

@maximemulder maximemulder force-pushed the api-dicom-upload branch 8 times, most recently from 4b8a6dc to 9765668 Compare August 29, 2024 18:04
Copy link
Collaborator

@cmadjar cmadjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximemulder I did a first pass of review. I did not have a chance to actually test the changes as my MCIN sandbox is not letting me logging in. Have to figure out why and it could take some time so figured I would do an initial review.

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Show resolved Hide resolved
modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Outdated Show resolved Hide resolved
modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Show resolved Hide resolved
modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Outdated Show resolved Hide resolved
GET /candidates/$CandID/$VisitLabel/dicoms/$Tarname/processes
```

The response contains all `mri_upload` attempts with the specified `$tarname`. And for
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

technically, there should only be one UploadID per TarchiveID. So when you overwrite, you should grep the UploadID associated to what is being overwritten instead of inserting a new entry in mri_upload. Especially since the incoming directory content will be overwritten anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that there is only one MRI upload per DICOM archive, but I've seen a few cases in testing (even without this PR) where there are a few failed MRI uploads and one successful for a single file. This is how the batch uploader does it (don't know about other pipelines).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. I think in principle, there should only be one upload per tarchive but unfortunately the batch upload script is not compliant with this set up. However, for the API, I think we should make sure this is the behaviour at minimum, to avoid duplicated entries.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this thread resolved?

modules/api/php/endpoints/candidate/visit/dicoms.class.inc Outdated Show resolved Hide resolved
modules/api/php/endpoints/candidate/visit/dicoms.class.inc Outdated Show resolved Hide resolved
modules/imaging_browser/test/imaging_browserTest.php Outdated Show resolved Hide resolved
php/libraries/DicomTar.class.inc Outdated Show resolved Hide resolved
src/Data/Models/DicomSeriesDTO.php Outdated Show resolved Hide resolved
@driusan
Copy link
Collaborator

driusan commented Sep 4, 2024

Please remove all the unrelated changes (whitespace, variable name changes, etc). If necessary you can send those in another PR. It is not feasible to review this PR with so many unrelated changed mixed together.

@maximemulder maximemulder force-pushed the api-dicom-upload branch 2 times, most recently from 65869da to 4c63612 Compare September 17, 2024 20:15
@maximemulder
Copy link
Contributor Author

maximemulder commented Sep 17, 2024

I replied to all the comments and applied the suggested changes.

I reverted the breaking changes to the API including floats / strings and naming. Back when I wrote them, I thought it was part of the original PR and not something that was already in LORIS.

I made the terminology more consistent by using $VisitLabel everywhere in the DICOM part (section 5) of the API documentation.

I also disabled my trailing spaces auto-removal and removed the associated changes.

Ready for review @cmadjar @driusan

"SeriesDescription" : "MPRAGE_ipat2",
"SeriesNumber" : "2",
"SeriesNumber" : 2,
Copy link
Contributor Author

@maximemulder maximemulder Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding this change, the current documentation is inaccurate, I can extract this to a separate PR if wanted.

Example:
https://demo.loris.ca/api/v0.0.4-dev/candidates/587630/V1/dicoms

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to look into this to see if it's a bug in the documentation or the code. It seems weird that it's only the SeriesNumber that does it and it might be related to PHP (not LORIS) changes in the PDO over time.

It might be faster to split it out.

@maximemulder maximemulder force-pushed the api-dicom-upload branch 2 times, most recently from 6f882bc to 8112fc8 Compare September 23, 2024 15:28
Copy link
Collaborator

@cmadjar cmadjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximemulder I re-reviewed quickly your PR. A few additional comments:

  • In the description of the PR, could you add a "Notes for existing project" with the information of the SQL patches to run?

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Show resolved Hide resolved
modules/api/php/endpoints/candidate/visit/dicoms.class.inc Outdated Show resolved Hide resolved
@driusan driusan self-requested a review November 11, 2024 18:41
@maximemulder
Copy link
Contributor Author

@cmadjar One of your comments was that you would like to the DICOM API upload to re-use an existing MRI upload if there is one. Is the following commit good for you ?

d3eb5c5

I notably do not update the upload timestamp as the DICOM archive has its own timestamp and the MRI upload may be referenced by other, older, data. But I can change that if you want to. I also only re-use an existing MRI upload if its parameters are similar (upload location, uploader id...).

Copy link
Collaborator

@cmadjar cmadjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the last change!

@maximemulder maximemulder added Category: Feature PR or issue that aims to introduce a new feature Project: C-BIG Issue or PR related to the C-BIG project Difficulty: Complex PR or issue that require a great effort to implementat, review, or test Area: Imaging PR or issue related to imaging Priority: High PR or issue should be prioritised over others for review and testing and removed Language: PHP PR or issue that update PHP code labels Nov 29, 2024
@maximemulder
Copy link
Contributor Author

maximemulder commented Dec 2, 2024

@driusan I believe I answered all the issues that were raised by Cécile (who approved the PR) and you. I also updated the 0.0.4-dev Swagger schema to match the markdown documentation. You can review the last 7 commits commit by commit to see the changes.

@maximemulder maximemulder added Release: SQL patch PR that contains an SQL patch to apply and removed Language: SQL PR or issue that update SQL code labels Dec 2, 2024
@maximemulder maximemulder removed the Priority: High PR or issue should be prioritised over others for review and testing label Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: API PR or issue related to the API Area: Imaging PR or issue related to imaging Category: Feature PR or issue that aims to introduce a new feature Difficulty: Complex PR or issue that require a great effort to implementat, review, or test Passed manual tests PR has been successfully tested by at least one peer Project: C-BIG Issue or PR related to the C-BIG project Release: SQL patch PR that contains an SQL patch to apply
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants