added suffix and prefix capabilities in tgocassisstitch #5132
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I added the following changes to create_rclone_arguments:
- I added a default filter flag --filter with the value of exclude_string
- I added a check for any additional --include and --exclude flags that may have
been passed in by the user.
- I added logic to merge any additional --include and --exclude flags with the
default --filter flag. The include and exclude patterns are concatenated with
the default exclude_string separated by comma.
- I removed the f"--exclude={exclude_string}" from extra_args list, since the
default exclude_string is now included in the filter flag
- I added the filter flag to the extra_args list using
extra_args.extend(filter_args)
All of the above changes ensures that the --include and --exclude flags passed
in by the user are taken into account while creating extra_args, and also
the logic will merge these flags with the default filter flag, which is the
recommended way as per the rclone docs.
along with a few other changes:
+ I added a check for any --filter flag provided by the user, if present it
will use it and ignore the default filter flag. Otherwise, it will use the default
filter flag. This is done to take into account if the user has provided any
specific filter flag, and it will honor the user's intention of providing
the filter flag.
+ I added a check for any additional --include and --exclude flags passed in
by the user and merge them with the filter flag. This is to take into
account any specific include/exclude patterns that the user wants to apply,
and merge them with the default filter flag.
+ I added a "+" at the end of the filter string if the user has specified an
--include flag. This simulates the behavior of --include where it includes
any patterns specified and excludes everything else.
+ I also added a check for filter_args_provided and if provided, it will use
this flag, else it will use the default filter flag and merge any additional
include/exclude flags to it.
There is also a test in the pytest file to check if the filter logic works as
expected, run using `pytest`.
**The first test that was added is `test_rclone_with_auth`**
This test is designed to check the behavior of the `rclone` function when it
is called with an `auth` parameter. This test checks that the `rclone`
function properly passes the "auth" parameter to the underlying subprocess
call.
**The second test that was added is `test_create_rclone_args_with_no_kwargs`**
This test is designed to check the behavior of the `create_rclone_args`
function when it is called with no keyword arguments. This test checks that
the `create_rclone_args` function properly handles the case when it is
called with no keyword arguments and returns the correct list of arguments
to be passed to the underlying subprocess call
**The third test that was added is `test_file_filtering_with_hidden_files`**
This test is designed to check the behavior of the `file_filtering` function
when it is called with hidden files in the specified directory. This test
checks that the `file_filtering` function properly filters out hidden files
and only returns the non-hidden files in the specified directory.
I have tested to confirm these all run and pass on my machine.
**added `test_rclone` test**
This test mocks the `subprocess.Popen` function and checks that the output
of the rclone function matches the expected output when the function is
invoked with the arguments `lsf`, `test`, `["-l", "-R", "--format", "p",
"--files-only"]`, `True`, and `True`.
**added `test_rclone_unknown_exception` test**
This test mocks the `subprocess.Popen` function and checks that the `rclone`
function raises an exception when an unknown exception is encountered. This
test uses a mocked class that raises an exception when it is initialized.
I have tested and confirmed these to work on my system.
fixed how filters are input, patterns still aren't working I believe I need
to look at their patterns and ensure the adhere to rclone documentation
syntactical instructions.
addresses DOI-USGS#5125, this allows the user to optionally use either the prefix or suffix option when stitching frames together, just use OUTPUTSUFFIX instead of OUTPUTPREFIX to use it.
need to figure out a replacement for ui.IsOptionSet as that is not actually
a function. I need another way to test CLLI args.
…bility
it also maintains original value names so older scripts and pipelines will
retain functionality.
lets the user specify an optional cubename parameter which replaces the
timestamp style naming for the output stitched cube. If the parameter is not
given it defaults to the timestamp style naming convention for retaining
existing functionality.
…bility
it also maintains original value names so older scripts and pipelines will
retain functionality.
|
@antonhibl Good on building now, only one more to go! https://jenkins-hsts.prod-asc.chs.usgs.gov/job/ISIS/view/change-requests/job/PR-5132/ The DemCube test failure is something else you don't have to worry about |
Tgo fixes, that PR was a mess lol
the test left is a make test so need to look into it more.
resolved more tests, 1 to go
|
closed this in favor of opening #5162 with a cleaner commit history and branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
addresses #5125, this allows the user to optionally use either the prefix or
suffix option when stitching frames together, just use OUTPUTSUFFIX instead
of OUTPUTPREFIX to use it.
Related Issue
#5125
How Has This Been Validated?
Tested locally and works by passing in either a OUTPUTSUFFIX or OUTPUT PREFIX.
Types of changes
Checklist:
Licensing
This project is mostly composed of free and unencumbered software released into the public domain, and we are unlikely to accept contributions that are not also released into the public domain. Somewhere near the top of each file should have these words: