Add download files to example Python Large Output Files #503
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #503 +/- ##
=======================================
Coverage 92.45% 92.45%
=======================================
Files 64 64
Lines 2599 2599
=======================================
Hits 2403 2403
Misses 196 196 ☔ View full report in Codecov by Sentry. |
saimanikant
force-pushed
the
mguntupa/example
branch
from
a70d70d to
c07e55f
Compare
for more information, see https://pre-commit.ci
| """ | ||
| Example to query resources from a project. | ||
|
|
||
| - Query values from evaluated jobs, computing some simple statistics on parameter values. | ||
| - Download files from the project | ||
|
|
||
| """ |
There was a problem hiding this comment.
please update the docstring, there's no querying of statistics here
| log.info( | ||
| f"=== Example 1: Downloading output files of {num} jobs using ProjectApi.download_file()" | ||
| ) | ||
| for job in jobs[0:num]: |
There was a problem hiding this comment.
| for job in jobs[0:num]: | |
| for job in jobs: |
| num = len(jobs) | ||
|
|
||
| log.info( | ||
| f"=== Example 1: Downloading output files of {num} jobs using ProjectApi.download_file()" |
There was a problem hiding this comment.
There's no example 2, please clean up the msg
| for f in files: | ||
| fpath = os.path.join(out_path, f"task_{task.id}") | ||
| log.info(f"Download output file {f.evaluation_path} to {fpath}") | ||
| start = time.process_time() |
There was a problem hiding this comment.
Best to use the same timing function you use 2 lines later, otherwise you get meaningless numbers
| start = time.process_time() | |
| start = time.time() |
| args = parser.parse_args() | ||
|
|
||
| logger = logging.getLogger() | ||
| logging.basicConfig(format="%(message)s", level=logging.DEBUG) |
There was a problem hiding this comment.
maybe default the log level to INFO, to avoid the many DT client debug messages
|
|
||
| def download_files(client, project_name): | ||
| """Download files.""" | ||
| out_path = os.path.join(os.path.dirname(__file__), "downloads") |
There was a problem hiding this comment.
I'd make the download dir a configurable CLI argument and even default it to a temp directory if unset, those downloaded files are of no use.
| for f in files: | ||
| fpath = os.path.join(out_path, f"task_{task.id}") | ||
| log.info(f"Download output file {f.evaluation_path} to {fpath}") | ||
| start = time.process_time() |
There was a problem hiding this comment.
The first timing won't be accurate because it will include initialization of the DT client. Is there a way to force that to happen outside of the loop?
saimanikant
force-pushed
the
mguntupa/example
branch
from
3a192aa to
e7b7b77
Compare
Description
Please provide a brief description of the changes in this pull request.
Checklist