Rob Shiv we need a way of tracking what has been processed to S3 to date

[robfatland]

No description provided.

[pshivraj]

@robfatland Since we now have a folder structure for each day processed, can we use boto to check for all the folder names (non-empty folders) and use google sheets as a tracker?

[robfatland]

I am against using google sheets because this creates a separate point of reference, i.e. not part of the GitHub repo; and I dislike bouncing around between websites. To elaborate: Anisha points out that tens of thousands of ten-second-clips is really pushing 'too much' for doing ml training; and just one day produces 8640 such clips. Which Jan 12 2019 happens to be great for because of the consistent presence of megapterae in the 500 meter (Endurance Offshore) signal. For this iteration I suggest we get that day "in the bag" (and possibly add another day with low (zero) humpback calls) and call that good.

[pshivraj]

Makes sense to have it all at one place then. I was thinking to then make these dates a part of Readme? So a Travis job that runs and creates a table and updates the Readme with table containing the dates, number of spectrogram and other information from s3. This would be again a way to keep everything on github though not a priority to be implemented right away?

[robfatland]

oh that's pretty cool. i replied by email to see if it gets there... should auto-mod of README is pretty cool, is what i mean

…

On Thu, May 2, 2019 at 9:56 PM Shivraj ***@***.***> wrote: Makes sense to have it all at one place then. I was thinking to then make these dates a part of Readme? So a Travis job that runs and creates a table and updates the Readme with table containing the dates, number of spectrogram and other information from s3. This would be again a way to keep everything on github though not a priority to be implemented right away? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPJRWOJFBHFD23WJURI7ILPTPAWJANCNFSM4GTU6JBQ> .

-- Rob Fatland UW Research Computing Director