Fix logging filesystem failing to initialize with high number of logs #1294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelist
On master, if there are a significant number of CAN logs present on the SD card (somewhere in the range 50-100) the watchdog trips at startup. This means the VC never gets past init.
Root cause: Currently, mounting the filesystem and starting logging takes about 250ms for every 100 logs on the SD card, and scales linearly. There is no significant correlation with the size of each log file. Since the watchdog trips if not checked in every 200ms, it just simply takes too long. Note that we start a new log after every boot.
Solution: I think anything less than a second of extra delay at bootup is not a big deal, so I don't think this motivates any big changes to our logging. There is no way to stop the STM32 watchdog peripheral once its started, so instead I added a
tasks_preInitWatchdogfunction which runs after all peripherals are initialized, but before the watchdog is. So, we initialize logging in here and then can proceed like normal.Also, I added some CAN signals so there will be visibility on when we've logged a significant amount of data.
Testing Done
Tested on VC #2. Able to init logging in ~500ms total with 200 logs without the watchdog tripping, and the VC behaves like normal after that. Tested with large files as well (>>100MB) without any slowdown. Performance is approximately O(N) at all values tested.
Also, interestingly it seems that different SD cards are faster than others. The KOOTION 64GB Micro SDXC card is approximately 2x as fast as the Lexar 300x 32GB SDHC card.