Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random UT Failure: PosixFile #2835

Closed
LeStarch opened this issue Aug 16, 2024 · 13 comments
Closed

Random UT Failure: PosixFile #2835

LeStarch opened this issue Aug 16, 2024 · 13 comments
Assignees
@rlcheng
Labels
bug
Milestone
Release v3.5.0

Comments

@LeStarch
Copy link
Collaborator

LeStarch commented Aug 16, 2024 •
edited
Loading

F´ Version
Affected Component

Problem Description

55/66 Test #55: PosixFileTest ....................***Failed    3.34 sec
[STest::Random] Generated seed 965653 from system time
[==========] Running 32 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 14 tests from Functionality
[ RUN      ] Functionality.OpenWithCreation
[       OK ] Functionality.OpenWithCreation (7 ms)
[ RUN      ] Functionality.Close
[       OK ] Functionality.Close (0 ms)
[ RUN      ] Functionality.AssignmentOperator
[       OK ] Functionality.AssignmentOperator (0 ms)
[ RUN      ] Functionality.CopyConstructor
[       OK ] Functionality.CopyConstructor (0 ms)
[ RUN      ] Functionality.OpenWithCreationOverwrite
[       OK ] Functionality.OpenWithCreationOverwrite (0 ms)
[ RUN      ] Functionality.OpenInvalidModes
[       OK ] Functionality.OpenInvalidModes (0 ms)
[ RUN      ] Functionality.PreallocateWithoutOpen
[       OK ] Functionality.PreallocateWithoutOpen (0 ms)
[ RUN      ] Functionality.SeekWithoutOpen
[       OK ] Functionality.SeekWithoutOpen (0 ms)
[ RUN      ] Functionality.FlushInvalidModes
[       OK ] Functionality.FlushInvalidModes (0 ms)
[ RUN      ] Functionality.ReadInvalidModes
[       OK ] Functionality.ReadInvalidModes (0 ms)
[ RUN      ] Functionality.WriteInvalidModes
[       OK ] Functionality.WriteInvalidModes (0 ms)
[ RUN      ] Functionality.RandomizedInterfaceTesting
Ran 100 steps.
[       OK ] Functionality.RandomizedInterfaceTesting (1245 ms)
[ RUN      ] Functionality.FullCrcInvalidMode
[       OK ] Functionality.FullCrcInvalidMode (0 ms)
[ RUN      ] Functionality.IncrementalCrcInvalidMode
[       OK ] Functionality.IncrementalCrcInvalidMode (0 ms)
[----------] 14 tests from Functionality (1256 ms total)

[----------] 9 tests from FunctionalIO
[ RUN      ] FunctionalIO.OpenWithCreationExists
[       OK ] FunctionalIO.OpenWithCreationExists (0 ms)
[ RUN      ] FunctionalIO.SeekInvalidSize
[       OK ] FunctionalIO.SeekInvalidSize (0 ms)
[ RUN      ] FunctionalIO.WriteReadBack
[       OK ] FunctionalIO.WriteReadBack (11 ms)
[ RUN      ] FunctionalIO.WriteReadSeek
[       OK ] FunctionalIO.WriteReadSeek (4 ms)
[ RUN      ] FunctionalIO.WriteFullCrc
[       OK ] FunctionalIO.WriteFullCrc (10 ms)
[ RUN      ] FunctionalIO.WritePartialCrc
[       OK ] FunctionalIO.WritePartialCrc (4 ms)
[ RUN      ] FunctionalIO.Flush
[       OK ] FunctionalIO.Flush (6 ms)
[ RUN      ] FunctionalIO.Preallocate
[       OK ] FunctionalIO.Preallocate (30 ms)
[ RUN      ] FunctionalIO.RandomizedTesting
/Users/runner/work/fprime/fprime/Os/test/ut/file/FileRules.cpp:1050: Failure
Expected equality of these values:
  crc
    Which is: 4294967295
  shadow_crc
    Which is: 2265461553
/Users/runner/work/fprime/fprime/Os/test/ut/file/FileRules.cpp:1050: Failure
Expected equality of these values:
  crc
    Which is: 4294967295
  shadow_crc
    Which is: 2552232243
Ran 1000 steps.
[  FAILED  ] FunctionalIO.RandomizedTesting (1621 ms)
[----------] 9 tests from FunctionalIO (1691 ms total)

[----------] 9 tests from InvalidArguments
[ RUN      ] InvalidArguments.OpenBadPath
[       OK ] InvalidArguments.OpenBadPath (39 ms)
[ RUN      ] InvalidArguments.OpenBadMode
[       OK ] InvalidArguments.OpenBadMode (26 ms)
[ RUN      ] InvalidArguments.PreallocateBadOffset
[       OK ] InvalidArguments.PreallocateBadOffset (19 ms)
[ RUN      ] InvalidArguments.PreallocateBadLength
[       OK ] InvalidArguments.PreallocateBadLength (37 ms)
[ RUN      ] InvalidArguments.SeekAbsoluteWithNegativeLength
[       OK ] InvalidArguments.SeekAbsoluteWithNegativeLength (48 ms)
[ RUN      ] InvalidArguments.ReadInvalidBuffer
[       OK ] InvalidArguments.ReadInvalidBuffer (35 ms)
[ RUN      ] InvalidArguments.ReadInvalidSize
[       OK ] InvalidArguments.ReadInvalidSize (34 ms)
[ RUN      ] InvalidArguments.WriteInvalidBuffer
[       OK ] InvalidArguments.WriteInvalidBuffer (19 ms)
[ RUN      ] InvalidArguments.WriteInvalidSize
[       OK ] InvalidArguments.WriteInvalidSize (19 ms)
[----------] 9 tests from InvalidArguments (282 ms total)

[----------] Global test environment tear-down
[==========] 32 tests from 3 test suites ran. (3230 ms total)
[  PASSED  ] 31 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FunctionalIO.RandomizedTesting

 1 FAILED TEST
@LeStarch LeStarch added the bug label Aug 16, 2024
@LeStarch LeStarch changed the title Random UT Failure Random UT Failure: PosixFile Aug 16, 2024
This was referenced Aug 16, 2024
Open
Merged
@thomas-bc thomas-bc added this to the Release v3.5.0 milestone Aug 21, 2024
@rlcheng
Copy link
Collaborator

rlcheng commented Aug 27, 2024

@LeStarch @thomas-bc which version of fprime (or branch) did this fail on? I'm running the latest in devel commit a31f557 and all UTs are passing.

@LeStarch
Copy link
Collaborator Author

@rlcheng You'll need to set the above see in the UT to get it to crash. The seed is: 965653 and can be updated at this line:

fprime/Os/Posix/test/ut/PosixFileTests.cpp

Line 141 in 4476828

STest::Random::seed();

@LeStarch
Copy link
Collaborator Author

I believe you want to change that to:

STest::Random::SeedValue::set(965653);

@LeStarch
Copy link
Collaborator Author

@rlcheng here is another failing seed 279364.

@JohanBertrand JohanBertrand mentioned this issue Sep 1, 2024
Closed
rlcheng added a commit to rlcheng/fprime that referenced this issue Sep 4, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
408155b 
 * Fix UT failing FinalizeCrc test when crc calculation miss match.
   Miss match due to file crc and shadow being out of sync when
   both are not reinialized at the same time.

 * Update File::open in File.ccp to match shadow operation in
   FileRules.cpp to only reset crc when status is OP_OK.

 * Update FileRules.cpp to print out rule name during test.

 * Remove changing file mode in shadow_open() in FileRules.cpp to
   OPEN_NO_MODE when status is not OP_OK.
@rlcheng
Copy link
Collaborator

rlcheng commented Sep 4, 2024

Fix complete. PR is here: #2860

LeStarch pushed a commit that referenced this issue Sep 4, 2024
@rlcheng
Fix Random UT Failure: PosixFile (#2835) (#2860)
66d5128 
* Fix UT failing FinalizeCrc test when crc calculation miss match.
   Miss match due to file crc and shadow being out of sync when
   both are not reinialized at the same time.

 * Update File::open in File.ccp to match shadow operation in
   FileRules.cpp to only reset crc when status is OP_OK.

 * Update FileRules.cpp to print out rule name during test.

 * Remove changing file mode in shadow_open() in FileRules.cpp to
   OPEN_NO_MODE when status is not OP_OK.
@JohanBertrand
Copy link
Contributor

@rlcheng Could you please check that the commented unit tests here are passing with your fix? #2854

It seems like it is still failing for me. I am not sure if it's because of my test setup, or the problems are still here.

I also ran the UT on devel several times today, and still got some UT failures. I am attaching the logs here:
PosixFileTest_0546.log
PosixFileTest_0504.log
PosixFileTest_0405.log
PosixFileTest_0237.log
PosixFileTest_0209.log
PosixFileTest_0205.log

rlcheng added a commit to rlcheng/fprime that referenced this issue Sep 5, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
2fbcb65 
 * Fix additional issue where randomly generate file name may be
   too large for POSIX.
@rlcheng
Copy link
Collaborator

rlcheng commented Sep 5, 2024

@JohanBertrand thank you for this. I was able to reproduce and able to find the cause and created a PR: #2862

Just out of curiosity were you running on linux? I was not able to reproduce on macOs but was able to on redhat linux.

rlcheng added a commit to rlcheng/fprime that referenced this issue Sep 5, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
c6bcae4 
 * Fix additional issue where randomly generate file name may be
   too large for POSIX.
rlcheng added a commit to rlcheng/fprime that referenced this issue Sep 5, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
2e39bf6 
 * Fix additional issue where randomly generate file name may be
   too large for POSIX.
@JohanBertrand
Copy link
Contributor

@rlcheng Thank you for the fix, it looks like it solves some more random failures. FYI, I am running on Ubuntu 20.04 through WSL..

I think there might be one or two last issues that are rarely happening, related to preallocate and seek. I am attaching the log here. (I extended the length of the random test to catch issues in a faster way, hence the bigger size of the log file)

PosixFileTest_0003.log
PosixFileTest_0007.log
PosixFileTest_0016.log
PosixFileTest_0038.log
PosixFileTest_0042.log

rlcheng added a commit to rlcheng/fprime that referenced this issue Sep 17, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
7d0bb43 
 * Fix preallocate test from picking 0 for length, which causes
   unintentional error in preallocate cmd.
 * Fix seek() in SyntheticFileSystem and assert_file_seek() to allow
   for offset being 0.
LeStarch pushed a commit that referenced this issue Sep 18, 2024
@rlcheng
Fix random UT failures on PosixFile (#2862)
3f8830b 
* Fix Random UT Failure: PosixFile (#2835)
 * Fix UT failing FinalizeCrc test when crc calculation miss match.
   Miss match due to file crc and shadow being out of sync when
   both are not reinialized at the same time.

 * Update File::open in File.ccp to match shadow operation in
   FileRules.cpp to only reset crc when status is OP_OK.

 * Update FileRules.cpp to print out rule name during test.

 * Remove changing file mode in shadow_open() in FileRules.cpp to
   OPEN_NO_MODE when status is not OP_OK.

* Fix Random UT Failure: PosixFile (#2835)
 * Fix additional issue where randomly generate file name may be
   too large for POSIX.

* Fix Random UT Failure: PosixFile (#2835)
 * Fix preallocate test from picking 0 for length, which causes
   unintentional error in preallocate cmd.
 * Fix seek() in SyntheticFileSystem and assert_file_seek() to allow
   for offset being 0.
@JohanBertrand
Copy link
Contributor

@LeStarch Just checking, was this task closed on purpose?
From what I've seen, there are still some rare random failures, as mentioned in my previous comment.

@rlcheng
Copy link
Collaborator

rlcheng commented Sep 18, 2024

@JohanBertrand the issues you saw with preallocate and seek are fixed with the latest commit to this PR.

@JohanBertrand
Copy link
Contributor

@rlcheng Indeed, thank you!

However, it seems like I have now issues with OpenForRead, OpenFileCreate and OpenForWrite.
I am unsure if this is coming from the Docker instance or from the code itself. Is it something you can also observe on your setup?

PosixFileTest_3559.log
PosixFileTest_2875.log
PosixFileTest_2680.log
PosixFileTest_2636.log
PosixFileTest_2247.log
PosixFileTest_2152.log
PosixFileTest_1539.log
PosixFileTest_1008.log
PosixFileTest_0763.log
PosixFileTest_0025.log

@rlcheng
Copy link
Collaborator

rlcheng commented Oct 9, 2024

@JohanBertrand sorry for the late reply, was working on something else. I'm looking at your latest and I am able to reproduce. Looking into this now.

rlcheng added a commit to rlcheng/fprime that referenced this issue Oct 10, 2024
@rlcheng
Fix Random UT Failure: PosixFile (nasa#2835)
1f78ffc 
 * Update get_test_filename() to exclude certain special characters
   and change minimum filename length to prevent posix open() from
   getting an unintentional bad file name.
LeStarch pushed a commit that referenced this issue Oct 10, 2024
@rlcheng
Random UT Failure: PosixFile (#2939)
4bee99e 
* Fix Random UT Failure: PosixFile (#2835)
 * Fix UT failing FinalizeCrc test when crc calculation miss match.
   Miss match due to file crc and shadow being out of sync when
   both are not reinialized at the same time.

 * Update File::open in File.ccp to match shadow operation in
   FileRules.cpp to only reset crc when status is OP_OK.

 * Update FileRules.cpp to print out rule name during test.

 * Remove changing file mode in shadow_open() in FileRules.cpp to
   OPEN_NO_MODE when status is not OP_OK.

* Fix Random UT Failure: PosixFile (#2835)
 * Fix additional issue where randomly generate file name may be
   too large for POSIX.

* Fix Random UT Failure: PosixFile (#2835)
 * Fix preallocate test from picking 0 for length, which causes
   unintentional error in preallocate cmd.
 * Fix seek() in SyntheticFileSystem and assert_file_seek() to allow
   for offset being 0.

* Fix Random UT Failure: PosixFile (#2835)
 * Update get_test_filename() to exclude certain special characters
   and change minimum filename length to prevent posix open() from
   getting an unintentional bad file name.
@rlcheng
Copy link
Collaborator

rlcheng commented Oct 10, 2024

@JohanBertrand the issue you found with OpenForRead, OpenFileCreate and OpenForWrite has been fixed. See #2939 for more info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rlcheng rlcheng

Labels
bug
Projects
None yet
Milestone
Release v3.5.0
Development

No branches or pull requests
4 participants
@LeStarch @JohanBertrand @rlcheng @thomas-bc