the integration test failed with timeout sometimes

[sio4]

The integration test failed with timeout sometimes (or maybe always now?). Any chance to reduce the testing time?

Example:

• panic: test timed out after 10m0s
• cli/internal/cmd/fix_test.TestFix_v0_16_27.func2

The sequence of testing is:

1. testhelpers.InstallOldBuffaloCMD(t, "v0.16.27")
2. exec.Command("buffalo", tc.newargs...)
3. testhelpers.RunBuffaloCMD(t, []string{"fix", "-y"})
4. testhelpers.RunBuffaloCMD(t, []string{"build"})

Issues:

• the first step takes time to build the old binary. could be replaced with install binary distribution
• the second line could work, but we cannot guarantee the generated app has really the old dependencies
• during the new app generation, node-related compilation happens and it takes a long time, and also could fail
• also the node modules are not a test target for buffalo fix

I would like to replace steps 1 and 2 with downloading a pre-configured app tree. Will try to fix it. (but not sure if this is a good timing or I need to wait until the new generation)

https://github.com/gobuffalo/cli/runs/7427376418?check_suite_focus=true#step:4:1886

but the timeout with panic itself may not be directly related to the sequence.

https://github.com/gobuffalo/cli/runs/7427376418?check_suite_focus=true#step:4:2888

[sio4]

Hi @paganotoni,

JFYI

After we discussed the usage of the oldapp repository, I tried organizing a new structure for collecting old versions of generated apps and the result is temporarily located at https://github.com/sio4/coke.

After that, I found that one of our integration tests fails while preparing the old app tree by "installing the old version of buffalo, run buffalo new (and node related things inside the step)" in some conditions. I think we can change this sequence with something like "run git clone .../coke then run git checkout tag-for-target-version".

However, I think we can postpone this discussion, decision, and change to the time after your current work for the new generation.

[paganotoni]

Hi Sio, Thanks for pointing this out.

I'm hoping to grow our testing and improve it with the new approach for the CLI. I might be putting too much pressure into that release but I think we will be able to test closer to the way our users do.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment. Otherwise, this will be closed in 7 days.

[paganotoni]

@sio4 now that we have discussed and decided to abandon the other CLI approach and centered on this one I think we should find ways to do better integration testing of the CLI. Lets find a way to make these more stable.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment. Otherwise, this will be closed in 7 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment. Otherwise, this will be closed in 7 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment. Otherwise, this will be closed in 7 days.