Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run test assemblies in parallel #1106

Merged
merged 61 commits into from
Jan 14, 2021
Merged

Run test assemblies in parallel #1106

merged 61 commits into from
Jan 14, 2021

Conversation

russcam
Copy link
Contributor

@russcam russcam commented Dec 30, 2020

This PR

  1. separates out the common test components from Elastic.Apm.Tests into a new assembly, Elastic.Apm.Tests.Utilities. This allows assemblies containing tests to be run in parallel, which is not possible when test assemblies reference another test assembly that may potentially be running at the same time.
  2. adds coverlet and JunitXml.TestLogger packages to all test projects using Directory.Build.Props. This removes the need to add them in CI scripts or to each project individually.
  3. updates linux CI scripts to run tests in the solution file. In conjunction with the xunit configuration, this allows test assemblies to run in parallel.
  4. Renames Elastic.Apm.DockerTests and Elastic.ApmPerfTests to Elastic.Apm.Docker.Tests and Elastic.Apm.Perf.Tests, respectively, to align the test assembly naming convention.
  5. Excludes all test assemblies from being included in code coverage output

This commit separates out the common test components
from Elastic.Apm.Tests into a new assembly,
Elastic.Apm.Tests.Utilities.

This allows assemblies containing tests
to be run in parallel, which is not possible when test assemblies
reference another test assembly that may potentially be running
at the same time.
This commit adds coverlet and JunitXml.TestLogger packages to
all test projects using Directory.Build.Props. This removes the
need to add them in CI scripts or to each project individually.
This commit updates linux CI scripts to run tests
in the solution file. In conjunction with the
xunit configuration, this allows test assemblies to run
in parallel.

Rename coverage.cobertura.xml files to include
the directory name.
to Elastic.Apm.Docker.Tests
to Elastic.Apm.Perf.Tests for consistency
@codecov-io
Copy link

codecov-io commented Dec 30, 2020

Codecov Report

Merging #1106 (c939702) into master (209a19b) will increase coverage by 0.48%.
The diff coverage is 62.16%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1106      +/-   ##
==========================================
+ Coverage   78.92%   79.40%   +0.48%     
==========================================
  Files         152      160       +8     
  Lines        7520     6535     -985     
==========================================
- Hits         5935     5189     -746     
+ Misses       1585     1346     -239     
Impacted Files Coverage Δ
src/Elastic.Apm.AspNetCore/ApmMiddleware.cs 96.29% <ø> (-0.77%) ⬇️
...lastic.Apm.AspNetCore/Extensions/ListExtensions.cs 100.00% <ø> (ø)
...tic.Apm.AspNetCore/Extensions/RequestExtentions.cs 89.83% <ø> (-0.17%) ⬇️
src/Elastic.Apm/Agent.cs 75.00% <ø> (+4.26%) ⬆️
src/Elastic.Apm/Api/DistributedTracingData.cs 94.11% <ø> (-0.33%) ⬇️
src/Elastic.Apm/Api/Http.cs 82.50% <ø> (-1.43%) ⬇️
src/Elastic.Apm/Api/SpanContext.cs 46.15% <0.00%> (-13.85%) ⬇️
src/Elastic.Apm/DistributedTracing/TraceContext.cs 71.42% <ø> (-1.95%) ⬇️
src/Elastic.Apm/Helpers/TimeExtensions.cs 100.00% <ø> (ø)
...rics/MetricsProvider/FreeAndTotalMemoryProvider.cs 65.95% <ø> (-7.38%) ⬇️
... and 146 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c716415...c939702. Read the comment docs.

@apmmachine
Copy link
Contributor

apmmachine commented Dec 30, 2020

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #1106 updated

    • Start Time: 2021-01-13T23:52:53.548+0000
  • Duration: 58 min 29 sec

  • Commit: ae985d9

Test stats 🧪

Test Results
Failed 0
Passed 16423
Skipped 59
Total 16482

Steps errors 1

Expand to view the steps failures

Shell Script
  • Took 0 min 2 sec . View more details on here
  • Description: .ci/linux/benchmark.sh

Log output

Expand to view the last 100 lines of log output

[2021-01-14T00:51:11.415Z]       Executed action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer), returned result Microsoft.AspNetCore.Mvc.OkObjectResult in 3.3578ms.
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
[2021-01-14T00:51:11.415Z]       Executing ObjectResult, writing value of type 'System.String'.
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.415Z]       Executed action Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) in 3.6178ms
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
[2021-01-14T00:51:11.415Z]       Request finished in 3.8899ms 200 text/plain; charset=utf-8
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
[2021-01-14T00:51:11.415Z]       Request starting HTTP/1.1 POST http://localhost:24566/intake/v2/events application/x-ndjson; charset=utf-8 3337
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[3]
[2021-01-14T00:51:11.415Z]       Route matched with {action = "Post", controller = "IntakeV2Events"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Post() on controller Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController (Elastic.Apm.Tests.MockApmServer).
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
[2021-01-14T00:51:11.415Z]       Executing action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) - Validation state: Valid
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.415Z]       Executed action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer), returned result Microsoft.AspNetCore.Mvc.OkObjectResult in 2.86ms.
[2021-01-14T00:51:11.415Z] info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
[2021-01-14T00:51:11.415Z]       Executing ObjectResult, writing value of type 'System.String'.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) in 3.1961ms
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
[2021-01-14T00:51:11.416Z]       Request finished in 3.4862ms 200 text/plain; charset=utf-8
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
[2021-01-14T00:51:11.416Z]       Request starting HTTP/1.1 POST http://localhost:24566/intake/v2/events application/x-ndjson; charset=utf-8 3337
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[3]
[2021-01-14T00:51:11.416Z]       Route matched with {action = "Post", controller = "IntakeV2Events"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Post() on controller Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController (Elastic.Apm.Tests.MockApmServer).
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
[2021-01-14T00:51:11.416Z]       Executing action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) - Validation state: Valid
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer), returned result Microsoft.AspNetCore.Mvc.OkObjectResult in 2.368ms.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
[2021-01-14T00:51:11.416Z]       Executing ObjectResult, writing value of type 'System.String'.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) in 2.6256ms
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
[2021-01-14T00:51:11.416Z]       Request finished in 2.8905ms 200 text/plain; charset=utf-8
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
[2021-01-14T00:51:11.416Z]       Request starting HTTP/1.1 POST http://localhost:24566/intake/v2/events application/x-ndjson; charset=utf-8 2360
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[3]
[2021-01-14T00:51:11.416Z]       Route matched with {action = "Post", controller = "IntakeV2Events"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Post() on controller Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController (Elastic.Apm.Tests.MockApmServer).
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
[2021-01-14T00:51:11.416Z]       Executing action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) - Validation state: Valid
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action method Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer), returned result Microsoft.AspNetCore.Mvc.OkObjectResult in 4.3148ms.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
[2021-01-14T00:51:11.416Z]       Executing ObjectResult, writing value of type 'System.String'.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action Elastic.Apm.Tests.MockApmServer.Controllers.IntakeV2EventsController.Post (Elastic.Apm.Tests.MockApmServer) in 4.6016ms
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
[2021-01-14T00:51:11.416Z]       Request finished in 4.8605ms 200 text/plain; charset=utf-8
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
[2021-01-14T00:51:11.416Z]       Request starting HTTP/1.1 GET http://localhost:24566/config/v1/agents?service.name=AspNetFullFramework_Tests_CentralConfigTests-CustomServiceName&service.environment=AspNetFullFramework_Tests_CentralConfigTests-CustomEnvironment  
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[3]
[2021-01-14T00:51:11.416Z]       Route matched with {action = "Get", controller = "AgentsConfig"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get() on controller Elastic.Apm.Tests.MockApmServer.Controllers.AgentsConfigController (Elastic.Apm.Tests.MockApmServer).
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
[2021-01-14T00:51:11.416Z]       Executing action method Elastic.Apm.Tests.MockApmServer.Controllers.AgentsConfigController.Get (Elastic.Apm.Tests.MockApmServer) - Validation state: Valid
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action method Elastic.Apm.Tests.MockApmServer.Controllers.AgentsConfigController.Get (Elastic.Apm.Tests.MockApmServer), returned result Microsoft.AspNetCore.Mvc.StatusCodeResult in 0.1383ms.
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.StatusCodeResult[1]
[2021-01-14T00:51:11.416Z]       Executing HttpStatusCodeResult, setting HTTP status code 304
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
[2021-01-14T00:51:11.416Z]       Executed action Elastic.Apm.Tests.MockApmServer.Controllers.AgentsConfigController.Get (Elastic.Apm.Tests.MockApmServer) in 0.2977ms
[2021-01-14T00:51:11.416Z] info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
[2021-01-14T00:51:11.416Z]       Request finished in 0.6398ms 304 
[2021-01-14T00:51:12.791Z] Elastic APM .NET Tests> 99> Console> [2021-01-14 00:51:12.649 +00:00][Info] - {LoggingTestBase} Finished test: Elastic.Apm.AspNetFullFramework.Tests.CentralConfigTests+MaxSpansAndSampleRateTests.MaxSpans_invalid_value
[2021-01-14T00:51:12.791Z] [xUnit.net 00:16:56.59]   Finished:    Elastic.Apm.AspNetFullFramework.Tests
[2021-01-14T00:51:13.051Z]   Passed Elastic.Apm.AspNetFullFramework.Tests.CentralConfigTests+MaxSpansAndSampleRateTests.MaxSpans_invalid_value [9 s]
[2021-01-14T00:51:13.051Z] 
[2021-01-14T00:51:13.051Z] JunitXML Logger - Results File: C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1106\apm-agent-dotnet\test\Elastic.Apm.AspNetFullFramework.Tests\junit-NETCoreApp21-Elastic.Apm.AspNetFullFramework.Tests.xml
[2021-01-14T00:51:13.051Z] 
[2021-01-14T00:51:13.051Z] Test Run Successful.
[2021-01-14T00:51:13.051Z] Total tests: 114
[2021-01-14T00:51:13.051Z]      Passed: 114
[2021-01-14T00:51:13.051Z]  Total time: 16.9826 Minutes
[2021-01-14T00:51:13.051Z] 
[2021-01-14T00:51:13.051Z] Build succeeded.
[2021-01-14T00:51:13.051Z]     0 Warning(s)
[2021-01-14T00:51:13.051Z]     0 Error(s)
[2021-01-14T00:51:13.051Z] 
[2021-01-14T00:51:13.051Z] Time Elapsed 00:16:59.89
[2021-01-14T00:51:14.255Z] Post stage
[2021-01-14T00:51:14.264Z] Running in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1106\apm-agent-dotnet
[2021-01-14T00:51:14.286Z] Archiving artifacts
[2021-01-14T00:51:14.793Z] Recording test results
[2021-01-14T00:51:16.686Z] Post stage
[2021-01-14T00:51:16.701Z] [WS-CLEANUP] Deleting project workspace...
[2021-01-14T00:51:16.701Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[2021-01-14T00:51:21.182Z] [WS-CLEANUP] done
[2021-01-14T00:51:21.392Z] Stage "Release to feedz.io" skipped due to when conditional
[2021-01-14T00:51:21.443Z] Stage "Release" skipped due to when conditional
[2021-01-14T00:51:21.464Z] Stage "Release" skipped due to when conditional
[2021-01-14T00:51:21.514Z] Stage "Release" skipped due to when conditional
[2021-01-14T00:51:21.599Z] Stage "AfterRelease" skipped due to when conditional
[2021-01-14T00:51:21.616Z] Stage "AfterRelease" skipped due to when conditional
[2021-01-14T00:51:21.928Z] Running on Jenkins in /var/lib/jenkins/workspace/net_apm-agent-dotnet-mbp_PR-1106
[2021-01-14T00:51:21.980Z] [INFO] getVaultSecret: Getting secrets
[2021-01-14T00:51:22.296Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-01-14T00:51:22.992Z] + chmod 755 generate-build-data.sh
[2021-01-14T00:51:22.992Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1106/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1106/runs/40 UNSTABLE 3509181
[2021-01-14T00:51:23.543Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1106/runs/40/steps/?limit=10000 -o steps-info.json
[2021-01-14T00:51:24.094Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-dotnet/apm-agent-dotnet-mbp/PR-1106/runs/40/tests/?status=FAILED -o tests-errors.json

@russcam
Copy link
Contributor Author

russcam commented Dec 30, 2020

jenkins run tests

@russcam
Copy link
Contributor Author

russcam commented Jan 5, 2021

run tests

When running tests in release configuration, don't allow
a test method to be inlined, which may cause test assertions
to fail.
And nodereuse:false from linux tests
@russcam russcam marked this pull request as ready for review January 7, 2021 11:42
@russcam
Copy link
Contributor Author

russcam commented Jan 7, 2021

This is now ready for review @gregkalapos.

The benchmark test fails currently as the the host does not have the net50 SDK installed, and the log switchable test looks like a timing condition, which I'll see if I can address.

@russcam russcam requested a review from gregkalapos January 7, 2021 11:44
.ci/linux/benchmark.sh Outdated Show resolved Hide resolved
This was referenced Jan 13, 2021
May be more than one session starting with the
session prefix, so assert on the session name
created in the test.
Jenkins cannot display Sourcelinked
source code
@gregkalapos gregkalapos self-requested a review January 13, 2021 19:12
Copy link
Contributor

@gregkalapos gregkalapos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants