CoreMark®-PRO Python Plugin

This plugin runs the CoreMark®-PRO certify-all workloads and returns the results in a machine-readable format.

Build the container

The plugin is intended to be run as a container. The provided Dockerfile will handle the build parameters.

podman build . -t arcaflow-plugin-coremark-pro

Run the benchmarks

Tuning the iterations

In order to adjust the run time for each of the benchmarks to a target number of seconds, each benchmark must have its iterations adjusted individually. The plugin provides a tune-iterations step to calculate the target number of iterations as appropriate for the target system under test. The tune-iterations step accepts a parameter target_run_time as integer seconds (default is 10).

NOTE -- If you are going to pass the output of the tune-iterations step to the certify-all step in a workflow, you should include in the input to this step all of the parameters for the certify-all step so that the output object generated is complete.

cat inputs/tune-iterations-example.yaml | podman run -i arcaflow-plugin-coremark-pro -f -

Running certify-all

The required input parameters are contexts and workers as integers. An optional object parameter iterations may be provided to set the number of iterations for each benchmark individually.

cat inputs/certify-all-example.yaml | podman run -i arcaflow-plugin-coremark-pro -f -

Or as a one-liner:

echo '{"contexts": 1, "workers": 1}' | podman run -i arcaflow-plugin-coremark-pro -f -

Sample workflow

A sample Arcaflow workflow file is provided to demonstrate how to connect the output of the tune-iterations step to the input of the certify-all step. This will require an Arcaflow engine binary version of v0.17.0-beta2 or greater.

cd sample_workflow
arcaflow -config config.yaml -input input.yaml -context .

Autogenerated Input/Output Documentation by Arcaflow-Docsgen Below

Certify All (certify-all)

Runs all of the nine tests, collects their output scores, and processes them to generate the final CoreMark-PRO score

Input

Type:	scope
Root object:	CertifyAllInput
Properties	contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int iterations (reference[Iterations]) Name: Benchmark Iterations Description: Number of iterations for each benchmark Required: No Type: reference[Iterations] Referenced object: Iterations verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int
Objects CertifyAllInput (object) Type: object Properties contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int iterations (reference[Iterations]) Name: Benchmark Iterations Description: Number of iterations for each benchmark Required: No Type: reference[Iterations] Referenced object: Iterations verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int Iterations (object) Type: object Properties cjpeg-rose7-preset (int) Name: cjpeg-rose7-preset iterations Description: Iterations for cjpeg-rose7-preset benchmark Required: No Type: int core (int) Name: core iterations Description: Iterations for core benchmark Required: No Type: int linear_alg-mid-100x100-sp (int) Name: linear_alg-mid-100x100-sp iterations Description: Iterations for linear_alg-mid-100x100-sp benchmark Required: No Type: int loops-all-mid-10k-sp (int) Name: loops-all-mid-10k-sp iterations Description: Iterations for loops-all-mid-10k-sp benchmark Required: No Type: int nnet_test (int) Name: nnet_test iterations Description: Iterations for nnet_test benchmark Required: No Type: int parser-125k (int) Name: parser-125k iterations Description: Iterations for parser-125k benchmark Required: No Type: int radix2-big-64k (int) Name: radix2-big-64k iterations Description: Iterations for radix2-big-64k benchmark Required: No Type: int sha-test (int) Name: sha-test iterations Description: Iterations for sha-test benchmark Required: No Type: int zip-test (int) Name: zip-test iterations Description: Iterations for zip-test benchmark Required: No Type: int

Outputs

error

Type:	scope
Root object:	ErrorOutput
Properties	error (string) Required: Yes Type: string
Objects ErrorOutput (object) Type: object Properties error (string) Required: Yes Type: string

success

Type:	scope
Root object:	SuccessOutput
Properties	coremark_pro_command (string) Name: CoreMark®-PRO command line Description: The complete command line passed to CoreMark®-PRO Required: Yes Type: string coremark_pro_params (reference[CertifyAllParams]) Name: Test Params Description: The parameters supplied to the CoreMark®-PRO benchmarks Required: Yes Type: reference[CertifyAllParams] Referenced object: CertifyAllParams coremark_pro_results (reference[CertifyAllResult]) Name: Test Results Description: Results of the CoreMark®-PRO benchmarks Required: Yes Type: reference[CertifyAllResult] Referenced object: CertifyAllResult
Objects CertifyAllItem (object) Type: object Properties Iterations (int) Name: Iterations Description: Number of benchmark iterations Required: No Type: int MultiCore (float) Name: MultiCore Description: Benchmark result - Multi-Core (iter/s) Required: Yes Type: float Scaling (float) Name: Scaling Description: Benchmark result - Scaling Required: Yes Type: float SingleCore (float) Name: SingleCore Description: Benchmark result - Single-Core (iter/s) Required: Yes Type: float CertifyAllParams (object) Type: object Properties contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int CertifyAllResult (object) Type: object Properties CoreMark-PRO (reference[CertifyAllItem]) Name: CoreMark-PRO Description: CoreMark-PRO benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem cjpeg-rose7-preset (reference[CertifyAllItem]) Name: cjpeg-rose7-preset Description: cjpeg-rose7-preset benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem core (reference[CertifyAllItem]) Name: core Description: core benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem linear_alg-mid-100x100-sp (reference[CertifyAllItem]) Name: linear_alg-mid-100x100-sp Description: linear_alg-mid-100x100-sp benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem loops-all-mid-10k-sp (reference[CertifyAllItem]) Name: loops-all-mid-10k-sp Description: loops-all-mid-10k-sp benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem nnet_test (reference[CertifyAllItem]) Name: nnet_test Description: nnet_test benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem parser-125k (reference[CertifyAllItem]) Name: parser-125k Description: parser-125k benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem radix2-big-64k (reference[CertifyAllItem]) Name: radix2-big-64k Description: radix2-big-64k benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem sha-test (reference[CertifyAllItem]) Name: sha-test Description: sha-test benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem zip-test (reference[CertifyAllItem]) Name: zip-test Description: zip-test benchmark results Required: Yes Type: reference[CertifyAllItem] Referenced object: CertifyAllItem SuccessOutput (object) Type: object Properties coremark_pro_command (string) Name: CoreMark®-PRO command line Description: The complete command line passed to CoreMark®-PRO Required: Yes Type: string coremark_pro_params (reference[CertifyAllParams]) Name: Test Params Description: The parameters supplied to the CoreMark®-PRO benchmarks Required: Yes Type: reference[CertifyAllParams] Referenced object: CertifyAllParams coremark_pro_results (reference[CertifyAllResult]) Name: Test Results Description: Results of the CoreMark®-PRO benchmarks Required: Yes Type: reference[CertifyAllResult] Referenced object: CertifyAllResult

Tune Iterations (tune-iterations)

Runs all of the nine tests, checks their run times, calculates the number of iterations for each test to roughly reach the 'target_runtime', and returns an object compatible with the 'certify-all' step. NOTE -- If you are going to pass the output of the 'tune-iterations' step to the 'certify-all' step in a workflow, you should include in the input to this step all of the parameters for the 'certify-all' step so that the output object generated is complete.

Input

Type:	scope
Root object:	TuneIterationsInput
Properties	contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int target_run_time (int) Name: Target Run Time Seconds Description: Target run time in seconds for each benchmark Required: No Default (JSON encoded): 10 Type: int verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int
Objects TuneIterationsInput (object) Type: object Properties contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int target_run_time (int) Name: Target Run Time Seconds Description: Target run time in seconds for each benchmark Required: No Default (JSON encoded): 10 Type: int verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int

Outputs

error

Type:	scope
Root object:	ErrorOutput
Properties	error (string) Required: Yes Type: string
Objects ErrorOutput (object) Type: object Properties error (string) Required: Yes Type: string

success

Type:	scope
Root object:	CertifyAllInput
Properties	contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int iterations (reference[Iterations]) Name: Benchmark Iterations Description: Number of iterations for each benchmark Required: No Type: reference[Iterations] Referenced object: Iterations verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int
Objects CertifyAllInput (object) Type: object Properties contexts (int) Name: Contexts Description: Number of contexts per benchmark Required: No Type: int iterations (reference[Iterations]) Name: Benchmark Iterations Description: Number of iterations for each benchmark Required: No Type: reference[Iterations] Referenced object: Iterations verify (bool) Name: Verify Description: Enable benchmark validation runs Required: No Default (JSON encoded): false Type: bool workers (int) Name: Workers Description: Number of workers per benchmark Required: No Type: int Iterations (object) Type: object Properties cjpeg-rose7-preset (int) Name: cjpeg-rose7-preset iterations Description: Iterations for cjpeg-rose7-preset benchmark Required: No Type: int core (int) Name: core iterations Description: Iterations for core benchmark Required: No Type: int linear_alg-mid-100x100-sp (int) Name: linear_alg-mid-100x100-sp iterations Description: Iterations for linear_alg-mid-100x100-sp benchmark Required: No Type: int loops-all-mid-10k-sp (int) Name: loops-all-mid-10k-sp iterations Description: Iterations for loops-all-mid-10k-sp benchmark Required: No Type: int nnet_test (int) Name: nnet_test iterations Description: Iterations for nnet_test benchmark Required: No Type: int parser-125k (int) Name: parser-125k iterations Description: Iterations for parser-125k benchmark Required: No Type: int radix2-big-64k (int) Name: radix2-big-64k iterations Description: Iterations for radix2-big-64k benchmark Required: No Type: int sha-test (int) Name: sha-test iterations Description: Iterations for sha-test benchmark Required: No Type: int zip-test (int) Name: zip-test iterations Description: Iterations for zip-test benchmark Required: No Type: int