Unacceptable high latency

[fanqiehc]

I'm using the Flyte sandbox on a PC have 20 cores and 32G memory.
I create a simple workflow with 2 nodes, and 1000 executions of it.
It takes about 40min before all workflows are in SUCCEEDED status.
Even the pods are all completed, the workflow is still RUNNING status for a long time.

Is the latency normal?

Durations:

Workflow code:

@task
def multiple(name: str, count: int) -> str:
    return name * count

@task
def say_hello(name: str) -> str:
    return f"hello world, {name}"

@workflow
def my_wf(name: str, count: int) -> str:
    n = multiple(name=name, count=count)
    res = say_hello(name=n)
    return res

[yindia]

cc: @kumare3 @EngHabu

[kumare3]

The default sandbox engine is not tweaked for lower latency

[kumare3]

A few tweaks should be done according to the docs to the config- https://docs.flyte.org/en/latest/deployment/cluster_config/performance.html#deployment-cluster-config-performance.

I like the setting as a perf test framework