perf(webapi): Increase maxWorkers limit to 10000

[pingsutw]

Tracking issue

NA

What changes were proposed in this pull request?

Increase maxWorkers limit to 10000 to run more agent tasks in parallel.

How was this patch tested?

Remote cluster

Setup process

from flytekit import workflow, task, LaunchPlan
from flytekitplugins.dummy_agent import Sleep


@task(task_config=Sleep(duration=0.5))
def sleep_task() -> str:
    return "Hello World!"


@workflow()
def sleep_wf():
    for i in range(100):
        sleep_task()


@workflow
def load_test_wf():
    sleep_lp = LaunchPlan.get_or_create(
        name="fixed_inputs", workflow=sleep_wf, max_parallelism=100
    )
    for i in range(50):
        sleep_lp()

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.08%. Comparing base (0a1c82c) to head (7e14b69).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5326      +/-   ##
==========================================
+ Coverage   61.07%   61.08%   +0.01%     
==========================================
  Files         794      794              
  Lines       51203    51203              
==========================================
+ Hits        31271    31277       +6     
+ Misses      17049    17043       -6     
  Partials     2883     2883              

Flag	Coverage Δ
unittests-datacatalog	69.31% <ø> (ø)
unittests-flyteadmin	58.82% <ø> (+0.04%)	⬆️
unittests-flytecopilot	17.79% <ø> (ø)
unittests-flytectl	68.30% <ø> (ø)
unittests-flyteidl	79.30% <ø> (ø)
unittests-flyteplugins	61.94% <ø> (ø)
unittests-flytepropeller	57.32% <ø> (ø)
unittests-flytestdlib	65.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kumare3]

What's the impact on memory

[pingsutw]

Memory usage increases by ~4% (325MB). Propeller has 8G memory.

[Future-Outlier]

Memory usage increases by ~4% (325MB). Propeller has 8G memory.

Love it