fix: Make it so the configured max_batch_size is respected when batching inference requests together

[RShang97]

What does this PR address?

Address issue #3710 by narrowing types passed to functions so we can access the batch size if a client pre batches their input. Also raise an error if a single batch submitted by the client exceeds the max_batch_size. This lets us accurately batch requests such that the batch is less than the configured max_batch_size in the main dispatcher loop.

I didn't touch any of the logic that dealt with inferring the queue size.

Testing

in Bentoml/examples/quickstart created a configuration.yaml

# configuration.yaml
runners:
  batching:
    max_batch_size: 5

started the server with the following command

BENTOML_CONFIG=./configuration.yaml b serve --production

added custom logging to sklearn.py.

def add_runnable_method(method_name: str, options: ModelSignature):
        def _run(
            self: SklearnRunnable, input_data: ext.NpNDArray | ext.PdDataFrame
        ) -> ext.NpNDArray:
            print(f"\n\nReceived input size:{len(input_data)}\n\n")
            # TODO: set inner_max_num_threads and n_jobs param here base on strategy env vars
            with parallel_backend(backend="loky"):
                return getattr(self.model, method_name)(input_data)

Swarmed the server with the locust module included in the examples/quickstart directory

Saw that the print statements never returned anything greater than 5 and that exceptions were raised when batches that were large than max_batch_size were returned e.g.
output.txt

Fixes #3710

Before submitting:

• Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's
  guide on how to create a pull request.
• Does the code follow BentoML's code style, both make format and make lint script have passed (instructions)?
• Did you read through contribution guidelines and follow development guidelines?
• Did your changes require updates to the documentation? Have you updated
  those accordingly? Here are documentation guidelines and tips on writting docs.
• Did you write tests to cover your changes?

[RShang97]

cc @ssheng @sauyon @bojiang

I don't think I have permission to add specific reviewers so flagging here

[sauyon]

Just needs a format; I believe make format should do the trick!

[sauyon]

LGTM! 🎉