Consumer batching

[vk496]

👋

I want to create a distributed AI model inference solver. For performance reasons, is better to use GPU 1 time for 20 requests rather than 20 times each request.

How can I accomplish that with Taskiq?

Thank you in advance

[s3rius]

Hi and thanks for looking into taskiq for such a task. I'm not sure about the details, but as long as you have a function to call, you can just call kiq 20 times and it should start 20 tasks asyncronously.

Although heavy computations require CPU and therefore it might be useful to increase number of worker processes (that handle incoming tasks) by specifying -w parameter to desired number.

If you provide more details I can help you a bit more.

[vk496]

I'm aware that Taskiq handles it asynchronously or with threadpool (nice!). However in this case, is that behaviour exactly the issue. Launching 100 async tasks doesn't make the AI model do the inference 100 times faster (in CPU). However, when a GPU is used, we can do batch solving and solve 100 in one shot.

To illustrate a little bit more my issue, I have a function like this:

def detect_objects(images_bytes: list[bytes]) -> list[str]:
  pass

I can pass n images and it will resolve with which animal/object is in that image. And currently the only way to implement it with taskiq I can find is like this:

@broker.task
def t_detect_objects(
    image: bytes
) -> str:
    result: list[str] = detect_objects(images_bytes=[image])
    return result[0]

instead of doing multiple images at the same time.

PS: Probably I can adapt the code and performance tests would show that the difference is very small (or not!), but batching is a feature that nobody mentioned, so I wanted to ask it here :)