Deepseek-R1 decendants and Nemo guardrail compatibility

[YaphetKG]

I was trying to use nemo with deepseek-distilled llama , and these models always start with <think> blah blah </think> actual resposne

after some debugging i noticed that nemo-guardrails makes calls to the llm providing max token =3 which captures this opening think tag and the rail blocks further flow .

Question is is there a way to customize this , maybe having a shim to parse the LLM response... or something to extend the token size etc..

[Pouyanpi]

Hi @YaphetKG, yes it is possible to set max_tokens in the configuration, In the respective prompt of the task (e.g. self check output prompt, add max_tokens)

For example have a look at content safety prompts.yml

If your issue persists please let me know, it could be that the task you are using does not support max_tokens in that case let me know and we can open an issue.