Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When use tool set to True every time it response "I apologize, I was unable to find the answer to your question. Is there anything else I can help with?" #26

Open
shrimad-mishra-cognoai opened this issue Aug 2, 2023 · 10 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@shrimad-mishra-cognoai
Copy link

shrimad-mishra-cognoai commented Aug 2, 2023

When use tool set to True every time it response "I apologize, I was unable to find the answer to your question. Is there anything else I can help with?"

@filip-michalsky filip-michalsky added the bug Something isn't working label Aug 2, 2023
@filip-michalsky
Copy link
Owner

@shrimad-mishra-cognoai what LLM are you using? Did you change the system prompt from the example?

This is the response from the parser when the LLM does not return the correct format. This is a known issue, see here:
https://github.com/filip-michalsky/SalesGPT/blob/main/salesgpt/parsers.py#L28

There is outstanding TODO to make the parser more robust, which means some API changes, logic changes, LLM changes and prompt changes, or all of the above. Anyone would like to help?

@filip-michalsky filip-michalsky added the help wanted Extra attention is needed label Aug 2, 2023
@filip-michalsky
Copy link
Owner

@shrimad-mishra-cognoai when you run python run.py without any changes whatsoever, does the example work for you?

@filip-michalsky
Copy link
Owner

@shrimad-mishra-cognoai can you confirm the bug still happens? We changed the type of the use_tools flag to string since agent config loading from JSON is string.

@RizkiNoor16
Copy link

I have same issue when attempting to customize the stages and the prompt for my specific use case. Additionally, the parser doesn't seem to parse correctly as it consistently outputs the message 'I apologize, I was unable to find the answer to your question. Is there anything else I can assist you with?

@filip-michalsky
Copy link
Owner

@RizkiNoor16 thanks for the feedback, we will prioritize this on the roadmap as it will definitely make SalesGPT more usable.

Few observations for myself or anyone who wants to help via a PR:

  • The parser breaks for the most part when the underlying LLM does not follow the desired generation format.
    Thats a function of 2 things:
    A) The LLM is not able to reason well enough to follow the format (Thought / Action / Observation) and hence we can try to use a different LLM better suited for the task or fine-tune
    B) The prompt itself is not good enough to make the LLM to follow the desired pattern.

People are actively working on this and I am sure there are people who make their parsers for Agents more reliable now. I find this super interesting and will look into it as soon as I can. I believe OpenAI functions which output JSON 100% of the time should be able to improve this significantly.

@ranjandsingh
Copy link

hey @RizkiNoor16 so one thing you can also do for agent parser is use pydantic here to define your exact output this works quite well

@RizkiNoor16
Copy link

thank for your feedback @ranjandsingh ,Mind if I ask which model you're using? I've tried the code provided in the documentation, and it doesn't perform very well with the GPT-3.5 Turbo model.

@ranjandsingh
Copy link

Works great with 3.5-turbo-313
New turbo is nerfed man

@RizkiNoor16
Copy link

In my case, I have use the new GPT-3.5 instruct, and this has resolved my issue."

@dandnm1992
Copy link

dandnm1992 commented Oct 5, 2023

@filip-michalsky
I found several problems with the parser and fixed it with this code:

def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
        logger.info(self.ai_prefix)
        logger.info(self.verbose)

        if self.verbose:
            logger.info("TEXT")
            logger.info(text)
            logger.info("-------")
        if f"{self.ai_prefix}: " in text or "<END_OF_TURN>" in text or "Action:" not in text:
            return AgentFinish({"output": text.split(f"{self.ai_prefix}:")[-1].strip()}, text)
        
        regex = r"Action: (.*?)[\n]*Action Input: (.*)"
        match = re.search(regex, text)
        if not match:
            # TODO - this is not entirely reliable, sometimes results in an error.
            return AgentFinish(
                {"output": self.apologize_message},
                text,
            )
            # raise OutputParserException(f"Could not parse LLM output: `{text}`")
        action = match.group(1)
        action_input = match.group(2)
        return AgentAction(action.strip(), action_input.strip(" ").strip('"'), text)

But I still have a problem when a truncated command gets into the parser, for example like this http://joxi.ru/Vm6LqXPHKpO7zA

Just a 'thought' without 'action' and 'action_input'

how can this happen? Any ideas?

P.S Im not english speaker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants