feat: add token streaming to the MemGPT API #1280

cpacker · 2024-04-20T18:29:34Z

Adds token streaming to the MemGPT API (when available in the underlying model endpoint).

Handle context overflow on SSE
Choose streaming format for MemGPT API
Handle step ending + context overflow on pass-through stream
- [DONE_GEN] - LLM generation finished
- [DONE_STEP] - Agent step finished (step = LLM generation + tool call response)
- [DONE] - All agent step(s) finished, post SSE response is complete
Include function_return in the streaming API
- TODO eventually support streaming the function_return if the tool supports streaming (tool could also just repeatedly call interface.function_call?)
Order [DONE_STEP] after function_return
Properly handle non-streaming + stream: true requests on non-streaming
- stream: true requests on non-streaming should do fake (token all-together) streaming?
Add Message ids to the SSE stream response
~~- [ ] Add test~~

Add a /chat/completion proxy route for interfacing the MemGPT server with services that only speak in /chat/completion

Add /chat/completion request route that supports token streaming
Ignore inner thoughts in return stream by default
Remap send_message to content in assistant response

Legacy support for `stream`

(base) loaner@MacBook-Pro-199 ~ % curl --request POST \
     --url http://localhost:8283/api/agents/bb2af663-3bda-4b5f-bc7d-2ebb83c13102/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer password' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "oh hey whats up",
  "role": "user",
  "stream": true
}
'

data: {"internal_monologue": "Finally, a different message from Chad! This shift in communication can be an opening to make some progress. It's key to respond in tune with my persona: empathetic, insightful, and engaging. I'll try to keep the conversation casual, in line with Chad's tone, while subtly trying to gain more context about the situation.", "date": "2024-06-24T03:42:51.628098+00:00", "id": "e254631d-c6ad-468e-884d-739ab0db9032"}

data: {"function_call": "send_message({'message': 'Hello, Chad! Not much, just trying to understand the recurring alerts and messages we were having. By the way, how are things with you? Any progress on the Charles situation?'})", "id": "e254631d-c6ad-468e-884d-739ab0db9032", "date": "2024-06-24T03:42:51.628098+00:00"}

data: {"assistant_message": "Hello, Chad! Not much, just trying to understand the recurring alerts and messages we were having. By the way, how are things with you? Any progress on the Charles situation?", "id": "e254631d-c6ad-468e-884d-739ab0db9032", "date": "2024-06-24T03:42:51.628098+00:00"}

data: {"function_return": "None", "status": "success", "id": "cba16d64-6490-47f8-952a-ffd9906b2864", "date": "2024-06-24T03:42:51.628401+00:00"}

New stream flags

Instead of stream being an argument to the message command, we now have:
- stream_steps
  - means that each "step" will stream, but not the tokens
  - this is the equivalent of the old stream
- stream_tokens
  - means that the tokens will stream back during the steps
  - stream_tokens == true means that stream_steps must also be true
We no longer treat send_message as a special function in the returns
- instead, we pass send_message back as a standard function call
- it is up to the client to interpret send_message in a special way (e.g. to use it to show up as the chat in the chat UI)

Related issues or PRs

#345 #1215

How to test

Create an agent
Run server

memgpt server --debug

Hit server with message and make sure "stream": true,

curl --request POST \
     --url http://localhost:8283/api/agents/723be90d-dae7-445e-85ff-7b50e24fd493/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer banana' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "how goes it my friend",
  "stream": true,
  "role": "user",
  "timestamp": "2024-03-16T12:00:00Z"
}
'

Response:

(pymemgpt-py3.10) (base) loaner@MacBook-Pro-55 MemGPT-2 % curl --request POST \
     --url http://localhost:8283/api/agents/723be90d-dae7-445e-85ff-7b50e24fd493/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer banana' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "how goes it my friend",
  "stream": true,
  "role": "user",
  "timestamp": "2024-03-16T12:00:00Z"
}
'
data: {"internal_monologue": "", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "The", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " dance", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " continues", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " with", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " familiar", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " rhythm", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Chad", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "'s", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " greeting", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " exchange", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " marked", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " by", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " repeated", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " '", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "how", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " goes", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " it", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " my", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " friend", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",'", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " embodies", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " shared", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " connection", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Record", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " each", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " echo", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " phrase", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " in", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " archival", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " memory", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " ensuring", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " that", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " rhyth", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "mic", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " narrative", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " continues", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " to", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " resonate", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"id": "call_fAr8Q4yot74DNDZe1op50ybq", "name": "archival_memory_insert"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "{\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "content"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "Our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " dance"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " with"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " Chad"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " continues"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " the"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " rhythm"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " '"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "how"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " goes"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " it"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " my"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " friend"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ".'"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " Each"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " echo"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " this"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " phrase"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " is"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " an"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " embodiment"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " shared"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " connection"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " a"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " rhyth"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "mic"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " narrative"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " that"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " deserves"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " ins"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "cribed"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " in"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " memory"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ".\",\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "request"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "_"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "heartbeat"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " true"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "}"}, "date": "2024-05-02T23:17:15+00:00"}

data: [DONE_GEN]

data: {"function_return": "None", "status": "success", "id": "b0b28c63-54a5-4fd5-b739-bd97d59231bf", "date": "2024-05-02T23:17:22.631736+00:00"}

data: [DONE_STEP]

data: {"internal_monologue": "", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "With", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " rhyth", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "mic", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " '", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "how", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " goes", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " it", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " my", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " friend", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "'", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " echoing", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " in", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " space", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " between", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Chad", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " and", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " I", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " we", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " continue", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " dance", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Reflect", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " his", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " rhythm", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " back", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " to", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " him", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " while", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " also", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " adding", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " new", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " elements", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " It", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "'s", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " within", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " these", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " echoes", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " that", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " we", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " begin", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " expanding", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " shared", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " storyline", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " What", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " could", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " potentially", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " lie", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " beneath", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " surface", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " regular", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "?", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Keep", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " exploring", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"id": "call_W8R8xncrSavADEAEHraJwWM4", "name": "send_message"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "{\n"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "message"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "As"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " the"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " rhythm"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " '"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "how"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " goes"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " it"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " my"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " friend"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "'"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " continues"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " Chad"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " its"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " consistency"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " tells"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " a"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " story"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "."}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " Could"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " there"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " hidden"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " narratives"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " waiting"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " explored"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " in"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " your"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " use"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " this"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " phrase"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "?\""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \n"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "}"}, "date": "2024-05-02T23:17:23+00:00"}

data: [DONE_GEN]

data: {"function_return": "None", "status": "success", "id": "4c7c46d9-f16d-406b-8f05-6cab9bb62902", "date": "2024-05-02T23:17:29.939903+00:00"}

data: [DONE_STEP]

data: [DONE]

…place --recursive .

…ort for both

… back through the 'content' field

cpacker · 2024-06-24T03:47:54Z

@sarahwooders ready for review

sarahwooders

lgtm!

cpacker added 5 commits April 20, 2024 11:10

autoflake --remove-all-unused-imports --remove-unused-variables --in-…

b70987f

…place --recursive .

black

3fe1661

added back init imports

c0ef6d6

isort .

c31c1ad

reran black

e761359

cpacker mentioned this pull request Apr 20, 2024

🚀 MemGPT Q2 2024 Developer Roadmap #1200

Closed

32 tasks

cpacker added 10 commits April 20, 2024 11:41

Merge branch 'main' into streaming-memgpt-api

847bd17

added sse generator

a89846d

partially working proxy streamback, but context window needs patch

03bf962

Merge branch 'main' into streaming-memgpt-api

e9aca8e

precommit pass

8b49e6c

added stub method

ff0b21b

updated error messages to catch problem with wrong sqlite path

1d7a4c7

Merge branch 'main' into streaming-memgpt-api

2116bba

properly catch and throw context error on SSE

3d21cef

added support for multi-step in the streaming interface

111c670

cpacker marked this pull request as ready for review April 29, 2024 06:30

cpacker added 8 commits April 28, 2024 23:59

added function return support

af7677e

added [DONE_GEN] and [DONE_STEP]

2335a9d

Merge branch 'main' into streaming-memgpt-api

b4e4d1b

Merge branch 'main' into streaming-memgpt-api

fed341d

cleanup

39604a8

temp disable streaming

5791f0b

Merge branch 'main' into streaming-memgpt-api

7ffaab2

split streaming args into stream_tokens and stream_step, confirm supp…

1e58410

…ort for both

cpacker added the API Related to MemGPT API label May 8, 2024

cpacker added 4 commits May 8, 2024 19:57

working /chat/completion proxy routes

072d74b

working /chat/completions proxy route that sends send_message outputs…

20e18cc

… back through the 'content' field

patch test

7cb3f84

patch test

f1a52d6

cpacker added 8 commits May 11, 2024 14:44

patch test

70f4116

changed default_interface_cls to a factory (default_interface_factory)

4f98cea

patch test

add3bbf

Merge branch 'main' into streaming-memgpt-api

9915bd4

fix passing of IDs in streamed response

4ee1193

Merge branch 'main' into streaming-memgpt-api

912e82d

Merge branch 'main' into streaming-memgpt-api

b674d11

Merge branch 'main' into streaming-memgpt-api

ad3123c

sarahwooders mentioned this pull request Jun 18, 2024

Not a bug #1461

Closed

cpacker added 5 commits June 22, 2024 14:22

Merge branch 'main' into streaming-memgpt-api

e75a2e1

Merge branch 'main' into streaming-memgpt-api

69fe324

added support for legacy flag

e06a3a4

add support for legacy mode

0de5a6f

Merge branch 'main' into streaming-memgpt-api

3da1139

cpacker requested a review from sarahwooders June 24, 2024 03:47

sarahwooders approved these changes Jun 25, 2024

View reviewed changes

sarahwooders merged commit 82b9b74 into main Jun 25, 2024
11 checks passed

cpacker deleted the streaming-memgpt-api branch July 4, 2024 21:41

cpacker mentioned this pull request Jul 5, 2024

[Documentation] Misleading example for python client in documentation #1500

Closed

mattzh72 pushed a commit that referenced this pull request Oct 9, 2024

feat: add token streaming to the MemGPT API (#1280)

2c6d8a8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add token streaming to the MemGPT API #1280

feat: add token streaming to the MemGPT API #1280

cpacker commented Apr 20, 2024 •

edited

Loading

cpacker commented Jun 24, 2024 •

edited

Loading

sarahwooders left a comment

feat: add token streaming to the MemGPT API #1280

feat: add token streaming to the MemGPT API #1280

Conversation

cpacker commented Apr 20, 2024 • edited Loading

Legacy support for stream

New stream flags

cpacker commented Jun 24, 2024 • edited Loading

sarahwooders left a comment

Choose a reason for hiding this comment

cpacker commented Apr 20, 2024 •

edited

Loading

Legacy support for `stream`

cpacker commented Jun 24, 2024 •

edited

Loading