Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Realtime API support #3722

Draft
wants to merge 21 commits into
base: master
Choose a base branch
from
Draft

feat: Realtime API support #3722

wants to merge 21 commits into from
+1,265 −5

Conversation

mudler
Copy link
Owner

@mudler mudler commented Oct 3, 2024 •
edited
Loading

Description

This PR fixes #3714

And also covers #191

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@netlify Netlify
Copy link

netlify bot commented Oct 3, 2024 •
edited
Loading

❌ Deploy Preview for localai failed.

Name Link
🔨 Latest commit 333c16d
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/674103da043fa3000830fdaf

@mudler mudler mentioned this pull request Oct 3, 2024
Open
@mudler mudler force-pushed the feat/realtime branch 3 times, most recently from 5435a07 to 2db3f3d Compare October 9, 2024 10:57
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 9, 2024

yamllint Failed

Show Output 
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery

@mudler
Copy link
Owner Author

mudler commented Oct 14, 2024

Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59

github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 16, 2024
View reviewed changes
core/http/app.go Fixed Show fixed Hide fixed
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 1c61aad to 22579bc Compare October 18, 2024 14:06
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 2886390 to deff060 Compare October 30, 2024 15:10
@mudler mudler changed the title WIP: realtime API stub feat: Realtime API support Oct 31, 2024
@mudler mudler force-pushed the feat/realtime branch 4 times, most recently from 7a9d6e8 to ee4ae33 Compare November 6, 2024 17:35
@mattkanwisher
Copy link
Contributor

whats best option here if we want to contribute just make forks of the branch and PRS against this?

@mudler
Copy link
Owner Author

mudler commented Nov 7, 2024

whats best option here if we want to contribute just make forks of the branch and PRS against this?

Yes, that would work just fine!

@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 60579fb to 4034562 Compare November 7, 2024 22:54
@mudler mudler mentioned this pull request Nov 8, 2024
Merged
1 task
@mudler mudler force-pushed the feat/realtime branch 6 times, most recently from 818122a to a1da931 Compare November 12, 2024 18:12
@mudler
Copy link
Owner Author

mudler commented Nov 14, 2024

mh. things are in the good direction but still VAD isn't right, it detects the start of the conversation, but can't detect the end segment yet.

@mudler mudler mentioned this pull request Nov 20, 2024
Merged
1 task
@mudler
Copy link
Owner Author

mudler commented Nov 20, 2024

Extracted silero-vad bits over here: #4204 so can be tackled separately

@mudler mudler mentioned this pull request Nov 20, 2024
Merged
1 task
@mudler mudler force-pushed the feat/realtime branch 2 times, most recently from 4cc5bd0 to fe51882 Compare November 21, 2024 14:22
@mudler
WIP
64b2687 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
WIP
acbd62e 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
debug
e3de76d 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
aujdio
44a9b75 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Add model interface to sessions
fcde474 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
setup ws upgrade
634ea15 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Fix route
3969ddb 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
chore: simplify passing options to ModelOptions
2caa50a 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Load wrapper clients
374a667 
Testing with:

```yaml
name: gpt-4o
pipeline:
 tts: voice-it-riccardo_fasol-x-low
 transcription: whisper-base-q5_1
 llm: llama-3.2-1b-instruct:q4_k_m
```

Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
refactor
2504697 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
feat: correctly detect when starting the vad server
f2c3469 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
chore: extract realtime models into two categories
940285f 
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.

Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
feat(vad): hook vad detection
e620169 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
6d50182 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Attach context for VAD
e907aad 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Move to debug calls
f9a463e 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
Fix lock handling
a257835 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
wip(vad)
5b7da59 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
gRPC client stubs
ce4f251 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
wip
771df88 
Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler
WIP
333c16d 
Signed-off-by: Ettore Di Giacinto <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for realtime API
2 participants
@mudler @mattkanwisher