Any tips on streaming LLM responses from Phoenix Sockets?

Did you check this article from Sean Moriarity? https://www.hackwithgpt.com/blog/streaming-chatgpt-responses-with-phoenix-liveview/

i saw, but this is not about LV

the main problem i saw was all the manual work he had to do to get streaming response from the LLM. For example if we want to switch to Gemini at some point or a custom model have to repeat same kind of code again

i was evaluating whether it worth the trouble

@kasvith what was a approach / solution have you settled on at the end of the day?

I did something like this in GitHub - ruslandoga/wat · GitHub

Relevant bits:

• liveview kicks off the request in a task that sends full response (so far) back to liveview for rendering as markdown
• stream_answer function
• streaming request payload
• streaming itself is a wrapper around Finch.stream/5 which concats the response

This has definitely been a pain point for us as well. I know it’s asking for a lot but an equivalent of something like AI SDK by Vercel for elixir would be huge. It’s one of the few times where we’ve really been bit by the smaller ecosystem of elixir. Most of the time you can just write it yourself but as @kasvith mentioned its a lot of manual work to implement streaming for each provider.

Hey guys, we moved on with Vercel AI SDK with SSE

Req to chat completions with :into and process send and no other libs required. OpenAI compatible endpoints gives you plug and play with many LLMs, including google’s. You can see langchain here for higher level building blocks and integrations, but it’s shockingly easy to get streaming tokens out to a client wether LiveView or js channels client.

Thanks, Chris. This is fantastic timing because I am researching approaches for building a generalized streaming client in Elixir.

Initial plan was to use a genserver or a supervised Task, then saw your post.

May I ask if you (or anyone) could read this and comment? Rube Goldberg, or good idea?

I’m working on gemini_ex, which will be a comprehensive adapter for Gemini both using Gemini keys and Vertex auth strategies:

https://github.com/nshkrdotcom/gemini_ex/

Today, I added streaming using SSE and a genserver:

https://github.com/nshkrdotcom/gemini_ex/blob/main/STREAMING_ARCHITECTURE.md

Colors for the mermaid diagram were chosen to be in the Elixir style. The implementation is still very much in alpha/unstable phase.

Open to feedback.