Any tips on streaming LLM responses from Phoenix Sockets?

[kasvith]

Hi,

We are trying to develop a LLM chat agent. So far lot of libraries have been written in JS or Python. But Elixir provides an excellent WebSocket support

So we decided to use Elixir Phoenix Sockets to stream responses from LLMs for Chats

Has anyone done this? what are the libraries you used? Any drawbacks on this approach?

Will SSE would be a better option than phoenix sockets?

[ndrean]

Did you check this article from Sean Moriarity? https://www.hackwithgpt.com/blog/streaming-chatgpt-responses-with-phoenix-liveview/

[kasvith]

i saw, but this is not about LV

the main problem i saw was all the manual work he had to do to get streaming response from the LLM. For example if we want to switch to Gemini at some point or a custom model have to repeat same kind of code again

i was evaluating whether it worth the trouble

[gmile]

@kasvith what was a approach / solution have you settled on at the end of the day?

[ruslandoga]

I did something like this in GitHub - ruslandoga/wat

Relevant bits:

• liveview kicks off the request in a task that sends full response (so far) back to liveview for rendering as markdown
• stream_answer function
• streaming request payload
• streaming itself is a wrapper around Finch.stream/5 which concats the response

[avi-turetsky]

This has definitely been a pain point for us as well. I know it’s asking for a lot but an equivalent of something like AI SDK by Vercel for elixir would be huge. It’s one of the few times where we’ve really been bit by the smaller ecosystem of elixir. Most of the time you can just write it yourself but as @kasvith mentioned its a lot of manual work to implement streaming for each provider.