Streaming LLM responses from Phoenix Sockets

Chat / Discussions
, ,
1

Hi,

We are trying to develop a LLM chat agent. So far lot of libraries have been written in JS or Python. But Elixir provides an excellent WebSocket support

So we decided to use Elixir Phoenix Sockets to stream responses from LLMs for Chats

Has anyone done this? what are the libraries you used? Any drawbacks on this approach?

Will SSE would be a better option than phoenix sockets?

1 Like
2

Did you check this article from Sean Moriarity? https://www.hackwithgpt.com/blog/streaming-chatgpt-responses-with-phoenix-liveview/

1 Like
3

i saw, but this is not about LV

the main problem i saw was all the manual work he had to do to get streaming response from the LLM. For example if we want to switch to Gemini at some point or a custom model have to repeat same kind of code again

i was evaluating whether it worth the trouble