Still catching up with the new features.
I just published v0.4.2
)
Anybody built on top of the new assistants/threads API yet? What’s the sequence of calls you should be making to get the threads use case working? It’s not linear anymore as it requires you to periodically check if a run is complete. OpenAI can now suggest you to make multiple function_calls and these can be executed in parallel. Is using a GenServer for every thread that keeps track of all this state and acts as a bridge between UI ↔ Phoenix App ↔ OpenAI a good idea?
I’ve been hearing good things about the zephyr model created by HF, and was curious whether something that capable could be run locally on my machine.
since there are multiple apps that proxy the openai api to other models (including local models), this seems to be quite straightforward. the only thing that needs to be changed is to parameterize the base_url.
I have done this now with the function OpenaiEx.with_base_url/2 to parameterize the api base URL. I tested it against the llama.cpp-python library and the non-streaming versions of the completion and chat-completion apis worked on the first try. the streaming versions seem to be causing some difficulty still. i will try the streaming calls against another proxy when i get a chance, to check if it’s a bug in the proxy.
my current dev machine is a mac with 8GB and it turns out that’s not enough to run a 7b model as well as a docker dev container (and assorted other apps). i suspect 16 Gb would work.
if anyone wants to try local LLMs via an OpenAI API proxy, please give the library a whirl and let me know what you think.
i have not published this to hex as yet, so the mix entry has to point to the github main branch for now.
I’m looking for feedback on potential new features for the library. Please upvote on the following issues if you think they should be implemented.
Thanks for the work on this Library. I have a couple questions.
openai_ex is a good fit for handling requests in a production app instead of just LiveBooks?finch to handle rate limiting issues?
@arton sorry for the delay, I’m not on the forum every day, so I didn’t realize you had asked the questions. Here are my 2 cents.
I just released v0.5.0 adding OpenaiEx.with_base_url to enable other LLMs via an OpenAI API proxy. AFAIK this only really applies to completion and chat completion endpoints.
Enjoy. Please let me know if you come across any issues.
It turns out that there is a bug in the streaming chat completion implementation when used with some 3rd party OpenAI proxies to non-OpenAI models.
Basically the SSE implementation in the library is not completely compliant with (what I assume to be) the SSE spec. Ideally this should be solved at the Finch/Mint level, rather than left to the JSON API wrappers, but if this is an important use case, I’ll tackle it in the library on a priority basis.
Please respond on this thread (or on the github bug) if this is important for you.
To be clear, this does not impact usage with the actual OpenAI API.
I have released v0.5.1 with fixes for a bug that affected streaming openai proxies to other models (think litellm to ollama, etc.).
Thanks to github user @Madd0g for reporting the bug and providing a fix.
If you’re using openai_ex to access non-open ai models via a proxy, please use this version and let me know of any other changes that you might want.
I have released v0.5.2. After another request for increasing the timeout, this time to 10 minutes, I decided to make the timeout configurable using OpenaiEx.with_receive_timeout/1. The default timeout stays at 2 minutes, although it can probably be reduced now.
I also removed the documentation for the Completion API which is now legacy. The OpenAI models that use it are deprecated and will be removed in Jan 2024.
I have not removed the actual API because there might still be proxied LLMs (enabled by the recent OpenaiEx.with_base_url/1 configuration) that continue to use this endpoint for a little longer.
I have documented the configurable options (receive timeout and base url) in the User Guide.