Santhosh-KS

Santhosh-KS

How to make LangChain work with local Ollama Models?

Hi,

I have questions on Elixir Langchain library. Not sure where to ask it as there is no category for the same. So I’m posting it here. Please tag it appropriately if required.

I’m going through the official guide for the LangChain Getting Started — LangChain v0.8.14

And I’m trying to make it work for the local Ollama Models. I’m able to get it to working for the simple chat without any context. When I try to provide the additional context it is not working. Need some help in getting it working.

here is my simple code which works

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message

 {:ok, olm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
olm_chain.last_message.content

Here is the piece of code which is not working.

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
alias LangChain.PromptTemplate

 {:ok, lm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
lm_chain.last_message.content

This line seems to have no effect to set the context. Any inputs will be helpful.

    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})

Most Liked

garrison

garrison

That was my first thought but it seems it is actually the instruction-tuned model, though ollama went to no lengths to make that easy to find out (you have to go through the list and compare the hashes).

Still, a 1.5B model trained on code is probably not going to be very good at answering general questions (or really, at anything).

sezaru

sezaru

Maybe it is a limitation of the model itself? I tried your code with llamma3.2:latest and it worked great

joelpaulkoch

joelpaulkoch

What are you trying to achieve? It looks to me like you want to set the system prompt? You can use new_system!/1 for that.

I also think that the qwen coder model could cause issues, I think it’s a base model, and as such not finetuned for conversations.
You could try the instruct variant which is the same but with additional finetuning for conversations.

Last Post!

Santhosh-KS

Santhosh-KS

Hi @garrison Yes, I did try to use the same context manually using Modelfile on Qwen-1.5b model and seems to pick it fine and able to give similar response as I posted in my previous response.

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

Other popular topics Top

grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

We're in Beta

About us Mission Statement