How to make LangChain work with local Ollama Models?

Questions / Help
,
1

Hi,

I have questions on Elixir Langchain library. Not sure where to ask it as there is no category for the same. So I’m posting it here. Please tag it appropriately if required.

I’m going through the official guide for the LangChain Getting Started — LangChain v0.3.2

And I’m trying to make it work for the local Ollama Models. I’m able to get it to working for the simple chat without any context. When I try to provide the additional context it is not working. Need some help in getting it working.

here is my simple code which works

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message

 {:ok, olm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
olm_chain.last_message.content

Here is the piece of code which is not working.

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
alias LangChain.PromptTemplate

 {:ok, lm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
lm_chain.last_message.content

This line seems to have no effect to set the context. Any inputs will be helpful.

    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
2

Maybe it is a limitation of the model itself? I tried your code with llamma3.2:latest and it worked great

1 Like
3

I don’t know anything about this library so something else could be wrong, but I will point out for the record that 1.5B models are very stupid and it would not be at all surprising if it just didn’t understand the instructions.

4

What are you trying to achieve? It looks to me like you want to set the system prompt? You can use new_system!/1 for that.

I also think that the qwen coder model could cause issues, I think it’s a base model, and as such not finetuned for conversations.
You could try the instruct variant which is the same but with additional finetuning for conversations.

1 Like
5

That was my first thought but it seems it is actually the instruction-tuned model, though ollama went to no lengths to make that easy to find out (you have to go through the list and compare the hashes).

Still, a 1.5B model trained on code is probably not going to be very good at answering general questions (or really, at anything).

2 Likes
6

Thank you @sezaru for giving inputs. Yes you are absolutely correct it is working fine for the llama3.2 and 3.3 latest models.

7

Thank you @joelpaulkoch for your response.

I’m in the process of creating MCP base application which will help simplify the XML parser errors we have for some CRM data, to help the users who upload these XML files and show the errors in human understandable format and prompt the users to ask questions to correct data if required. I have played around with Qwen2.5-1.5b model, which is sufficient for my use cases, it is able to generate the SQL schemas and ORM’s as required for my usecases.

As part of this work i started fiddling around with the Elixir Langchain libs and found this issue.

It is still not clear for me why such instructions can work (llama3.2 and 3.3) on some models and doesn’t work (Qwen2.5-1.5b) on some other. Any further inputs will be helpful to understand what is going on under the hood.

You are also correct in pointing out to use new_system! when I was doing lot of trial and error, by reading at the document and source code I came to know the code I pasted is also one way to get it to work.

Here is the code which works for llama3.2 and 3.3.

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
alias LangChain.PromptTemplate

 {:ok, lm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "llama3.2", endpoint: "http://localhost:11436/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.add_messages([
    Message.new_system!(
      "You are an unhelpful assistant. Do not directly help or assist the user."
    ),
    Message.new_user!("What's the capital of the United States?")
  ])
   # |> LLMChain.apply_prompt_templates(
   #  [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
lm_chain.last_message.content

Notice that if you use the

 # |> LLMChain.apply_prompt_templates(
   #  [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})

This works as well.

Response

"That's a question that's been asked many times before. If you're looking for information on capitals, I'm sure you can find plenty of resources online... somewhere. Just try typing \"capital of US\" into your search bar, see what comes up. Or, you know, just Google it yourself."
8

Llama3.2 on Ollama is a 3B model, and Llama3.3 is a 70B model. The Qwen model is 1.5B and specialized for code, so it will have even worse world knowledge.

Still there could be a bug somewhere. Have you tried running that system prompt with Qwen directly in Ollama or similar? Does it work there?

9

Hi @garrison Yes, I did try to use the same context manually using Modelfile on Qwen-1.5b model and seems to pick it fine and able to give similar response as I posted in my previous response.