lud

lud

Hello,

I’m evaluating this library and I have two questions:

  • is it possible to integrate JSON schema validation with a custom validator (we want to use JSV to automatically validate and cast output to structs, using $ref schemas for various entities)
  • is it possible to enforce not using environment variables like OPENAI_API_KEY that may be defined.

Thank you

0
Post #11
aaronrussell

aaronrussell OP

Hi, thanks for your interest.

  • JSON schema validation - as it stands right now, we validate against the JSON schema using an admittedly cobbled together solution. It does a decent enough job for basic shapes, but won’t cast to structs or resolve refs. That said, there’s room for improvement here. I previously looked at a few JSON-schema packages to see if we could just offload the problem to a dependency, but none that I looked at fitted they way I wanted. But, I didn’t look at JSV, and honestly it looks perfect. So if it’s of interest, I’d be more than happy to bake JSV in as the way we validate against schemas. I’ll dig a bit deeper in the week, but potentially that could be done this week.

  • Env variables - the auth method resolves like this:

    1. If the caller provides an API key, use that
    2. If the provider has an API key in the app config, use that
    3. If the provider environment variable is set, use that
    4. Fail

    So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?

1
Post #12
mikehostetler

mikehostetler

Congrats OP!!! ReqLLM author here - any time you’d like to chat about this problem space I’m happy to grab time.

Only suggestion would be to consider collaborating on LLMDB: llm_db | Hex

Nailing the model database and model capabilities turned out to a huge challenge and that DB is a place where it makes more sense to work together if you’re open!

Great work.

PS

ReqLLM stands alone - but Ash and Jido build on top of it extensively.

1
Post #13
lud

lud

Hello,

Thank you for your answer!

If you want to integrate with JSV you may want to use JSV.Schema.normalize_collect to build a self contained schema with $defs and $refs that can be shared to an external service. This would work nicely with schema defined as modules (any module that exports a json_schema/0 function):

use JSV.Schema

defschema NextAction,
  task: string()

defschema MeetingRecap,
  summary: string(),
  next_action: NextAction

JSV.Schema.normalize_collect(MeetingRecap, as_root: true)

#=> %{
#   "$defs" => %{
#     "NextAction" => %{
#       "properties" => %{"task" => %{"type" => "string"}},
#       "required" => ["task"],
#       "title" => "NextAction",
#       "type" => "object",
#       "x-jsv-cast" => "Elixir.NextAction"
#     }
#   },
#   "properties" => %{
#     "next_action" => %{"$ref" => "#/$defs/NextAction"},
#     "summary" => %{"type" => "string"}
#   },
#   "required" => ["summary", "next_action"],
#   "title" => "MeetingRecap",
#   "type" => "object",
#   "x-jsv-cast" => "Elixir.MeetingRecap"
# }

Also please test with the main branch, an incoming release with breaking changes is on its way, it’s the last one before v1.0, I just did not have time to polish the docs.

If you don’t want to bake in, an option that accepts the output and returns {:ok, casted_value} | {:error, _} could be nice too.

So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?

Yeah this is OK. The problem we have is we define an API key per-feature, but we have python apps that can only read the global env variables. So on dev machines it is a problem, but option 1 is ok for production code, thanks :slight_smile:

0
Post #14
aaronrussell

aaronrussell OP

I worked on this a little yesterday so can tell you where I’ve landed.

Have decided not to integrate JSV directly. Instead sticking with our simple schema builder and validation code which is “good enough” 90% of the time. And then on top of that we provide a simple behaviour module for pluggable schema validation - so can integrate with JSV, Zoi or whatever. For JSV it would look a little like this:

defmodule JSVAdapter do
  @behaviour Omni.Schema.Adapter

  def to_schema(%JSV.Root{raw: raw}), do: raw

  def validate(%JSV.Root{} = root, input) do
    case JSV.validate(input, root) do
      {:ok, data} -> {:ok, data}
      # error pathway should return the error as a string
      {:error, error} -> {:error, format_error(error)}
    end
  end
end

Then in your call site you use it like this:

root = JSV.build!(MeetingRecap)
Omni.stream_text(model, context, output: {JSVAdapter, root})

And if needed, you can use the same pattern in tools:

defmodule MeetingTool do
  use Omni.Tool

  @root JSV.build!(MeetingRecap)
  def schema, do: {JSVAdapter, @root}

  # .... etc
end

If you want to play with this it’s already in the main branch - so will be in the next release.

1
Post #15
lud

lud

I think the schema needs to be collected to have proper refs when using modules but otherwise looks good.

Is there a way to use an external schema with the built-in schema system?

0
Post #16
aaronrussell

aaronrussell OP

The :output option ordinarily receives a plain map that can serialise to a JSON schema. You can use the Omni.Schema builders if you want, or build it with some other tool. Then after the request the output is validated against the schema - the default implementation offloads this to Peri, using Peri.from_json_schema. I don’t know how well that will validate against complex schemas with $defs etc. Maybe it will be fine. But if not, then reach for an adapter

1
Post #17
lud

lud

I can’t find this on the main branch. Maybe you did not push it yet?

0
Post #18
aaronrussell

aaronrussell OP

Looks sheepish :see_no_evil_monkey: :sweat_smile:

So sorry. It’s there now.

0
Post #19
lud

lud

Thank you :slight_smile:

0
Post #20