Hello,
I’m evaluating this library and I have two questions:
Thank you
Hi, thanks for your interest.
JSON schema validation - as it stands right now, we validate against the JSON schema using an admittedly cobbled together solution. It does a decent enough job for basic shapes, but won’t cast to structs or resolve refs. That said, there’s room for improvement here. I previously looked at a few JSON-schema packages to see if we could just offload the problem to a dependency, but none that I looked at fitted they way I wanted. But, I didn’t look at JSV, and honestly it looks perfect. So if it’s of interest, I’d be more than happy to bake JSV in as the way we validate against schemas. I’ll dig a bit deeper in the week, but potentially that could be done this week.
Env variables - the auth method resolves like this:
So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?
Congrats OP!!! ReqLLM author here - any time you’d like to chat about this problem space I’m happy to grab time.
Only suggestion would be to consider collaborating on LLMDB: llm_db | Hex
Nailing the model database and model capabilities turned out to a huge challenge and that DB is a place where it makes more sense to work together if you’re open!
Great work.
PS
ReqLLM stands alone - but Ash and Jido build on top of it extensively.
Hello,
Thank you for your answer!
If you want to integrate with JSV you may want to use JSV.Schema.normalize_collect to build a self contained schema with $defs and $refs that can be shared to an external service. This would work nicely with schema defined as modules (any module that exports a json_schema/0 function):
use JSV.Schema
defschema NextAction,
task: string()
defschema MeetingRecap,
summary: string(),
next_action: NextAction
JSV.Schema.normalize_collect(MeetingRecap, as_root: true)
#=> %{
# "$defs" => %{
# "NextAction" => %{
# "properties" => %{"task" => %{"type" => "string"}},
# "required" => ["task"],
# "title" => "NextAction",
# "type" => "object",
# "x-jsv-cast" => "Elixir.NextAction"
# }
# },
# "properties" => %{
# "next_action" => %{"$ref" => "#/$defs/NextAction"},
# "summary" => %{"type" => "string"}
# },
# "required" => ["summary", "next_action"],
# "title" => "MeetingRecap",
# "type" => "object",
# "x-jsv-cast" => "Elixir.MeetingRecap"
# }
Also please test with the main branch, an incoming release with breaking changes is on its way, it’s the last one before v1.0, I just did not have time to polish the docs.
If you don’t want to bake in, an option that accepts the output and returns {:ok, casted_value} | {:error, _} could be nice too.
So, in theory if your code provides keys in 1 or 2, then it will never drill down to 3. Is that good enough or do you need a way to explicitly remove 3 from the flow?
Yeah this is OK. The problem we have is we define an API key per-feature, but we have python apps that can only read the global env variables. So on dev machines it is a problem, but option 1 is ok for production code, thanks 
I worked on this a little yesterday so can tell you where I’ve landed.
Have decided not to integrate JSV directly. Instead sticking with our simple schema builder and validation code which is “good enough” 90% of the time. And then on top of that we provide a simple behaviour module for pluggable schema validation - so can integrate with JSV, Zoi or whatever. For JSV it would look a little like this:
defmodule JSVAdapter do
@behaviour Omni.Schema.Adapter
def to_schema(%JSV.Root{raw: raw}), do: raw
def validate(%JSV.Root{} = root, input) do
case JSV.validate(input, root) do
{:ok, data} -> {:ok, data}
# error pathway should return the error as a string
{:error, error} -> {:error, format_error(error)}
end
end
end
Then in your call site you use it like this:
root = JSV.build!(MeetingRecap)
Omni.stream_text(model, context, output: {JSVAdapter, root})
And if needed, you can use the same pattern in tools:
defmodule MeetingTool do
use Omni.Tool
@root JSV.build!(MeetingRecap)
def schema, do: {JSVAdapter, @root}
# .... etc
end
If you want to play with this it’s already in the main branch - so will be in the next release.
I think the schema needs to be collected to have proper refs when using modules but otherwise looks good.
Is there a way to use an external schema with the built-in schema system?
The :output option ordinarily receives a plain map that can serialise to a JSON schema. You can use the Omni.Schema builders if you want, or build it with some other tool. Then after the request the output is validated against the schema - the default implementation offloads this to Peri, using Peri.from_json_schema. I don’t know how well that will validate against complex schemas with $defs etc. Maybe it will be fine. But if not, then reach for an adapter
