Spectral - Type-driven JSON encoding/decoding, validation, and OpenAPI generation

Maybe combine the struct type and defstruct into one (with a macro)?

It’s been a while since I developed my deftypestruct and I’ve seen someone posting a library doing something very similar here like a week or two ago.

ex (my lib creates {module_name}.t(), but can be made to use a different type name too):

defmodule Person do
  deftypestruct %{
    name: String.t(),
    age: non_neg_integer() | nil,
    role: role()
  }
end

I also check against nil so unless explicitly permitted (like the age field above) the lib raises.

You might be interested in taking a look at estructura allowing transparent nesting, coercion, validation, and (!) generation for stream_data property-based testing out of the box.

The deftypestruct approach is elegant. As your library generates the type and put it in the beam it works well with spectral

Thanks for pointing out the nil handling in spectral, you can find more info in the spectral docs nil section.

Hi, good library, I’ve read the code and it has a lot of very strange approaches and things

• It uses a spectra erlang library, which uses caching in persistent_term for type-specs. But why do you need to introduce lazy-loading of the type-information (which would certainly introduce random spikes in runtime performance), when you can just create an encoder/decoder for a specific type during compilation? Just introduce a macro.

• Library says

  Spectral provides type-safe data serialization and deserialization for Elixir types. Currently the focus is on JSON.

  but it also provides features of OpenAPI spec generations, including endpoint documentation, etc. So its not JSON codec generation, it is a code-first OpenAPI integration.

• If you’re doing OpenAPI, there is a problem that OpenAPI JSON schema is not “compatible” with the elixir type specs. I mean that JSON Schema can define types which can’t be expressed in elixir type specs. And I don’t see any way to integrate JSON Schema definitions into elixir type specs.

• It uses spectra, which encodes record tuples as maps and there is no option to specify codecs for generic tuples as far as I can see. I tried naive approach with custom codec for :erlang.tuple/0 type and it didn’t work.

• It fails to work in iex, because it uses abstract_code chunk received with :code.get_object_code, which returns only for modules which are present in a code path. There must be an option to accept module binary

• Code cache is not invalidated on recompilation, which would make development with this library a hell

I found some more issues, so feel free to contact me so I can perform a more detailed review (for a reasonable price of course)

Wow, had a quick look at estructura, nice work! Worth noting is that spectral doesn’t do coercion of integers that are encoded as strings in json. Eg, for the type:

@type my_int :: integer()

Spectral will return error for the value “1”, but will work for 1. There is currently no option to do such coercion, but it can be added.
{:error, …} = Spectral.decode(~s(“1”), MyModule, :my_int, :json)
{:ok, 1} = Spectral.decode(~s(1), MyModule, :my_int, :json)

Internally in spectra (the erlang library that does most of the heavy lifting), there is some support for property-based testing, but it is not ready for general use.

Apart from this I think we are aligned in features?

More or less, yep. I am not sure if spectral works with nested structs as in Estructura.User — estructura v1.13.0

Ah, and estructura allows calculated fields.

It looks nice, but does it generate typespecs under the hood (for those types declared there with atoms)?

It’s on my todo-list since the day 1, but I never needed it myself and therefore this not-so-complicated mapping is not yet there. ATM, it does generate stubs.

Hi Asd,

I started out gathering up all type information at compile time (as you suggest), but when you reference types from modules that don’t use the Spectral macro, you still need to figure out the types from those modules somehow. The cache (which is turned off by default, so it shouldn’t have bothered you when developing) is documented here: Configuration If you are doing hot code reloading in production, then I can expose primitives for cleaning the cache.

In the documentation there is an example that defines a “point type”, which is a 2-tuple. Imho, there is no good general way to convert tuples to JSON, when you have such a type you have to create your own codec: Custom codecs. My hope is that you should be able to create elixir types that map to most things OpenAPI has support for. With the codec “escape hatch” you should be able to do whatever you want

It is correct that the types currently need to be expressed in a beam file that is compiled with debug_info, this is a current limitation of the library. If this is hindering developers from using the library, then I think it can be solved.

I’m happy to hear about those other issues.

All the best

Yes, exactly, you can call the ensure_compiled? on the module and then extract the types. It is possible that module is not compiled and cyclic references are possible too, that’s why the best approach would be to use the compiler or a compile tracer for the problem.

I’d solve it like this: compile tracer collects info about every type spec definition, and then generates a module like Spectral.Generated.<Application name> which exposes a function get_type_info(Module) -> TypeInfo. Then functions like encode and decode just use this module.

But it is all hacks. Metaprogramming in Elixir and Erlang is purposefully limited to block any attempts to change the compilation of one module, based on the contents of another, aside function calls and macros.

Dont get me wrong, but one get_object_code is a bunch of disk IO operations. And one type info gathering may be result in multiple get_object_code calls, because one type can reference another and so on. Now imagine that your code now is guaranteed to take 2 seconds to just encode the response, at least once (with type info caching) or every time (without caching).