`:any` type for embedded schemas in non-virtual fields

Questions / Help
, ,
1

Hey there!

We are using Ecto to parse some API response (it works pretty well!).
We want this response to be converted to a struct, so we first do all the needed validations and at the end we do an apply_action(changeset, :parse).
To do so, we would like to have a module that defines an embedded_schema like the following (we won’t save this):

defmodule MyApp.Response do
  use Ecto.Schema

  @primary_key false
  embedded_schema do
    field(:custom_field, :any)
    ...
  end
end

The “problem” comes with this :any type. When compiling, Ecto complains that only virtual fields can be defined with the :any type.
Here the error:

== Compilation error in file lib/my_app/api/response.ex ==
** (ArgumentError) only virtual fields can have type :any, invalid type for field :custom_field
    (ecto 3.11.1) lib/ecto/schema.ex:2012: Ecto.Schema.__field__/4
    lib/my_app/api/response.ex:36: (module)
    (ecto 3.11.1) lib/ecto/schema.ex:2241: Ecto.Schema.__embeds_module__/4
    lib/my_app/api/response.ex:32: (module)

Is there an explanation of why they must be virtual?

In our use case, this field can have different types (boolean, integer, string, etc.), so we need this flexibility here.

We think that, as embedded_schemas are just saved as blobs or, like in our case, not saved at all, this doesn’t make a lot of sense. But maybe someone has a better understanding about this.

Thanks in advance! :purple_heart:

2

I’m assuming this is for similar reasons as to why you can’t use nil for embeds_many—which came up again recently—that embeds are meant to be treated the same as regular relations. I think your only option if you want to keep using Ecto is to create your own type. Otherwise you could check out Drops which is a more general schema library.

1 Like
3

You can use a schemaless changeset

{%MyStruct{}, %{custom_field: :any}}
|> Ecto.Changeset.cast(%{"custom_field" => "1"}, [:custom_field])
|> Ecto.Changeset.apply_changes()
3 Likes
4

But now actually answering that part, and expanding on what @sodapopcan . Understand that althought schemas and changesets are just mapping abstractions. both schema and embedded schemas are expected to be at some point dumped to a database.
Since that is not your intention, i’d recommend to use schemaless changeset(you can use that with structs or plain simple maps).
in case you need features like nesting, relations and some more complex stuff that doesn’t work with schemaless changeset i’d do something like:

@default [virtual: :true]
schema "" do
  field :custom_field1, :any, @default
  field :custom_field2, :any, @default
  ...
end
1 Like
5

Thank you all for answering here!!
@cevado yes… Actually this variable will be inside an assoc of the main struct
I will try then to find a better way to check these types :pray:
Maybe Drop is a nice candidate! Thx @sodapopcan !!

1 Like
6

What happens if you pass virtual: true to the field call like the error message suggests?

7

With :any ecto cannot be sure it can encode the data for storage in whatever db driver is in use. Therefore it is only allowed for virtual fields, which are not persisted anyways and therefore do not run into that problem.

1 Like
8

Ecto can work for some inputs I found I quickly outgrew it and wrote GitHub - Adzz/data_schema: Declarative schemas for data transformations. too. If you need to map data that isn’t essentially a map or a list then ecto schemas aren’t the solution imo.

2 Likes