Are Ecto embedded schemas useful without a database?

Questions / Help
,
1

I am still new to elixir, so once in a while getting confused with the reasoning for basic stuff

Context
I am building a small service that is essentially a nice looking wrapper to an external API. It is somewhat similar to reading users, their organizations, orders, products via API and then presenting these nicely with whichever cool filters client wants (e.g. something like "find all products this user is expressed interest in).

I don’t even have any database, login happens via external system, everything is read and written from/to external API and several LiveViews + small contexts on top of Tesla-based REST client work very well there.

Data modeling
Building something without database is not very usual situation for me, so I naturally started with Ecto’s embedded schemas in style of:

defmodule MyCoolApp.User do
...
  embedded_schema do 
    field :first_name, :string
    field :last_name, :string
...
  end
end

But I noticed that I am not really using Ecto’s capabilities much. I don’t need query building (or I don’t understand how to convert them into REST params for Tesla conveniently), there are no real repos that could make transactions possible - I use these essentially as just a Map with properties.

Now I’ve got to a point where I’d want to link different objects (temporarily, just one live view execution probably) and with a simple map I could do things like linked_user = %{user | organization: org} while with embedded_schema I need to learn and use things like build_assoc and other association related things.

What’s the point of Ecto in databaseless app?
Once in a while I see mentions on the forum that embedded_schemas work well without the database, but can’t quite get a reasoning why.

  • I am certainly missing something basic, so could you please, suggest some reasoning or at least pros and cons of living with Ecto schemas and without when there’s no real database behind?
  • Or am I wrong with the whole approach and something very different is better to be tried?
2 Likes
2

One of my friends once told me that he wants to make every struct in his project an Ecto schema, no matter whether it maps to a database table or column.

I think he has his point. By making a struct an Ecto schema, we have a very good casting and validation mechanism for free.

Maybe I’ll try this idea in my future projects, too.

2 Likes
3

By the way, I’m facing some very serious problems with embedded schema. Because they map to JSONB columns, I lost the type information of my structs.

So recently I went for the custom struct + Ecto type + Erlang serialization approach.

4

Have you looked at Polymorphic embed?
https://hexdocs.pm/polymorphic_embed/readme.html

5

Yep, I use it a lot, but still, when the data is highly dynamic and recursive, and you don’t know what will appear where at which time, polymorphic embed does not help.

6

No, it does not :slightly_smiling_face:

7

The project I’m working on has such a data structure like this:

@type val() :: 
  nil |
  number() |
  String.t() |
  boolean() |
  Date.t() |
  DateTime.t() |
  [val()] |
  %CustomStruct1{} |
  %CustomStruct2{} |
  %Foo{val: val()}

If Foo needs to be an Ecto embedded schema, how do you code it?

In the end, I just made my own Ecto.Type that maps such data as a binary using :erlang.term_to_binary/2.

8

Short answer to your opening question: Yes, I believe they can be.

I’m using regular Ecto.Schema for representing database data, but I do not use those same Ecto.Schema structs for driving the user interface. I represent form data using Ecto embedded schemas and their related Changeset processing functions to represent UI/External API data (“UI data” collectively herein) for the purposes of having a ready made validation framework of that data.

More to the point, I don’t want a tight coupling between UI data and the database model because in many cases what works the best for the database structure isn’t necessarily what works the best for user interactions with that data. From the software design side, I want those separate interests to be clearly independent; this way I’m encouraging form design and form data handling to do what’s best for the user interaction in question and the database related code to do what is best for the information architecture. Yes, there is a point where these two interests must come together, but that translation from UI to data oriented business logic can be handled in a discrete, clear way independent of the direct UI and database related concerns.

In a situation like yours, my sense is you only have one of the two concerns that my project has: dealing with the user interface. So it seems to me is the real question you need to answer is: “Does the Ecto.Changeset methodology, a ready-made, well-known, widely-used system for data validation, bring enough value to the data validation problem to make it a worthwhile pursuit?” For many (maybe most), the answer should probably be “yes”; but I’m guessing on that last point. Otherwise, you need to think what alternative is better/simpler/closer fit for your project.

5 Likes
9

Ecto schemas are essentially DTOs… You use them to move data from one place to the other (including but not exclusive to databases). The advantage you have over maps is being able to define that information in a more type-safe/ structured way.

IMHO, the first thing that you need to consider before using an Ecto schema is: “Do I need to validate or transform this inbound data”? I use “inbound” here very loosely, as this can mean data coming from external sources (like a user form or an API payload) or data coming from other layers within your application (like context boundaries).

Exactly this. I find embedded schemas particularly useful to define value objects (if you are familiar with DDD terminology).

And this is why I like to think of schemas like DTOs… They can represent database records, but they are much more powerful than this… You can also think about schemas as “windows” that let you peek into a subset of your data. This is extremely helpful for doing things like Command/Query segregation and overall decoupling your data from your domain representation.

5 Likes
10

Thanks! So Ecto schemas (or any schemas probably) are really useful when you need validations and possibly when you find it convenient for UI model not to map backend data closely. IIRC Phoenix forms can even provide some nicer looking error messages out of the box of there’s schema behind.

What about the relationship between schemas? Associations seem to be particularly difficult to build if there’s just a REST API instead of usual database behind as e.g. you need to load Organization first before setting it for a User. This certainly provides validation which… you may never need if your app is never going to change which Organization user belongs to.

Then specifically for never-changing associations schema’s associations seem to be not that useful and we could just link to e.g. in-memory Organization loaded a little earlier, correct? Or am I missing something?

11

More or less, yes. Especially embedded schemas and using schemaless Ecto are both amazing for (1) parsing and (2) validating data coming from the outside and then converting it to neat structs that obey the invariants of your business logic.

That’s really up to you though, you can just make them proper DB-backed Ecto schemas? Embedded schemas work well too but need a little grooming i.e. put_assoc et. al. at times.

2 Likes
12

I’m probably not the best person to comment on this aspect.

While I will have to decide exactly how to represent these kinds of more complex relationships, I haven’t gotten there yet. I look at it this way: if trying to build a highly structured model around a fully representative embedded schemas model is complex, doesn’t feel sufficiently natural, and doesn’t come with sufficient benefits to merit the trouble… I’ll likely not try to create a single embedded schema which faithfully models the entire form and be content with managing simpler collections of embedded schema structs which, on their own, represent discrete, independent form elements.

My goal with embedded schemas is simply validating input by “users” (meaning any external input), coming through a web interface or via an external API, so any answer that accomplishes that will be sufficient with the only question being which approach is optimal. Ecto facilitates a number of approaches in this regard. Again, figuring out the optimal approach, including how embedded schema associations work and their pros/cons, is a “future me” problem.

1 Like