Recommendations for Virtual Attributes in Ecto

Imagine an application where you need to store a User Profile. In there, you want to have a field for storing profile picture path in a storage bucket.

When getting the profile from DB, what’s the best way to populate the profile picture path with the CDN URL.
Also it may have other calculated fields like full_name

defmodule UserProfile do
  schema "user_profile" do
    field :first_name, :string
    field :last_name, :string
    field :full_name, :virtual
    field :age, :integer
    field :profile_image, :string
    field :profile_image_url, :virtual
  end
end

Whats the recommended way to populate virtual attributes in a scalable way?

1 Like

Well, it depends.
If something always has to be present or is logical to be present (like full name based on first and last names) I would implement that in the changeset function (Ecto.Changeset — Ecto v3.10.1).
That allows you to create custom validations.
If you look at Ecto.Changeset — Ecto v3.10.1, you will see functions like add_error/4 and put_change/3, this is where I would start.

If you use GraphQL (and Absinthe GitHub - absinthe-graphql/absinthe: The GraphQL toolkit for Elixir) and your value has to be present only on demand and not in most of the scenarios, I would recommend implementing a resolver (Query Arguments — absinthe v1.7.1).

Hope that helps :sunglasses:.

My use-case is to have them in the domain/context rather than in another layer like resolver.

Btw isnt changesets are for validating data? I’m doing quite the opposite. After fetching data i want virtual fields to work

This might be handy: GitHub - vereis/ecto_hooks: Ecto addon that re-implements Ecto.Model callbacks

1 Like

but isnt this kinda a bad practice in elixir?

I wouldn’t say bad practice. Elixir prefers not to have things done “magically”. I would say this is magic.
The docs talk about this here: GitHub - vereis/ecto_hooks: Ecto addon that re-implements Ecto.Model callbacks that similar functionality was available before but since then has been removed.

I only used this library in 1 project so far.
In the past similar issues like what you are facing I have managed in 2 ways.

  1. Depending on the application, resolve the urls in the view files where you build the responses (then you might not need the virtual field at all).
  2. In your relevant context have a function that takes your model and resolves the virtual field. Have a singular and a list version of it. Then pipe your get/list etc funtion results to this function. This results in the same as ecto_hooks in a less magical way.
2 Likes

If the path URLs can be constructed predictably then you won’t need a virtual attribute, just have a function e.g. UserProfileHelpers.profile_image_url(user_profile) and you should be set. Same for the full name.

1 Like

This was my main concern

its not scalable, imagine having multiple virtual fields and accessing them in different places

I’m skeptical that it would be a problem. Yes it might poke our eyes out to see code that calculates a value 3-4 times and not have it persisted / cached, I get that, I’m just not convinced the virtual attributes are worth the trouble.

Consider this thread: What's the easiest way to populate ecto's virtual field with database data

It has a few ideas that might help you.

4 Likes

This is similar to the approach we’ve taken at my current workplace, seems to work well. We use a protocol so something like:

defprotocol MyApp.Formatter do
  def format(value)
  def format(value, opts)
end

Which can then be implemented in our schemas, so something like:

defmodule MyApp.MySchema do
   schema "some_schema" do
    field :first_name, :string
    field :middle_name, :string
    field :last_name, :string
    field :gender, Ecto.Enum, values: [:male, :female, :other]
   ...
   ...

  defimpl Greenlight.Formatter do
    def format(user), do: "#{user.last_name}, #{user.first_name}"
    def format(user, :gender), do: user.gender |> Atom.to_string() |> String.capitalize()
    ...
1 Like

how do you use this across the application

You would just call the format function, i.e. Formatter.format(user). tbh, it’s probably better to start with just doing what @dimitarvp suggested and use plain functions. Easy enough to switch things out if you find you’re consistently needing this kind of stuff for a lot of your structs and it would make things easier to use a protocol with a consistent name/interface.

I’m not sure I understand the worry here. Is it was Dimitar was saying?

I think a helper function is the way to go, though, mainly because I feel it is more scalable, or at least more maintainable. These are also really presentation concerns. OO makes it convenient to stuff these things onto an object but it’s really not the “proper” place.

The way I did do this when I was using virtual fields in this fashion was have a populate_virtual_fields function which I’d call from my context:

# schema (or wherever you keep your queries)
def populate_virtual_fields(query) do
  from u in query,
    select_merge: %{full_name: fragment("concat(?, ' ', ?)", u.first_name, u.last_name)}
end

# context
def get_user!(id) do
  User
  |> User.populate_virtual_fields()
  |> Repo.get!(id)
end

def list_users do
  User
  |> User.populate_virtual_fields()
  |> Repo.all()
end

This works (well, it does if I didn’t make any syntax errors, I did not test) but it’s just another thing to keep track of and it can get confusing as to what is virtual and what isn’t. I feel virtual fields are best left up for temporary storage for stuff like flags (like a delete flag). Having said that, I don’t think it’s particularly unidiomatic or anything to use the virtual field. It might work out but in my estimation you’re probably going to end up having a bad time. You might not! But you might.

2 Likes

Interesting. I do something similar but on the other end…

User
|> Repo.all()
|> User.load_virtual_fields([:full_name])

I do kinda wish there was on_load callback or something to automatically load them though.

For me, it’s all about using them in HTML forms. For some reason (maybe it’s because I’m bad at UI stuff), I feel like my database tables/columns don’t exactly translate to what I want to present as form inputs in the UI.

1 Like

Ah ya, that’s a tough one! I have yet to encounter that myself. Is this something that happens often?

Incidentally first name/last name is generally an anti-pattern anyway. Many peoples’ name’s do not conform to this.

At a certain point of difference I‘d consider read models, where you have a separate schema, which conforms better to the data you read off of the db than what you write to it.

A similar idea would be view models, just that those would be created closer to the view layer, without contexts being involved.

Imo the „issue“ being skirted around here is wanting to keep similar, but not the same, data structures the same. Virtual fields can be a shortcut to keep them the same, but aren‘t a general purpose solution to the problem.

3 Likes

I’d say that’s a good thing. Data storage and presentation are two separate things and neither UI should dictate how the database is organized (except maybe read models), not DB schema should limit what can be achieved in the UI. However, I wouldn’t use virtual fields too much for this. Like @LostKobrakai, I’d usually choose creating a different struct, specifically for passing it to the view/template and optimized for what needs to be displayed.

1 Like

WTF, of course they should be different.

I’ll echo the others: just have a transformer function and pass its result to the UI. Some might even call them “views” in these circles. :yum: (I know, they are not exactly that.)

Documentation, examples, and tutorials all seem to suggest otherwise, in my experience. Especially ActiveRecord, but even Ecto.

The workflow is always: validate your changeset, send your changeset to Repo.update. Not validate your changeset, now take those values and make a new changeset or update a schema directly.

I don’t disagree with y’all, I’m just saying it isn’t exactly an obvious workflow given how all the docs/examples are written.