Emulating preload in Ecto Repo stream

Questions / Help
1

Hello,

I need to load data from two tables but I cannot use preload since my query has to be used in a stream (not my call).

So I am doing this:

    from(u in User,
      join: p in assoc(u, :profile),
      select: %{u | profile: p}
    )

And it seems to work just fine. Anything I should know that would make that a no-go?

Thank you.

2

If you are going to process records via Repo.stream I’d say you should make a batch query i.e.

    ids = values_you_got_from_repo_stream()

    from(u in User,
      where: id in ^ids
      join: p in assoc(u, :profile),
      select: %{u | profile: p}
    )

So you need to avoid the N+1 query problem by loading associations of a batch of records.

Outside of that I can’t think of any problems you might encounter.

1 Like
3

Hey thanks!

Infortunately I need to process the stream within the transaction. I’ll see if I can remove that constraint. But I do have a where clause, so there are not too many records (70k) and I can hold the transaction for quite a while, it’s on a task that is run periodically.

I do believe that Repo.stream is doing its own batches with the query, using a cursor, so I guess it’s not doing a SELECT for each user.

(If it’s not clear that query is the argument given to Repo.stream.)

1 Like