jmurphyweb

jmurphyweb

Advantages of Ecto preloads with/without joins

I have a slow query as my data has grown. I wanted to test joining before preloads to see if i could speed it up.

I did a test on some mock data, first with the join and the second without. It seems like the speed gain is minimal.

Are there any other benefits to joining first before preloads?

Is it more efficient for the databases memory usage if we join before preloading?

If I did some additional nested preloads would that start to affect things?

With join

iex(1)> query = from tc in Timecard, join: tc_d in Timecards.TimecardData, on: tc.id == tc_d.timecard_id, preload: [timecard_data: tc_d]

#Ecto.Query<from t0 in Ev2.Timecards.Timecard,
 join: t1 in Ev2.Timecards.TimecardData, on: t0.id == t1.timecard_id,
 preload: [timecard_data: t1]>

Has the following output:

iex(2)> Repo.all(query)
[debug] QUERY OK source="timecards" db=1242.5ms decode=760.5ms
SELECT t0."id", t0."date", ...,  t0."offer_id", t0."timecard_setting_id", t0."inserted_at", t0."updated_at", t1."id", t1."approver_id", ... , t1."updated_at" FROM "timecards" AS t0 INNER JOIN "timecard_data" AS t1 ON t0."id" = t1."timecard_id" []

Which looks like 1243ms.

Without join

Whereas the following with no joins and super simple syntax:

iex(3)> q2 = from tc in Timecard, preload: :timecard_data
#Ecto.Query<from t in Ev2.Timecards.Timecard, preload: [:timecard_data]>

Has the following output:

iex(4)> Repo.all(q2)
[debug] QUERY OK source="timecards" db=321.0ms decode=116.9ms
SELECT t0."id", t0."date", t0."week_ending_date", t0."datetime_submitted", t0."auto_submitted?", t0."auto_generated?", t0."employee_id", t0."proxy_creator_id", t0."offer_id", t0."timecard_setting_id", t0."inserted_at", t0."updated_at" FROM "timecards" AS t0 []
[debug] QUERY OK source="timecard_data" db=965.8ms decode=322.6ms
SELECT t0."id", t0."approver_id", ... , t0."timecard_id" FROM "timecard_data" AS t0 WHERE (t0."timecard_id" = ANY($1)) ORDER BY t0."timecard_id" [[15716, 15693, ...]]

Which to me looks like a total of 321 + 965 = 1286ms

Thanks for any input!

Most Liked

hubertlepicki

hubertlepicki

I think the issue with join-preloads can arise when you are attempting to preload has_many or many-to-many relationships. For these, the resulting SQL will output the same column values multiple times. If these are large text/blobs, the amount of data to be transferred over the wire quickly grows. Ecto deduplicates that on it’s end when returing data to the user so it is largely invisible, but joins behind the scenes still create this huge result that has to be moved from DB to Elixir in first place.

So, I think it makes total sense to preload something like belongs_to with joins, and possibly associations that just have a few related (small) records could be beneficial but for has_many associations probably you generally don’t want to do join-preloads.

check out this post: Composing Ecto queries: filters and preloads | AmberBit Sp. z o. o. and discussion I had with @michalmuskala on twitter: https://twitter.com/michalmuskala/status/1118120234991403009

hubertlepicki

hubertlepicki

This operation is very likely to be faster, especially as the number of records grow larger.

Where Next?

Popular in Questions Top

mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42920 311
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New

We're in Beta

About us Mission Statement