jeffdeville
I’ve got a macro that wasn’t impacting my project compilation speed much. I added a small feature to it, and now it’s ~50x slower. But at least to look at it, it certainly doesn’t appear as if 50x more work is being done.
Are there any tools I can use to determine why this is the case? After the AST is generated (this part is fast), I’m not sure where things ‘go’, or what tools I might have to identify the source of my slowdown.
Trending in Questions
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
Hello,
I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
So my question is quite simple and i have found no conclusive answer on forum, google or AI.
Should we use :erlang.float for Integer to ...
New
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
Documentation
While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
New
Other Trending Topics
I am happy to introduce the very α version of the new programming language compiled to BEAM.
Welcome Cure.
It has literally three kille...
New
Hi there! We created Gust: A task orchestrator inspired by Airflow.
For those who have never heard about Aiflow, it’s a Python-based wor...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself.
My main conc...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming










Showing Posts 1 to 9- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
jeffdeville
One piece about the updated, slow macro code, is that the new macro code calls the existing (compiled fast) macro code. The old code was all basically independent.
I’ve been gutting the macro trying to figure out where things are going awry. Turns out that adding in just this code (formatted Macro.to_string output) is enough to double my project compile time from 4-8s.
josevalim
If this is being called a lot of times, then yes, the amount of code you generate will have a direct impact on compilation times. If you can post a snippet, we can help reduce the compilation time, but this may very well be a hint to find another approach that generating many many functions.
In any case, it is also worth mentioning that Elixir v1.5 speeds up how function definitions for large modules, so maybe try out Elixir master and see if that improves the situation.
jeffdeville
Sorry @josevalim, I usually try and post more detail up front. In this case that’s a little hard and I didn’t want anyone to have to spend too much time following my business domain here. My apologies in advance.
Conceptually, I am making binary TCP calls to read data from a device using the Modbus protocol. I’ve created a macro that let’s you define these values like this:
I was following (I hope) Chris’s guidelines in Metaprogramming Elixir, so I have the overall macro split across multiple files.
https://github.com/jeffdeville/ex_modbus/blob/fixing-group-fields/lib/ex_modbus.ex
https://github.com/jeffdeville/ex_modbus/blob/fixing-group-fields/lib/ex_modbus/macros.ex
Each
fieldbecomes either 1 method (:r) or 2 (:rw)Reads request data starting at a memory address, and reading N bytes. Then I convert the bytes to the type
Writes of course do the same thing, but send data down the wire.
This all compiles quickly. It’s the ‘old’ code I mentioned.
But I often need to load groups of fields, and it’s more efficient to load them all at once.
field_group defines a method that will load the given fields all in 1 shot. This is the slow one.
https://github.com/jeffdeville/ex_modbus/blob/fixing-group-fields/lib/ex_modbus/macros.ex#L6
What it does, is define a composite memory block to load all at once. Then it will slice off the bytes for each field, and ask the ‘field’ generated code to deserialize each piece for me. Returning all of the data in a Map.
The small example above isn’t all that painful. But I’m doing this with about ~140 fields spread over 6 field_groups. What’s interesting, is that those 139
fieldcalls are fast. It wasn’t until I added thefield_groupsthat things slowed down.Possibly Relevant
Because it’s the field_group that caused the speed issue, I don’t think this is relevant. But just in case…
The project that is actually slow to compile is this one, because of this test file:
https://github.com/jeffdeville/ex_sunspec/blob/enable-field-groups/lib/ex_sunspec/test_inverter.ex
Each of those model numbers is mapped to an XML file that contains all of the field definitions for SunSpec. So at compile time, the sunspec macros load the xml, and then invoke the ex_modbus macros.
josevalim
You are generating large function bodies over and over again. You want each generated function to be something like:
I am not sure this will make it faster but it will certainly lead to cleaner code.
jeffdeville
Thanks Jose. I fixed the problem and I think I’ve got something interesting to report.
First, I made the changes you suggested. Boiled each method down to 1 line in another module. The compilation time was still 108 seconds.
The fix was to inspect all of the values I was writing into my @docs that were not strings. (atoms, and ints)
I changed my @docs from:
to
and my compilation times sped up 43x
josevalim
That’s very interesting. Thanks for reporting back. Which Erlang version were you using?
Can you also post what those strings would look like before and after your changes? I am wondering if inspecting is limiting something that would otherwise be very large or if it is forcing the string to be rendered as a binary which would affect the rest of compilation.
Compilation of large literal binaries/strings have been generally improved in OTP 20 though: Move expansion of strings in binaries to v3_core by josevalim · Pull Request #1131 · erlang/otp · GitHub
jeffdeville
erlang 18.3
elixir 1.4.1
Here’s the result of Macro.to_string for the fixed version:
and the slow version:
Bear in mind that the real files were a lot longer. I shorted these to prevent exceeding the forum length restrictions.
I’ll try it OTP 20 in the morning and update w/ my results.
jeffdeville
Results for:
erlang 20.0
elixir 1.4.5
No ‘inspect’ compile time: 117s
With ‘inspect’ compile time: 11.42s
This was a different, larger project, which is why the timing’s a little different. Clearly though the difference remains. @jose would you like to see the AST’s generated by this version, or is that likely to be the same as what I posted previously?
OvermindDL1
/me wonders if the original version is slow because it is doing compile-time protocol dispatch for the
to_stringcall so the protocol dispatch is not optimized yet, but why is the inspect dispatch fast, is inspect already a more simple protocol like a single module in most cases?