Fl4m3Ph03n1x
Background
In my fictitious car company I have a module that tells me if I can sell a car with some specifications in a given country. This module Cars.Jurisdiction reads a CSV file at compile time and it exposes a function that given a triple, compares that with the compiled CSV and returns true or false depending if I can sell or not the car in that country.
defmodule Cars.Jurisdiction do
#########################
# Compile time madness! #
#########################
csv_data =
:cars
|> :code.priv_dir()
|> Path.join("cars_to_sell.csv")
|> File.stream!
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Stream.uniq
|> Enum.map(fn {car, country, engine_type} ->
{String.trim(car), String.trim(country), String.trim(engine_type)}
end)
cars_with_electric_engine =
csv_data
|> # Filter and mapping operations
car_country_combo_with_diesel_engine =
csv_data
|> # Filter and mapping operations
allowed_cars =
csv_data
|> # Filter and mapping operations
allowed_countries =
Enum.map(csv_data, fn {_car, country, _engine} -> country end)
allowed_engines =
Enum.map(csv_data, fn {_car, _country, engine} -> engine end)
@allowed_cars_countries_engines csv_data
@cars_with_electric_engine cars_with_electric_engine
@car_country_combo_with_diesel_engine car_country_combo_with_diesel_engine
@allowed_cars allowed_cars
@allowed_countries allowed_countries
@allowed_engines allowed_engines
################################
# End of compile time madness! #
################################
@spec is_blocked?(String.t, String.t, String.t, [{String.t, String.t, String.t}]) :: boolean
def is_blocked?(car, country, engine, csv_input \\ @allowed_cars_countries_engines) do
# never trust a Human made CSV! Humans are carbon based! How flimsy!
csv_data =
Enum.map(csv_input, &trim_csv/1)
cond do
car not in @allowed_cars -> true
car in @cars_with_electric_engine -> false
{car, country} in @car_country_combo_with_diesel_engine -> false
country not in @allowed_countries -> true
engine not in @allowed_engines -> true
true -> {car, country, engine} not in csv_data
end
end
defp trim_csv({car, country, engine}), do:
{String.trim(sport), String.trim(country), String.trim(league)}
end
Problem
Now, because I have a lot of stuff happening at compile time, I expected the compile time to go up a little, specially since this is running in Elixir 1.5, which is missing a ton of improvements to compile time that newer versions have.
However,I didn’t expect this to take over 20 minutes to compile. I also didn’t expect to see compilation failing with the following error:
== Compilation error in file lib/cars/jurisdiction.ex ==
** (CompileError) Elixir.Cars.Jurisdiction: function 'is_blocked?'/2+1029:
An implementation limit was reached.
Try reducing the complexity of this function.
Instruction: {bif,'=:=',{f,0},[{x,2},{literal,<<"handball">>}],{x,1023}}
(stdlib) lists.erl:1338: :lists.foreach/2
(stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
make[1]: *** [compile] Error 1
I am not sure I understand what is going on here. It is complaining about my cond statement.
I am aware that this code would be cleaner with functions using pattern matching (in fact if I do so, I avoid this error, but the code still takes half an hour to compile) but this cond statement only has 6 cases.
Questions
- Why am I getting this error?
- Why don’t I get the error if I use functions with pattern matching?
- Why does it take over half an hour to compile?
And most importantly:
- How can I git rid of the error and the long compilation time ?
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #phoenix_html
- #elixirconf-us
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming










Showing Posts 1 to 8- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
lud
Looks like the
var in listoperation does not support over 1024 list elements.You should compile lookup functions based on your data, it would be more practical to use.
ericmj
You are getting the error because the function is too complex which means it has too many expressions.
You can try avoiding the
inexpansion by usingEnum.member?/2instead or by moving the literals to other functions:Fl4m3Ph03n1x
@ericmj This raises quite a few questions for me:
inexpansion orEnum.member??Enum.member?will compile faster than theinexpansion. Is this correct?Where can I read about the
inexpansion to learn more about it?ericmj
In this case I don’t think you should focus on which is more efficient since one of them doesn’t work. In the general case
inshould be more efficient when given a literal on the right hand side.Enum.member?is a normal function call, whileinis a macro that will do more work at compile time and can therefor be slower, but it’s unclear if it’sinby itself that is slow or the code it produces that is slow to compile. It can also be something else that causes the slowdown, we don’t have enough information to reproduce your issue. My suggestion was more about fixing the compilation error and less about fixing the slowness issue.You can find
ins documentation here: Kernel — Elixir v1.20.2.josevalim
You probably want to use a MapSet, which provides better performance for element lookups compared to lists, which are linear.
Fl4m3Ph03n1x
I have changed the
condto look like the followoing:With this change the error no longer happens. The compile time also went down from 25 minutes to just 3!
Going back to my list of questions:
Answers (based on my current understanding):
inoperator is a macro, which expands code behind. This expanded code generates a list with more than 1024 expressions to evaluate, which over the limit.inshould be faster at runtime, however it is slower at compile time.Good point! Will take this into consideration!!
benwilson512
We ran into this with Absinthe a bit. If the body of a module expanded into overly complex code the compile times eventually became non linear. We never did figure out why, and instead refocused our efforts on just minimizing the amount of code we dumped in the module body. I’m not 100% sure that your scenario is the same though because your expansion is happening principally inside function bodies.
eksperimental
just for the record Elixir v1.11 solves this limiation