Fl4m3Ph03n1x
Background
I have to read a CSV and currently this is happening at compile time as the function runs in a module attribute:
# Imagine this csv file has 3 columns "sport, country, league"
@csv_sports_data
:my_app
|> :code.priv_dir()
|> Path.join("awesome_csv.csv")
|> File.stream!()
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Enum.uniq()
So now, because this runs at compile time (iirc) I have a variable with the data I need in tuple format. So far so good.
Problem
The problem comes when I need to do the same thing, multiple times, with small variations:
# The duplication, IT BURNS !!!
# Imagine this csv file has 3 columns "sport, country, league"
@csv_sports_data
:my_app
|> :code.priv_dir()
|> Path.join("awesome_csv.csv")
|> File.stream!()
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Enum.uniq()
@sports
:my_app
|> :code.priv_dir()
|> Path.join("awesome_csv.csv")
|> File.stream!()
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Stream.uniq()
# Always trim data from pesky users!
|> Stream.map(
fn {sport, country, league} ->
{String.trim(sport), String.trim(country), String.trim(league)}
end)
|> Stream.map(fn {sport, _country, _league} -> sport end)
#No empty sports!
|> Enum.filter(fn sport -> sport != "" end)
@countries
:my_app
|> :code.priv_dir()
|> Path.join("awesome_csv.csv")
|> File.stream!()
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Stream.uniq()
# Always trim data from pesky users!
|> Stream.map(
fn {sport, country, league} ->
# Always trim data from pesky users!
{String.trim(sport), String.trim(country), String.trim(league)}
end)
|> Stream.map(fn {_sport, country, _league} -> country end)
# We allow empty countries to make the example interesting
As you can see, I have a lot of duplicated code. At the very least I could place
:my_app
|> :code.priv_dir()
|> Path.join("awesome_csv.csv")
|> File.stream!()
|> CSV.decode!(headers: false, separator: ?;)
|> Stream.map(&List.to_tuple/1)
|> Stream.uniq()
Into a function or variable and then re-use it in @sports and countries. The trimming function is also another candidate. And then there are the little differences for @sports and @countries where I select only the values I want.
Things I tried
So, my first try was to use the @csv_sports_data inside the @sports and @countries attributes. Obviously this didn’t work, as I can’t use something that was not yet compiled into an attribute that is itself being generated at compile time.
# This wont work
@sports
@csv_sports_data
# Always trim data from pesky users!
|> Stream.map(
fn {sport, country, league} ->
{String.trim(sport), String.trim(country), String.trim(league)}
end)
|> Stream.map(fn {sport, _country, _league} -> sport end)
#No empty sports!
|> Enum.filter(fn sport -> sport != "" end)
My second try was to consider Macros. According to my understanding, I could create a Macro that reads the CSV file at compile time and then have @sports and @countries use it. However, I personally am a believer of the saying:
“The first rule about Macros - don’t use Macros”
And I feel the usage of a Macro for this specific situation would be quite overkill. So I would like to avoid it.
And then there is also the trim function:
Stream.map(
fn {sport, country, league} ->
# Always trim data from pesky users!
{String.trim(sport), String.trim(country), String.trim(league)}
end)
Which I cannot place inside a def or defp for the sake of reuse.
What now?
Surely I am missing something. Perhaps the solution I was given to work with the CSV is flawed, or perhaps I am forgetting some mechanism that would reduce the amount of duplicated code I have.
- How can I remove all the duplication?
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #phoenix_html
- #iex
- #elixirconf-us
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming










Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
michallepicki
You can operate on values (not module attributes) in a module body, do some computations and only then assign them to attributes:
And re-use the already computed value to declare other module attributes:
michallepicki
You can also move logic to other module that will become a dependency so it will get compiled earlier, where you can split your logic in functions however you like, for example:
and then you’ll be able to use it directly in your other module:
edit: or re-use this logic in any other module
eksperimental
I think what you are trying to achieve defining attributes is what you should be doing but defining macros.
Using a macro to optimize things that can be calculated at compiled time is a perfectly use for a macro.
You just need to figure out what is available at compile time and what is not, reuse and move into functions the rest.
eksperimental
in addition if calculating countries and sports is an expensive operation, build a really simple caching system with ETS.
I don’t think you can get any faster than this
eksperimental
exactly! but i would convert
read_sports_data/0into a macroLostKobrakai
That‘s not really needed. Macros would only make things more complex, as at no point AST has to be modified.
Fl4m3Ph03n1x
Does this work for Elixir 1.5?
Currently this code is not working:
Error
I believe this happens because of
csv_data, which is not executed at compile time (I think).lud
You must remove the
=here. This is a common mistake I do all the timeI agree with @LostKobrakai , you don’t need macros here as you are not creating code by generating AST.
You should create a helper module specialized in reading your CSVs with all the required variations and parameters.
Those functions will be available at runtime, obviously. Then, in your main module
you wouldyou can also call the functions at compile time.requirethis helper module, somichallepicki
Yes, as mentioned by @lud you need to fix the module attribute declaration. The error message is not intuitive, though!
michallepicki
I believe you don’t need to
requirea module to use its functions at compile time