ityonemo

ityonemo

Elixir scripting (.exs, not escript)

Every year I help my friend with his taxes, he has an itinerary of places that he’s been to for work and I scrape the google maps API to get driving distances. This year I did it in elixir. The major challenge is figuring out how to load external libraries (CSV, JSON) into an .exs script (no framework). Mix isn’t like python-pip (and that’s a really good thing TM) but every once in a while you just want it to do what you want it to. I can’t say that what I did is necessary “best practice”, but after a lot of hemming and hawwing, I figured out a reasonable solution. I’m leaving it here to hopefully help/inspire other people that need to do something quick 'n dirty.:

#  fail early if no filename is provided.
[filename] = System.argv()
File.exists?(filename) || raise "bad filename."
file_root = Path.basename(filename, ".csv")

#
#  before running, install jason and csv.
#
#  mix archive.install hex jason 1.1.2
#  mix archive.install hex csv 2.3.1
#  mix archive.install hex parallel_stream 1.0.6
#

# load up all of the libraries.
~w(jason 1.1.2 csv 2.3.1 parallel_stream 1.0.6)
|> Enum.chunk_every(2)
|> Enum.map(fn [lib, ver] -> "~/.mix/archives/#{lib}-#{ver}/#{lib}-#{ver}/ebin" end)
|> Enum.map(&Path.expand/1)
|> Enum.map(&Code.prepend_path/1)

# make sure the libraries are there.
Code.ensure_compiled(Jason)
Code.ensure_compiled(CSV)

# marshall the address into a dict.
addresses = "addresses.csv"
|> File.stream!
|> CSV.decode!
|> Enum.map(&List.to_tuple/1)
|> Enum.into(%{})

# read the paths.
driving_lines = filename
|> File.stream!
|> CSV.decode!

# check that they are all kosher
driving_lines
|> Stream.with_index
|> Enum.each(fn
  {_, 0} -> :ok
  {[_, start_name, dest_name | _], line} ->
    :erlang.is_map_key(start_name, addresses) || raise "bad line #{line + 1}, #{start_name}"
    :erlang.is_map_key(dest_name, addresses)  || raise "bad line #{line + 1}, #{dest_name}"
  _ -> :ok
end)

# this time I will remember to deactivate this API key
api_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
# generate the curl query  Mostly just using google's API.
curl_query = fn start_addr, dest_addr ->
  start_http = Regex.replace(~r/\s/, start_addr, "+")
  dest_http  = Regex.replace(~r/\s/, dest_addr, "+")
  "https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=#{start_http}&destinations=#{dest_http}&key=#{api_key}"
end

{rows, _} = driving_lines
|> Stream.reject(fn [_, a, b | _] -> a == "" || b == "" end)  # get rid of empty lines.
|> Stream.with_index
|> Enum.map_reduce(%{}, fn
  # ignore the first line, which is just for fun titles.
  {line, 0}, acc -> {line, acc}

  # if our start and destination are cached, then use that instead.
  {[r, start_name, dest_name | _], _}, acc when
      :erlang.is_map_key({start_name, dest_name}, acc) ->

    start_addr = addresses[start_name]
    dest_addr = addresses[dest_name]
    {[r, start_addr, dest_addr, acc[{start_name, dest_name}]], acc}

  {[r, start_name, dest_name | _], _}, acc ->
    # dereference the stard and destination addresses from our dictionary.
    start_addr = addresses[start_name]
    dest_addr  = addresses[dest_name]

    # run the curl command
    {res, _} = System.cmd("curl", ["--stderr", "/dev/null", curl_query.(start_addr, dest_addr)])

    # destructure google's insane JSON here.
    v = case Jason.decode!(res) do
      %{"rows" => [%{"elements" => [%{"distance" => %{"text" => distance}}]}]} ->
        # of course the string has "mi" attached to it.  Strip it out.
        {v, _} = Float.parse(distance); v
      _ -> "?"
    end

    # put the line back together, this time with full addresses and distances for the IRS.
    # cache the result into our accumulator map.
    {[r, start_addr, dest_addr, v], Map.put(acc, {start_name, dest_name}, v)}

  # lines which don't conform to our dogma get released
  {line, _}, acc -> {line, acc}
end)

# encode back into CSV.
rows
|> CSV.encode
|> Stream.into(File.stream!(file_root <> "-finished.csv"))
|> Stream.run

Most Liked Responses

LostKobrakai

LostKobrakai

You can put .exs files in a mix project and run them using mix run script.exs. That’ll start the script with all dependencies of the mix project present.

ityonemo

ityonemo

who’s got time to make a mix project =D

AndyL

AndyL

Also you can put a shebang in the file, chmod a+rx <yourfile> and just run it standalone like a bash script (without the dependencies).

#!/usr/bin/env elixir

IO.puts "HELLO WORLD"

Where Next?

Popular in Discussions Top

Qqwy
Looking at the stacks that existing large companies have used, WhatsApp internally uses Mnesia to store the messages, while Discord uses ...
New
MarioFlach
Hello, I want to share a project I’ve been working on for a while: https://github.com/almightycouch/gitgud Background Some time ago I ...
New
chuck
Let me start by stating an assumption: Phoenix is a great approach to building REST APIs. There are many reasons for this, but I will ass...
New
nburkley
AWS re:Invent is on at the moment with some interesting announcements. One new feature in particular is the Lambda Runtime API for AWS La...
New
AstonJ
I’ve just started the Phoenix part of the utterly brilliant online course by @pragdave. On generating the Phoenix app he uses the --no-ec...
New
IVR
Hi all, I’ve seen a number of related threads in the past, but I’d still be very curious to hear an up-to-date opinion on this topic. I...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
RudManusachi
What configs will make sense to put to runtime.exs? – A bit of how I configure apps: I have generic configs in config/config.exs, dev...
New
Owens
Hello all, I am developing a new mobile app with Flutter frontend and Phoenix backend. The mobile app has real-time task management and c...
New

Other popular topics Top

AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New

We're in Beta

About us Mission Statement