I have a large text file that I need to read line-by-line and collect various statistics about its contents, e.g. the amount of certain phrases, words, characters, etc. Then, I need to turn those statistics into a neat human-readable string.
My current code roughly looks like this:
defmodule MyStruct do
@moduledoc false
defstruct total: 0, foo: 0, bar: 0, baz: 0
end
defmodule Parser do
@moduledoc false
defp translations do
%{
"Example String 1" => %{translation: :foo},
"Example String 2" => %{translation: :bar}
# Some complex strings that optionally may need to be further parsed with regexps, etc., hence maps as values
# ...
}
end
def parse_file do
"large_text_file.txt"
|> File.stream!()
|> Enum.reduce(%MyStruct{}, fn line, acc -> maybe_update_acc(line, acc) end)
end
defp increment(map, key), do: Map.update(map, key, 1, &(&1 + 1))
defp find_tl(string) do
Enum.find_value(translations(), fn {k, v} -> if String.contains?(string, k), do: v.translation end)
end
defp maybe_update_acc(line, acc) do
tl = find_tl(line)
acc
|> increment(:total)
|> increment(tl)
# Irrelevant logic for regexps and other conditions to update the acc further
# ...
end
end
defmodule Stringifier do
@moduledoc false
defp translations do
# Is a list because the result needs to be ordered in a certain way
[
total: "Everything, duh",
foo: "Those fabulous foos",
bar: "Them beautiful bars"
# ...
]
end
def translate(%MyStruct{} = struct) do
Enum.map(translations(), fn {key, string} ->
struct_value = struct[key]
if struct_value, do: translate_key(key, string, struct_value)
end)
end
def translate_key(key, string, value) do
case key do
# Keys that need unique formatting or other logic
# ...
key -> "#{string} - #{value}"
end
end
end
This approach has some serious flaws. The most obvious one is that every time I need to add a new string to collect data about or remove/change an existing one, I need to edit my code in three places: the struct definition, the parser and the stringifier. This is bad. If I forget about one, in the best case scenario my application crashes, in the worst case scenario I get a completely wrong result.
However, I can’t come up with a better way to do things. Maybe I could make a single keyword list like this, [foo: %{term: "Example String 1", string: "Those fabulous foos"}], which will give me the required ordering and will allow to unify the translations from Parser and Stringifier modules.
At a first glance, this probably won’t add too much strain on my find_tl/1 function, but because I also have “meta” keys (like :total) that don’t resolve to any terms from the file, this, in fact, will actually increase the complexion. Usually I don’t care about such overhead if it means for better readability, but the text file can have billions of lines, so I need all the speed I can get.
Some “meta” keys need to be in the front, some in the back, some in the middle, so simply prepending/appending them in the Stringifier module is not an option.
Besides, this would mean that I still will have to edit the struct. Though I guess this is probably not that important, because I don’t do any struct-related compile-time checks.
