CharlesO
Hi.
Please I’m getting this unusual rounding error when summing numbers using the Decimal Library
My result is off by 0.01
208,505,442,949.08 # Excel
208,505,442,949.09 # sum function with Decimal
Any suggestions please?
Thanks.
def sum() do
f = "C:/Downloads/MMR100.txt"
{:ok, b} = :file.read_file(f)
a = String.split(b,"\n")
c = Enum.count(a)
IO.puts c # 299,725
_sum(a, "0.00") |> Decimal.to_string(:normal)
# 208,505,442,949.08 - Excel
# 208,505,442,949.09 - sum
end
defp _sum([], acc), do: acc
defp _sum(["" | rest], acc), do: _sum(rest, Decimal.add("0", acc))
defp _sum([v | rest], acc), do: _sum(rest, Decimal.add("#{v}", acc))
Trending in Questions
Hi all,
I’m working on a Phoenix project where I need to make external HTTP requests and I’m trying to choose between Req and HTTPoison....
New
I’m very new to Elixir, so sorry if this is a silly question:
I am wondering what happens to the mailbox of a process when the machine i...
New
How do I tell Hologram not to escape HTML when substituting into templates?
{Textile.render(@product.long_description)}
escapes the HTM...
New
Background
When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error:
* assembling mark...
New
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
Other Trending Topics
Hey there,
I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first.
...
New
Hi everyone :waving_hand:
I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly)
AI dream has many faces, but general...
New
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #elixirconf-us
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance










Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
LostKobrakai
Afaik excel does use floats, so maybe it’s excel, which is wrong.
CharlesO
all my values are 2-decimal places
The Validator Program (some proprietor java program) also tells me the sum is wrong:
When i manually adjust the sum to that excel value and feed it to the validator .. it validates
NobbZ
Parsing and summing the list as float gives:
208505442949.07806, rounding this to 2 decimal places returns208505442949.08, so obviously Excel and your verification program do use floats and therefore are inaccurate by design…PS: I’ve heard there are still people using Excel to manage their money…
PPS: my iex session:
CharlesO
All the numbers in that list are 2-dp, how can you get
208505442949.07806???NobbZ
The magic of floats.
LostKobrakai
This exactly is the reason, why you’d even need a library like
Decimal. Because floats are not suited for calculating correctly.CharlesO
Every single finance department i have dealt with keeps parallel records in excel.
For some some reason there is an innate distrust of records from IT systems
NobbZ
No, I do get
9:LostKobrakai
sribe
Bah. I only looked at the text in the post. Did not notice the “this file has been truncated”…