I am working with monetary data, for which floats are largely discouraged. As a result, I am using the fairly popular Decimal package and the associated Ecto support to store monetary values in my database and process them in my app.
I’m also a big fan of Livebook and Explorer to process my data. However, Explorer does not currently have support for Decimals, which means that I have to manually convert to floats, losing all the benefits of working with Decimal in the first place.
Therefore, I am wondering what it would take to add support for Decimals to Explorer, and whether anyone is working on such support already. If not, how would such a task even be completed? Would we need to add support directly in the Explorer library, or could some sort of external plugin be enough? Would anyone else be interested in this?
Explorer doesn‘t use elixir datatypes. It uses a rust library to deal with data in raw binary form. For explorer to support fixed precision numbers the underlying rust library would need to support those. Or you convert to integers at the precision you need.
1 Like
Hi @munksgaard,
@LostKobrakai is right. We rely on the Polars library under the hood. So my first instinct is no, we won’t be able to support the Decimal library.
However, it may be worth making an issue on Explorer anyway. Polars has support for a decimal datatype:
We add new data types relatively frequently, and this may be worth pursuing. The tricky part would be figuring out how we represent things on the Elixir side. But it’s certainly possible.
2 Likes
Follow-up: We did end up adding decimal support via Decimal.
-
Add support for the decimals data type.
Decimals dtypes are represented by the {:decimal, precision, scale} tuple, where precision can be a positive integer from 0 to 38, and is the maximum number of digits that can be represented by the decimal. The scale is the number of digits after the decimal point.
With this addition, we also added the :decimal package as a new dependency. The Explorer.Series.from_list/2 function accepts decimal numbers from that package as values - %Decimal{}.
This version has a small number of operations, but is a good foundation.
4 Likes
