Adding support for Decimal in Explorer

Follow-up: We did end up adding decimal support via Decimal.

• Changelog — Explorer v0.11.1

• Add support for the decimals data type.

  Decimals dtypes are represented by the {:decimal, precision, scale} tuple, where precision can be a positive integer from 0 to 38, and is the maximum number of digits that can be represented by the decimal. The scale is the number of digits after the decimal point.

  With this addition, we also added the :decimal package as a new dependency. The Explorer.Series.from_list/2 function accepts decimal numbers from that package as values - %Decimal{}.

  This version has a small number of operations, but is a good foundation.

Explorer doesn‘t use elixir datatypes. It uses a rust library to deal with data in raw binary form. For explorer to support fixed precision numbers the underlying rust library would need to support those. Or you convert to integers at the precision you need.

Hi @munksgaard,

@LostKobrakai is right. We rely on the Polars library under the hood. So my first instinct is no, we won’t be able to support the Decimal library.

However, it may be worth making an issue on Explorer anyway. Polars has support for a decimal datatype:

• polars/py-polars/src/datatypes.rs at 849953b097e0d15fe107f3683f003228103b425d · pola-rs/polars · GitHub

We add new data types relatively frequently, and this may be worth pursuing. The tricky part would be figuring out how we represent things on the Elixir side. But it’s certainly possible.