Fire-Hound
How to replace positive and negative infinity to NaN in a dataframe for all columns?
I have a big dataframe with 74 columns and 756270 rows, I would like to make sure there are no infinities in any of the rows of the dataframe. I tried using mutate but I got stuck with trying to check for the type of the column.
df,
for col <- across(0..73) do
{
col.name,
cond do
Series.and(Series.equal(Series.dtype(col), {:f, 64}), Series.is_infinite(col)) ->
:nan
true ->
col
end
}
end
)```
Marked As Solved
billylanchantin
Hi @Fire-Hound, welcome to the forum!
Few points:
-
You can filter the
dtypeas part of the comprehension, and you don’t need0..73:for col <- across(), col.dtype == {:f, 64} do -
When you say you want to make sure there are no infinities, do you want to check? Or do you want to replace (impute) them?
If you just want to check, this will work:
require Explorer.DataFrame, as: DF df = DF.new(a: [:infinity, 1.0, 2.0], b: [0.0, 1.0, 2.0], c: ["x", "y", "z"]) DF.summarise(df, for col <- across(), col.dtype == {:f, 64} do {col.name, any?(is_infinite(col))} end ) # #Explorer.DataFrame< # Polars[1 x 2] # a boolean [true] # b boolean [false] # >If you want to impute, you’ll want this:
DF.mutate(df, for col <- across(), col.dtype == {:f, 64} do {col.name, if(is_infinite(col), do: :nan, else: col)} end ) # #Explorer.DataFrame< # Polars[3 x 3] # a f64 [NaN, 1.0, 2.0] # b f64 [0.0, 1.0, 2.0] # c string ["x", "y", "z"] # >
1
Also Liked
Fire-Hound
Popular in Questions
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum.
...
New
In templates/appointment/index.html.eex:
<%= for appointment <- @appointments do %>
<tr>
<td><%= appoi...
New
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: )
Hello all, this is ...
New
Hey,
Just curious what are the main benefits of Elixir compared to Clojure?
When is Elixir more useful than Clojure and vice versa?
Th...
New
i’m a new one to elixir
which editor can i use
vs code? or atom?
Thanks! :smiley:
New
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
I tried installing
elixir 1.11.2
erlang 23.3.4
via asdf in my zsh shell. Enabled the versions locally and globally.
When I list them ...
New
Other popular topics
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
What is the idiomatic way of matching for not nil in Elixir?
E.g.,
First way:
defp halt_if_not_signed_in(conn, signed_in_account) when...
New
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
Hello!
tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability.
After spen...
New
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









