Fire-Hound

Fire-Hound

How to replace positive and negative infinity to NaN in a dataframe for all columns?

I have a big dataframe with 74 columns and 756270 rows, I would like to make sure there are no infinities in any of the rows of the dataframe. I tried using mutate but I got stuck with trying to check for the type of the column.

  df,
  for col <- across(0..73) do
    {
      col.name,
      cond do
        Series.and(Series.equal(Series.dtype(col), {:f, 64}), Series.is_infinite(col)) ->
          :nan
        true ->
          col
      end
    }
  end
)```

Marked As Solved

billylanchantin

billylanchantin

Hi @Fire-Hound, welcome to the forum!

Few points:

  1. You can filter the dtype as part of the comprehension, and you don’t need 0..73:

    for col <- across(), col.dtype == {:f, 64} do
    
  2. When you say you want to make sure there are no infinities, do you want to check? Or do you want to replace (impute) them?

    If you just want to check, this will work:

    require Explorer.DataFrame, as: DF
    
    df = DF.new(a: [:infinity, 1.0, 2.0], b: [0.0, 1.0, 2.0], c: ["x", "y", "z"])
    
    DF.summarise(df,
      for col <- across(), col.dtype == {:f, 64} do
        {col.name, any?(is_infinite(col))}
      end
    )
    
    # #Explorer.DataFrame<
    #   Polars[1 x 2]
    #   a boolean [true]
    #   b boolean [false]
    # >
    

    If you want to impute, you’ll want this:

    DF.mutate(df,
      for col <- across(), col.dtype == {:f, 64} do
        {col.name, if(is_infinite(col), do: :nan, else: col)}
      end
    )
    
    # #Explorer.DataFrame<
    # Polars[3 x 3]
    #   a f64 [NaN, 1.0, 2.0]
    #   b f64 [0.0, 1.0, 2.0]
    #   c string ["x", "y", "z"]
    # >
    

Also Liked

Fire-Hound

Fire-Hound

Thank you! This is really helpful :heart:

Where Next?

Popular in Questions Top

Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New

Other popular topics Top

dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 44778 311
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement