tegmentum

tegmentum

Using Explorer, I have a df where rows have id, size columns. the df has multiple entries for 1 id, but sometimes has a nil in the size column.

I’m trying to grab all the ids that have at least 1 nil value in the size column, to later exclude them from my df.

I try this:

ids_to_exclude = 
  df
  |> DataFrame.filter(size == nil)

But get the following error:

** (ArgumentError) cannot invoke Explorer.Series.equal/2 with mismatched dtypes: :string and nil
    (explorer 0.8.1) lib/explorer/series.ex:6274: Explorer.Series.dtype_mismatch_error/3
    #cell:jtyrwno7aej2bj3n:3: (file)
    #cell:jtyrwno7aej2bj3n:3: (file)

I can’t figure out what’s the correct/expected approach to handling this without doing this in plain elixir.

Any help would be much appreciated!

Thanks :slight_smile:

Showing Posts 1 to 3

hugobarauna

hugobarauna

Livebook Core Team

You can do something like:

ids_to_exclude = 
  df
  |> DataFrame.filter(is_nil(size))
  |> then(fn df -> df["id"] end)
hugobarauna

hugobarauna

Livebook Core Team

Inside an Explorer Query, you can use any operation from Explorer.Series, like the is_nil/1 one.

tegmentum

tegmentum OP

Hey Hugo, thanks for getting back to me. I ended up using is_nil like this:

order_ids_to_exclude = 
  df
  |> DataFrame.filter(is_nil(product_size_category))
  |> DataFrame.select("Order ID")
  |> DataFrame.to_series()
  |> (fn %{"Order ID" => series} -> series |> Explorer.Series.cast(:integer) end).()

followed by:

df =
  df
  |> DataFrame.filter_with(fn df -> 
    Series.in(df["Order ID"], order_ids_to_exclude) 
    |> Series.not()
  end)
— All posts loaded —

Where Next? Top

Trending in Questions Top

Blokh
Hey guys, I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly Do you guys have any suggestions what is the best prac...
New
RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
samoloth
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
FlyingNoodle
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
New

Other Trending Topics Top

dimamik
Hey! At Software Mansion we’ve just released Legion, a library that puts AI agents inside your Elixir app, and we’d love to hear what you...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews