Equivalent of panda commands in Explorer

I also came from Pandas so Explorer concepts were confusing at first.

First, Explorer is built on Polars rather than Pandas. So you’re already at a bit of a disadvantage if you’re trying to carry over your Pandas intuition. This guide may help a bit with that aspect: Coming from Pandas - Polars user guide.

As for the specific questions:

What is the idiomatic way to access data in Explorer?

EDIT: left this section out at first.

This is IMO the biggest mindset shift in transferring from Pandas to Polars. In Polars, accessing data like Pandas does with .loc[] and .iloc[] is discouraged. I suggest reading over that “Coming from Pandas” tutorial to better understand why. There’s a section specifically for selecting data.

The thinking is roughly this: it’s better to try and express what data you want though the built-in functions that are available. Those functions are designed with fast querying/manipulating in mind.

For example, selecting data by its index isn’t usually what you want to do because the index of a particular row is usually incidental. It’ll be better to try and access it by its properties through filtering. E.g. I want the row where name == "Billy", not where index == 5. Filtering by name is just as fast as filtering by index, but filtering by name also works regardless of the row order.

If you really do need data at a specific index, you can do it just like you’ve described. But the Polars philosophy discourages this access pattern in favor of filtering/selecting for a good reason.

Drop columns with nils

I don’t think we have a specific function. This should work:

require Explorer.DataFrame, as: DF
require Explorer.Series, as: S

df = DF.new(a: [1, 2, 3], b: [4, nil, 6])
# #Explorer.DataFrame<
#   Polars[3 x 2]
#   a s64 [1, 2, 3]
#   b s64 [4, nil, 6]
# >

columns_with_nils =
  df
  # Benefit of Polars: this step happens in parallel.
  |> DF.summarise(for(col <- across(), do: {col.name, nil_count(col) > 0}))
  |> DF.pivot_longer(df.names, names_to: "column", values_to: "has_nil")
  |> DF.filter(has_nil)
  |> DF.pull("column")
  |> S.to_list()
  #=> ["b"]

DF.discard(df, columns_with_nils)
# #Explorer.DataFrame<
#   Polars[3 x 2]
#   a s64 [1, 2, 3]
# >

Drop rows with nils

• drop_nil/2

Replace nils:

• fill_missing/2 for a specific column

For multiple columns at once, do:

col_to_value = %{"a" => 1, "b" => 2}

DF.mutate(df, for col <- across() do
  {col.name, fill_missing(col, ^col_to_value[col.name])}
end)
# #Explorer.DataFrame<
#   Polars[3 x 2]
#   a s64 [1, 2, 3]
#   b s64 [4, 2, 6]
# >

Conversion from string to integer

• categorise/2… ?

I’m not as sure about this one. I’d need to know more about what you’re trying to accomplish specifically.

I understand that these are newby questions, but I could not figure them on my own

Newby questions are welcome here

I’m also biased towards the “pandas way” since I’ve been using it for a long time

Related: try out Polars for your Python work. I heavily prefer it these days.

Wow thanks a lot for the detailed response

I obviously need to change my mindset, especially the “mask → filter → mask → filter…” logic inherited from numpy and pandas, but I’ve been using them for a long time so this is not easy…

I will also try to learn how to use across() which you use in your examples.

Regards, Aurélien