bgpratt2000

bgpratt2000

Explorer.DataFrame and Livebook help

Hello!

I am a BEGINNER and apologize in advance for the elementary questions.

I want to do some data transformations to a dataframe in Livebook, but currently one of the dataframe’s columns/series representing a :utc_datetime from ecto has a :string dtype, and I cannot find a simple/automated way to convert this from a :string to a :utc_datetime and then to an Explorer.Series :date.

I followed this procedure to set up the livebook:

  1. ran an ecto.query on my postgresql database, where my select statement returns a map with atom keys
  2. passed the query as an argument to Ecto.Repo.all()
  3. converted the map from Repo.all() to a csv file
  4. so far have been doing useful data transforms and visualization using Explorer and Vega_Lite in Livebook, HOWEVER: in livebook the function Explorer.DataFrame.from_csv!/1 interpreted a :utc_datetime column as a :string series, and I don’t know how to convert the :string back to any useful/date-related Explorer.Series dtype.

At this exact moment I would appreciate a quick fix to this conversion struggle, but in general I hope to soon use livebook for more rigorous machine learning algorithms and data manipulations on data from my db, so any and all ways to streamline the above clunky process would be GREATLY appreciated.

Thanks! Y’all are awesome!!!

Marked As Solved

dtew

dtew

The following is Claude’s LLM answer to your query - hope it helps you!

Here’s how you can handle this:

  1. When loading the CSV, you can use the parse_dates option to automatically parse date/datetime strings:
df = Explorer.DataFrame.from_csv!("your_file.csv", parse_dates: true)

If that doesn’t work automatically, you can manually convert the column using mutate. Here’s how:

require Explorer.DataFrame

df = Explorer.DataFrame.mutate(
  df,
  datetime_col: cast(string_datetime_col, :datetime)
)

For more control over the parsing, you can use Explorer’s datetime casting functions. Here’s a complete example:

require Explorer.DataFrame, as: DF

# Starting with your string datetime column
df = DF.from_csv!("your_file.csv")

# Convert string to datetime
df = DF.mutate(
  df,
  datetime_col: strptime(string_datetime_col, "%Y-%m-%d %H:%M:%S", unit: :microsecond)
)

# If you specifically need just the date portion:
df = DF.mutate(
  df,
  date_col: dt_date(datetime_col)
)

For a more streamlined workflow from Ecto to Explorer, consider:

  1. Use Ecto’s select_merge to format dates in the query itself
  2. Or use Explorer’s ADBC functionality to read directly from your database:
{:ok, df} = Explorer.DataFrame.from_query(
  conn,  # ADBC connection
  "SELECT * FROM your_table",
  []
)

This would give you properly typed columns directly from your database without needing the CSV intermediate step.

The key functions to remember for datetime manipulation in Explorer are:

  • strptime - Parse string to datetime
  • cast - General type conversion
  • dt_date - Extract date from datetime
  • parse_dates option in from_csv

Where Next?

Popular in Questions Top

gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call t...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 43657 311
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36432 110
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 127536 1222
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

We're in Beta

About us Mission Statement