sehHeiden
Explorer: DataFrame with StartDate and EndDate columns to timeline
I have a DataFrame Value columns and a Column with a start date (NaiveDateTime) and an end date (NaiveDateTime) at which the values where meassured. I would like to “unroll” the dateframe so that, I get a row for every date (day of the year), for every sample in the data frame.
I know, how I would do it in pandas (perhaps not the most elegant solution).
I would compute the day of the year (day) of the start and end dates, construct a list of every doy withing this range as a new column and than explode to return t 1st NF and that it is.
How would I do that with Explorer?
I started with:
filtered_polls = polls
|> DF.filter(start_date >= ^start_date)
polls_timeline = filtered_polls
|> DF.put("start_day", S.day_of_year(filtered_polls["start_date"]))
|> DF.put("end_day", S.day_of_year(filtered_polls["end_date"]))
polls_timeline
|> DF.put("date_range", polls_timeline["start_day"]..polls_timeline["end_day"])
|> DF.print()
Which is due to the range can not work on two series. But un two zipped lists.
dates = polls_timeline["start_day"]
|> S.to_list()
|> Enum.zip(S.to_list(polls_timeline["end_day"]))
|> Enum.map(fn {start_date, end_date} -> start_date..end_date end)
|> Enum.map(&Enum.to_list(&1))
polls_timeline
|> DF.put("dates", dates)
|> DF.print()
Which does not work. Because I can not add a list as data type. Next I still would not know how to unroll.
Marked As Solved
billylanchantin
Thank you! That makes it much easier to answer.
So I think @josevalim’s answer was what you want (for now, see below):
df = DF.new([
%{start_day: 244, end_day: 248, fw: 15.0},
%{start_day: 247, end_day: 249, fw: 16.0}
])
df
|> DF.to_rows()
|> Enum.with_index()
|> Enum.flat_map(fn {row, index} ->
row["start_day"]..row["end_day"]
|> Enum.map(&Map.merge(row, %{"day" => &1, "index" => index}))
end)
|> DF.new()
|> DF.arrange(index) # You can order by any column
|> DF.print(limit: :infinity)
# +---------------------------------------------------------+
# | Explorer DataFrame: [rows: 8, columns: 5] |
# +-----------+-----------+---------+-----------+-----------+
# | day | end_day | fw | index | start_day |
# | <integer> | <integer> | <float> | <integer> | <integer> |
# +===========+===========+=========+===========+===========+
# | 244 | 248 | 15.0 | 0 | 244 |
# +-----------+-----------+---------+-----------+-----------+
# | 245 | 248 | 15.0 | 0 | 244 |
# +-----------+-----------+---------+-----------+-----------+
# | 246 | 248 | 15.0 | 0 | 244 |
# +-----------+-----------+---------+-----------+-----------+
# | 247 | 248 | 15.0 | 0 | 244 |
# +-----------+-----------+---------+-----------+-----------+
# | 248 | 248 | 15.0 | 0 | 244 |
# +-----------+-----------+---------+-----------+-----------+
# | 247 | 249 | 16.0 | 1 | 247 |
# +-----------+-----------+---------+-----------+-----------+
# | 248 | 249 | 16.0 | 1 | 247 |
# +-----------+-----------+---------+-----------+-----------+
# | 249 | 249 | 16.0 | 1 | 247 |
# +-----------+-----------+---------+-----------+-----------+
Also, I don’t want to speak for the core team, but explode does appear to be on the roadmap:
https://github.com/elixir-explorer/explorer/issues/296
So if/when that functionality is added, you can do a nearly one-to-one translation of the Pandas code.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








