sehHeiden

sehHeiden

Explorer: DataFrame with StartDate and EndDate columns to timeline

I have a DataFrame Value columns and a Column with a start date (NaiveDateTime) and an end date (NaiveDateTime) at which the values where meassured. I would like to “unroll” the dateframe so that, I get a row for every date (day of the year), for every sample in the data frame.

I know, how I would do it in pandas (perhaps not the most elegant solution).
I would compute the day of the year (day) of the start and end dates, construct a list of every doy withing this range as a new column and than explode to return t 1st NF and that it is.

How would I do that with Explorer?

I started with:

filtered_polls = polls
|> DF.filter(start_date >= ^start_date)

polls_timeline = filtered_polls
|> DF.put("start_day", S.day_of_year(filtered_polls["start_date"]))
|> DF.put("end_day", S.day_of_year(filtered_polls["end_date"]))

polls_timeline
|> DF.put("date_range", polls_timeline["start_day"]..polls_timeline["end_day"])
|> DF.print()

Which is due to the range can not work on two series. But un two zipped lists.

dates = polls_timeline["start_day"]
 |> S.to_list()
 |> Enum.zip(S.to_list(polls_timeline["end_day"]))
 |> Enum.map(fn {start_date, end_date} -> start_date..end_date end)
 |> Enum.map(&Enum.to_list(&1))

 polls_timeline
 |> DF.put("dates", dates)
 |> DF.print()

Which does not work. Because I can not add a list as data type. Next I still would not know how to unroll.

Marked As Solved

billylanchantin

billylanchantin

Thank you! That makes it much easier to answer.

So I think @josevalim’s answer was what you want (for now, see below):

df = DF.new([
  %{start_day: 244, end_day: 248, fw: 15.0},
  %{start_day: 247, end_day: 249, fw: 16.0}
])

df
|> DF.to_rows()
|> Enum.with_index()
|> Enum.flat_map(fn {row, index} ->
  row["start_day"]..row["end_day"]
  |> Enum.map(&Map.merge(row, %{"day" => &1, "index" => index}))
end)
|> DF.new()
|> DF.arrange(index) # You can order by any column
|> DF.print(limit: :infinity)
# +---------------------------------------------------------+
# |        Explorer DataFrame: [rows: 8, columns: 5]        |
# +-----------+-----------+---------+-----------+-----------+
# |    day    |  end_day  |   fw    |   index   | start_day |
# | <integer> | <integer> | <float> | <integer> | <integer> |
# +===========+===========+=========+===========+===========+
# | 244       | 248       | 15.0    | 0         | 244       |
# +-----------+-----------+---------+-----------+-----------+
# | 245       | 248       | 15.0    | 0         | 244       |
# +-----------+-----------+---------+-----------+-----------+
# | 246       | 248       | 15.0    | 0         | 244       |
# +-----------+-----------+---------+-----------+-----------+
# | 247       | 248       | 15.0    | 0         | 244       |
# +-----------+-----------+---------+-----------+-----------+
# | 248       | 248       | 15.0    | 0         | 244       |
# +-----------+-----------+---------+-----------+-----------+
# | 247       | 249       | 16.0    | 1         | 247       |
# +-----------+-----------+---------+-----------+-----------+
# | 248       | 249       | 16.0    | 1         | 247       |
# +-----------+-----------+---------+-----------+-----------+
# | 249       | 249       | 16.0    | 1         | 247       |
# +-----------+-----------+---------+-----------+-----------+

Also, I don’t want to speak for the core team, but explode does appear to be on the roadmap:

https://github.com/elixir-explorer/explorer/issues/296

So if/when that functionality is added, you can do a nearly one-to-one translation of the Pandas code.

Where Next?

Popular in Questions Top

siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

We're in Beta

About us Mission Statement