I’m trying to track down a weird behavior from req_athena when running queries that take longer than 10 seconds. This is all running in Livebook.
What happens is the req_athena query/2 call runs for 10 seconds, then the DataFrame that’s returned gets rendered by kino_explorer as “Output data no longer available, please reevaluate this cell”. If you wait a few more seconds (depending on the query), the data appears and everything is fine. Everyone’s first reaction is to reevaluate the cell as directed. That obviously doesn’t help, so I’m trying to figure out if I can either make the query call wait long enough to avoid the message or change the message.
I tried calling Explorer.DataFrame.collect/1 on the results. This helps, but I think it helps mostly by delaying a little longer. The “Output data no longer available” error still shows momentarily.
Just in case we’re doing it wrong, here’s how we’re calling req_athena in the cell:
query = "<long SQL query>"
query_hash = q |> :erlang.md5() |> Base.encode16()
opts = [format: :explorer, output_location: "#{bucket}/livebook-#{query_hash}", ...]
response = ReqAthena.new(opts) |> ReqAthena.query!(q)
response.body
Any ideas or suggestions would be most appreciated.
1 Like
@fhunleth when the cell finishes evaluating, the output is blank for like 2s, and then “Output data no longer available, please reevaluate this cell” is shown, correct?
I tried calling Explorer.DataFrame.collect/1 on the results.
That actually makes sense. I believe the query returns a lazy dataframe, so if you call collect, it means the whole dataframe is in memory.
What happens is that when rendering the data table for the first time, the “table server” computes the first page for the table, but since the dataframe is lazy, it means it needs to download data, which takes more time. The Livebook client expects the initial data to arrive within 2s, otherwise it assumes the “table server” is dead and therefore shows “Output data no longer available”.
This is something that we can improve and it is actually already on my list : )
1 Like
Hi @jonatanklosko. Thanks for the response!
What you’re saying sounds good. It takes about 10.6 seconds for the cell to evaluate and a small blank gap opens up below the cell. About 2 seconds later, I get the “Output data no longer available, please reevaluate this cell”. About 10 seconds after that, the table renders.
I didn’t think to grep Livebook for the the error message, but I totally see it now that you pointed me to it.
I’ve very glad to know this is on your list. I’m not in a rush for a fix, but if it helps for me to test this case, please let me know.
Thanks!
1 Like
