I have been going around with this issue for a while and will like some help from you guys.
Use case
I am making a web application that will help you find your GMAIL INBOX attachments faster.
Basically you will go to this public web application, sign in with your gmail credentials and give the app read only permissions to your inbox.
The app will concurrently download the metadata of all the messages that have a attachment and store it locally.
Finally, you will be able to download the attachment file via a link.
The app works fine with a Postgresql but I wanted to use ETS to store the metadata temporarily in memory and when the user signs out delete it.
The issue
When the user signs in, it takes a while to download the metadata but I have to get back to the user fast so I make a GenServer cast call to download the metadata and return to the user. This works fine.
The problem is that I try to access the ETS table a few seconds after starting the download and I get this error:
It should work, but I’m not sure why you chose ETS over a plain old map.
Have the GenServer spawn a separate Task to do the actual downloading. The task can communicate back by messages, or write to the table directly if you set it public.
The problem with this solution is that while the ETS table will be accessible almost immediately it will not contain all the data for a while. Whether this is acceptable depends on your application. This problem is not dependant on how you decide to build your table.
The reason for your original error is that there is an implicit 5000 ms timeout in GenServer.call. You can make this longer by adding an extra argument with an explicit timeout. This is probably not a good way to solve your initial problem.
Robert, I totally agree. All the data will not be available at the time of the first. I will reduce the amount of data to download for the first time and leave a background job downloading the rest. For example. the first download will focus on the past month only.
I will still need to access this table while it is been written to.
There are no problems reading from an ETS table while it is being updated, as long as the processes can access the table. The interface to ETS tables is safe in the sense that reading and writing at the same time will never corrupt the table or the data in it. That is one thing you don’t have to worry about. This works automatically without using the read_concurrency or write_concurrency flags.
From what I can see is that a process does a GenServer.call to process #PID<0.1409.0>, get_contacts it looks like. The error comes from that the calling process times out after 5000 ms. So the GenServer never replies to the calling process, or at least replies too slowly. What does the code in the handle_call callback which handles the get_contact look like? Does it return a reply tuple, or is there some reason it hangs?
Just want to point out that even if there is no explicit timeout given in the GenServer.call there is a default timeout of 5000 ms.
Ok, the handle_call shouldn’t take 5 seconds (the timeout). So perhaps the gen_server is already doing something.
For example, is the GenServer downloading the list and blocking the other callers?
For example if you have:
def handle_call({:download}, _from, state) do
thing = download_takes_long_time(..)
{reply, ..., state}
end
you would block the entire GenServer until the the {:download} call returns.
Also, I am not sure that an ets table is the ideal choice for you here. An ets table is generally used when a GenServer with normal state is too slow or needs concurrent reads to it.
In your case, you are reading from the ets table in the GenServer which doesn’t really need to be the case. You could read directly in the get_contacts function.
Can you please use copy and paste code and put it in a fenced code block instead of screenshotting it?
The way you are creating the map feels a bit overcomplicated.
If all entries are equally shaped you should be able to achieve a similar result but maybe faster by doing the following:
def handle_call({:get_contacts}, _from, state = [table|_]) do
contacts = :ets.foldl(fn {k, v, _, _}, acc -> Map.put(acc, k, v) end, %{}, table)
{:reply, contacts, state}
end
This is not tested though. Of course you need to adjust the shape of the first argument to the fn. If it is a 2-tuple already you could experiment with just removing the map you currently do…
