I am building a stateful system that uses processes to store the current state. Each process (GenServer) is essentially polling an external API at a predefined interval, calculates differences between the current response and the previous one, publishes those changes and stores the latest response in the process’s state until it polls again. Each node has about 20.000 to 25.000 of these processes. I use the Registry module to store attributes for each such process so that I can locate all processes with a common characteristic using :ets.select/3
Now I’m looking to create a GraphQL API on top of this layer to expose the data. My problem is this: each process has a pretty big state with lots of nested stuff (let’s say 20-30 KB). The most straight forward solution for returning results is to issue a GenServer.call/2 to each matched process and then let Absinthe run its normal course resolving the document based on the state, but I’m worried that there will be a lot of copying involved for documents that only require a few properties. Ideally I’d be able to return from each process only the data requested in the document.
I started working on such a solution but it’s turning out pretty complicated as I believe execution can be suspended only for sibling fields, while I’d need to know all fields requested for all levels of nesting before issueing a call to the process to fetch only the data requested. So now I’m writing a custom plugin resolving each field using a token :defered value in the resolution struct to allow Absinthe to run the entire document and then I’m intending to fetch the requested data from each process and re-run the document, only this time with the data from each process available. My questions are:
Has anyone done something similar? I’d like to use something ready like Batch/DataLoader but I think those plugins do not cater to my needs
If not, I’m I thinking about this in the correct way? Sometimes I feel like I’m fighting with Absinthe but I’d really love for us to be friends instead. Am I trying to fit a square peg in a round hole with Absinthe?
Am I over-engineering this? Perhaps using the Async plugin and calling it a day would be OK but the stingy old man inside me won’t let me see this.
I believe one of the Elixir books explores this approach (maybe the Ecto book?) where you start off not using DBs at all and gradually start finding the need for them – and thus integrating with one. But I can’t remember which book exactly, sorry.
Outside of that, perhaps the Absinthe book will help. Haven’t read it though so I don’t know.
I am not seeing your problem being with Absinthe and more like with the fact how you fetch your data and that now your requirements have changed and that you have to rework your own storage layer to accommodate. Phoenix contexts and Absinthe modules are usually 100% indifferent of your storage AFAIR (do correct me if I missed some assumptions they make but I am currently not remembering any).
I am not informed enough on your project to give you a well-argued-for answer but from a bird’s eye view it seems you will introduce more complexity to grapple with.
I do have a question for you: what’s wrong with using a DB in your case? You don’t want to use an entire separate server for that? That would be a valuable objection for some apps indeed so I am curious what’s your reasoning.
Hi @dimitarvp, thank you for taking the time to answer!
I was hoping to avoid the extra storage layer if I could, because then I’d have to maintain that as well. But perhaps it is more straightforward than introducing weird behaviour in absinthe- I’ll have to think more about it.
Seeing that I’ll soon have a first ready-for-use version of an sqlite Ecto 3 adapter then that won’t be a separate service and won’t be an Ops concern (since sqlite is a single-file DB that runs inside your application’s process).
Like … tomorrow? 
