Hi, I am trying to understand, how to implement the following functionality in the elixir
Suppose, we have a web service, which obtains some text as input, and return some other text as a result.
From the cost perspective, it is much more efficient to perform requests to web service, when one provides a batch of texts, and obtains a batch of outputs.
However, end clients are sending one piece of text, when calling web service.
Therefore, one is interested in implementing an intermediate web service,
which is configured with max batch size and maximal allowed time to process request in milliseconds, and acts as a proxy, which gathers clients requests in the batch, calls original web service as a batch, and then returns results to the original callers.
Original callers usually would be different clients
As well, one expectes that original callers are calling the service via HTTP, so, they expect to provide input text and obtain a result in one HTTP request-response.
How the following request batching/providing results to end client functionality could be implemented in elixir?
For any type of batching service, the requests will need to be queued and then dispatched. For the queuing, you would need some sort of data storage - S3 / database / disk / etc.
I’m not sure exactly what sort of scale you’re looking at, but for a self-contained elixir solution, the easiest storage would be either ETS or Sqlite for the initial client call. The standard web framework is Phoenix. The initial connection would store the data with a timestamp and possibly the pid of the caller.
At that point, you’d want to periodically query the storage to see what needs to be transformed. There are a few ways to do this. You can use a package like Quantum for cron type scheduling. If you need something more granular, then you would need to create a GenServer that is supervised that calls itself with Process.send_after and handles it with handle_info.
Finally, the data has to be returned to the original client. Step 2 needs to put the data back into the storage and can send a message to the caller to pick it up if you stored the original PID of the process that stored the data. As a backup, you probably would need to do some type of long-polling on the client-side browser.
So to summarize, I would go with
a) Phoenix to handle the initial call, data upload to ETS for storage
b) a custom genserver to handle the batching, re-upload the data to ETS, and notify the original caller.
I’d have a GenServer to which you can send messages to accumulate items in batches. After the message is processed and the item is put inside the batch it should also check if the deadline has passed (or if the batch has hit maximum size) and then process all items in the batch, sending messages back to the consumers – a good way of doing this is described in GenServer.reply: Don't Call Us, We'll Call You.
Well, it won’t take you all the way there because f.ex. if you have a timeout of 30 seconds before sending a batch then the callers have to have huge (bigger) timeouts just so they never get left without a response. That’s brittle, insecure and prone to errors.
At that point you might be better off your requests just doing GenServer.cast and then waiting and reacting to events in a PubSub queue.
In any case, Elixir has all the building blocks for your task.
