GenServer / GenStage / Flow / or what?

Questions / Help
1

I have a scenario, which I am trying to code in Elixir. So bear with me I just need an idea about which would be the better tool for me to develop it.

  1. There are 50 Servers.
    a. Each server got an authentication key.
    b. Each server has some setting defined by servers owner that, how much I can consume it. Consume in a way, as each server will provide jpeg (nearly 4MB of size), but that either can be 1 per second, 1 per 5 seconds or 1 per 10 seconds or 1 per 5 minutes.
    c. Each request would be authenticated.
    d. Each server can go offline but the request for jpeg will continue going until the failure count get to 100. There are multiple types of errors and each error have different marks, such as nx-domain is 10, non-existent 20 and so on so when the total count get to 100, Server will be marked as offline, and the request will still going on, with some pauses (basis on the reason of going offline) , until it comes back online.

Now if the servers settings are: 1 request per second. I want to start a process, which will send an HTTP request to the server for jpeg, of course, it won’t come back in a second and the next request would have been shot, as its next second and so on, So I want to hold all requests if any request comes successful, It will contain and image and I want to upload it to another cloud server such as s3.

So, I don’t want to lose any jpeg request if its successful and each request jpeg is going to be saved with a timestamp, on which the request was sent (Not received ).

In a request sent at 8:00:00 and then 8:00:01 so on, requests should be sent, if it will be a jpeg, it will be uploaded, if its an error, that would be added to error count, but jpeg request will be continuing.

I also want to respect the CPU and RAM. as in future the server can go to 1000+

So this is all scenario, I am not asking anyone to do any homework for me, but when you read it. what does the first best tool come into your mind?

Is it a , GenStage / GenServer / Flow use case?

2

Each “server” sounds like a good fit for GenStateMachine, which is a thin abstraction on top of a plain GenServer.

HTTP requests could be performed from an individual state machine with a Task. It would need to keep track of the return value of Task.async to match up replies with requests.

1 Like
3

I read your post and it seems like a complex mix of requirements, so I’m not sure I fully understand everything, but here are my initial thoughts :slight_smile:

Technically you can accomplish everything using GenServer, but to me it sounds like GenStage and/or Broadway might be a better abstraction. Certainly Broadway is more opinionated, but you can easily consume and acknowledge events from a message broker (RabbitMQ/SQS/Kafka/etc), which might be a good idea in your case. It has some great features built-in, which you may be interested in, such as message acknowledgement and rate limiting. Just like GenStage, Broadway handles the backpressure for you, so you can tune it according to your system load requirements (re:CPU/RAM).

Anyway, this sounds like something you can build in several different ways, so when you start breaking it down, you may even find that you need a mix of different tools, not just one.

4

how genserver will call it self? in a loop? back and forth?

it will start and do the 3 parts of job and sleep if required and start again?