GenStage examples or suggestions

Questions / Help
1

I’m trying to put together a plan for a library that I am working on whereby I need to provide an interface that allows the client to asynchronously call out to external APIs (HTTP endpoints in this case). I also need to make sure that this is rate limited to adhere to the rate limiting on the externals APIs.

From what I have read GenStage sounds perfect for this and using a ConsumerSupervisor I would be able to dynamically start workers for each request as they come in and let GenStage limit the number of workers it will start concurrently.

Does anyone know of any good examples of this type of setup in practice?

I am trying to get my head around how this all fits together and feel a real world example would help me out here.

4 Likes
2

Discord published a blog post covering how they use genstage to buffer calls to Firebase APIs. It has some code examples as well as performance statistics.

8 Likes
3

That looks awesome thanks @mechabyte. I will have a proper read through when I get home from work.

3 Likes
4

In addition to the discord article linked above, I came across this talk which is probably the best explanation and demo of an end to end genstage setup I have come across.

Well worth a watch (especially if you’re new to genstage)

6 Likes
5

Nice. haven’t seen this talk yet. :slight_smile:

Q: Is there use case for GenStage for example for simple CRUD API calls for users, projects, etc. Just for simple data requests?

1 Like
6

From my basic understanding so far I would say yes there is. Genstage looks to be a great fit for when you’re having to deal with external forces such as networking for example (rest apis, databases, etc…).

It allows the workers to dictate how much work they are given which prevents the likely important part of the system being overloaded and to remain free to do its thing and cause the backup of events further up the stack.

Someone please correct me if I have got that wrong.

2 Likes
7

Ok, this sounds cool. Is there any example of how you write GenStage server for example for User CRUD module? I don’t have an idea how you manage those actions in GenStage server.

2 Likes
8

I would suggest definitely take the 40 minutes to watch that talk as that will probably answer your question better than I could.

The demo is a restruant simulation which demonstrates the full cycle of tables placing an order (that’s processed through genstage) and then receiving thier food all within strict timeouts. Akin to sending CRUD data into genstage and receiving some data when it’s finished.

3 Likes
9

Ok, super. Added to my tomorrows to do list. Thank you! :thumbsup:

2 Likes
10

Here’s a simple example of a multi-stage pipeline. I found Jose’s talk really helpful (linked in example).

4 Likes
11

Thanks @wfgilman, that’s a very interesting example. I haven’t seen any others that are dynamically starting and linking multiple stages like that.

2 Likes
12

Just watched the talk and it was really nice. What I missed there, that there was no real diving in code and how to structure everything, so CRUD module would work under gen_stage.

1 Like
13

I thought the same initially but then I found all the code that was used here

5 Likes
14

You can try my tutorial here or check out the Elixir School lesson

6 Likes
15

I like this repo https://github.com/antonmi/flowex

5 Likes
16

Those links are awesome, thanks a lot!

2 Likes
17

Thanks, that’s what I was looking for! :slight_smile:

1 Like
18

@ybycode, thank you for the great lesson, just fix the link: https://elixirschool.com/en/lessons/advanced/gen-stage/

19

cc @yburyug :slight_smile: