I’ve been working on Bitboard for quite some time. It’s a great side project, and I’ve learned quite a bit doing it.
Bitboard lets you manage Bitbucket issues using boards and cards. One of the first things I developed was the importer which now needs to be cleaned up. What it does is simple:

• Fetch all Bitbucket issues from a repository
• Fetch comments for each issue
• Fetch information about the members of the repository (so it is possible to assign and mention other members)

What I would like to add now is importing all pull requests, so these are linked to the relevant issues. I could shoehorn this feature into the existing codebase, but I would much rather figure out how to write this properly. Right now it’s a separate Context with sub context modules responsible for each step. But if one comment fails to import the entire import fails. And I don’t think this is desirable behavior. I would like it to be more fault tolerant and more loosely coupled, so it’s easy to add new steps into the importer and replace existing ones.

So long story short - is this a proper use case for something like GenStage?

I was thinking that each step should be its own stage (Fetch issues, Fetch comments etc.). And after everything is fetched, I would have additional stages for importing this into the database (just to separate those concerns). I would properly kick off a process for each comment being downloaded (but it should be rate limited somehow).

So is GenStage the right tool for the job? How would you structure this? I don’t expect a complete answer from anyone. I don’t mind reading and learning on my own. But it would be great if someone could let me know if I am on the right track or not. Is there a better approach?