Just use different modules for different games. Maybe also create a basic module with @callbacks and use it as a @behaviour on the implementation modules, to define a common interface for them, and ensure they have the functions the base game module calls.

If You like Poker with Elixir, there is an old post about it here https://tokafish.com/playing-poker-with-elixir-part-1/ You might need to adapt for newer version, but the post is nice.

Other games I often saw, Tic Tac Toe, Tetris…

I wrote a little game engine for go here elixir_go | Hex

For a more complete example, there is this video

I would take care not to link the game implementation (e.g. :card_played) to a GenServer. You can implement all of the logic with pure modules and functions.

To Spawn, or not to Spawn? by Saša Jurić gives a good overview about when and how to reach for processes and even talks about an implementation of a card game to boot (Blackjack).