Does anybody have a OTP/Genserver/Flow web app project?

synthslave · February 2, 2018, 6:47pm

Hello all,

I’ve been trying to learn and study how OTP/Genserver/Flow works within the context of a web application and am at the point where I grasp the concepts but think it would really solidify the lessons if I “look under the hood” in a real web app project.

Does anybody know of any projects?

I was going to post this in the projects section but couldn’t find anything there.

Thanks in advance!
Sy

idi527 · February 2, 2018, 6:54pm

There are

https://github.com/hexpm/hexpm
https://github.com/dailydrip/firestorm
https://github.com/JEG2/hanabi_umbrella
probably some others as well …

ElixirConf 2016 “Selling Food With Elixir by Chris Bell” talk is also quite nice https://www.youtube.com/watch?v=fkDhU-2NWJ8

jordiee · February 2, 2018, 7:59pm

Here is my authentication library. It may help to look at it because its quite small for the most part. Uses genserver and sub supervisor to manage tokens(the actual logic probably needs cleaned up though).

idi527 · February 2, 2018, 8:07pm

Do you keep the tokens in a single genserver?

jordiee · February 2, 2018, 8:12pm

I went with the “idea” of short long lived refresh tokens and short lived access tokens so they each have their own genserver and their own supervisor

idi527 · February 2, 2018, 8:13pm

If you get a lot of reads, these processes might become the bottleneck very quickly. I would suggest using ets tables for storing the tokens, since they allow for concurrent reads.

jordiee · February 2, 2018, 8:15pm

it may be the case because of it being serial but it would have to be A LOT of reads because it just does key lookup on a map. But yes that’s definitely something I am thinking about for the next version. That or a dynamicSupervisor that creates a genserver process for each “session”

idi527 · February 2, 2018, 8:32pm

would have to be A LOT of reads

It just said “service” in its description, so I assumed it gets a lot of reads … There is a library, goth which keeps it’s config in a genserver, and it was a bottleneck for me. And I didn’t have A LOT of reads. I used it to generate signed urls for google storage and on each request there were two calls to a single genserver (Goth.Config)… I first moved it to ets, and then to compiled modules, since the config rarely changes. The performance gain was quite significant. And it also seemed cleaner, but that’s subjective.

That or a dynamicSupervisor that creates a genserver process for each “session”

ets approach would be simpler to implement and probably more efficient.

jordiee · February 2, 2018, 8:41pm

I will probably look into ets or Registry(still ets under the hood) this weekend or next as the logic for the genservers got a little messy anyways.

entone · February 2, 2018, 8:44pm

Almost all of them will use Cowboy/Ranch under the hood. Might be a good place to start. https://github.com/ninenines/cowboy

subetei · February 2, 2018, 9:16pm

Hello there
I learned elixir in a similar way by grinding away and trying things on genserver so I may have a couple useful examples. For a bigger app, here is a section of code in my data project that has several supervisors and genservers for solving a simple need of having a cache with very specific expirey logic (activity counts in a rolling time window):

You can find the the full web project and such easily from there, it has a lot of code ranging from high comfort level down to first learning or powering through objectives to get somewhere.

And here is a simpler and very small library that just handles keeping simple site statistics, that happens to have a genserver implementation in case of wanting to hold stats in memory instead of a database.

Hope they help. I can post code snippets of Flow usages that I used at work recently as well if your interested. I’ll do my best to answer any questions if you do take a look.

And welcome to elixir world

synthslave · February 2, 2018, 10:16pm

Wow! Thanks guys @subetei @entone @jordiee @idi527 !

Really appreciate the help

synthslave · February 3, 2018, 1:59am

@subetei

Hi again, could I take you up on your offer of screen shots of Flow in action when you have a minute or two?

Thanks and have a great weekend all!

bibekp · February 3, 2018, 7:30pm

I don’t know if this is what you’re looking for but here’s an example of Flow from over a year or more ago.

Flow is used to ingest hangman dictionary words in parallel and puts them into ETS while computing statistics…

I just got around updating a bunch of my elixir projects with the new mix format and remembered about these files. Theres also tons of OTP in this particular project. Have fun… Bibek

github.com

brpandey/hangman/blob/master/lib/hangman/ingestion_first_flow.ex

defmodule Hangman.Ingestion.First.Flow do
  @moduledoc """
  Module implements the introductory ingestion flow logic given an initial 
  hangman dictionary file.  Serves to split dictionary into windowed word list
  chunks storing them in intermediate cached files arranged by word key length
  """

  alias Experimental.Flow
  alias Hangman.Dictionary
  require Logger

  # Use the number of keys as the same number to partition our dictionary words
  @num_stages Dictionary.key_range() |> Enum.count()

  @doc """
  Run reads in dictionary words file, and checkpoints snapshots of so many events, 
  e.g. 500 events for each partition.  Each checkpoint snapshot is written to the
  corresponding word key length dictionary partition file.  E.g. partition 0 data
  is written to key file 2, partition 8 data is written to key file 10.  Data
  is written in a way so that it is easy to extract the key (word key length) and

This file has been truncated. show original

github.com

brpandey/hangman/blob/master/lib/hangman/ingestion_cache_flow.ex

defmodule Hangman.Ingestion.Cache.Flow do
  @moduledoc """
  Loads partitioned dictionary word files into ingestion table
  piecewise through words list chunks.  Random words are generated
  from these words lists.

  Letter frequency counters of the dictionary words 
  are arranged by length and also stored in the ingestion db after the 
  words are stored

  Optimization Note 1: Generating counter tallies outside of 
  the ingestion table allows us to use Flow's parallelism to generate these

  Optimization Note 2: Converting word_list chunks to binaries
  and counters to binaries drastically reduces table memory footprint
  """

  alias Experimental.Flow
  alias Hangman.{Counter, Dictionary}
  require Logger

This file has been truncated. show original