How to determine contexts with Phoenix 1.3

@imetallica Thanks. I have started studying those topics now. GenServer is making sense now.

I don’t know if you can write a sample on how PubSub can be achieved.

I’m not sure if that’s what you need, but have you checked Registry module in elixir? It has a note about how to use it as a local pub/sub system in the docs [0].

You can put it in a separate app under an umbrella and let processes from other apps subscribe to certain topics in it.

apps/pubsub/lib/pubsub.ex

defmodule Test.PubSub do

  def start_link do
    Registry.start_link(:duplicate, __MODULE__)
  end

  def subscribe(topic) do
    Registry.register(__MODULE__, topic, [])
  end

  def unsubscribe(topic) do
    Registry.unregister(__MODULE__, topic)
  end

  def publish(topic, message) do
    Registry.dispatch(__MODULE__, topic, fn entries ->
      for {pid, _value} <- entries, do: send(pid, message)
    end)
  end
end

apps/pubsub/lib/pubsub/application.ex

defmodule Test.PubSub.Application do
  @moduledoc false

  use Application

  def start(_type, _args) do
    import Supervisor.Spec, warn: false

    children = [
      supervisor(Test.PubSub, [])
    ]

    opts = [strategy: :one_for_one, name: Test.PubSub.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

To subscribe for updates in user app in a phoenix channel you would do something like

defmodule Test.Web.UserChannel do
  use Test.Web, :channel

  def join("user:" <> name = topic, _params, socket) do
    {:ok, _owner_pid} = Test.PubSub.subscribe(topic) # subscribe to updates in "user:#{name}"
    {:ok, socket}
  end

  # and handle these updates
  def handle_info(:sacked, socket) do
    broadcast! socket, "sacked", %{}
    {:noreply, socket}
  end
end

Somewhere in the user app you would publish the message :sacked in case of a ban

...
Test.PubSub.publish("user:idiot", :sacked)
...

These are basically the examples from the Registry docs, and there are some more about other possible uses for it.

Note however that this pub/sub implementation is local. And if you are using phoenix, you might want to pick Phoenix.PubSub [1] instead which is not local, so that for the example above (with channels) you would probably use phoenix’s pub/sub library.

[0] https://hexdocs.pm/elixir/master/Registry.html#module-using-as-a-pubsub
[1] Phoenix.PubSub — Phoenix.PubSub v2.1.3

Using Phoenix’s PubSub is better for this because it can broadcast messages between nodes, either using Erlang’s pg module, Redis or RabbitMQ. I think there’s a Kafka as well, but I’m not sure.

2 Likes

@imetallica, @karmajunkie, thanks for your replies. I guess I need to do more exercise on Elixir/Phoenix, including PubSubs. thanks again!

Can you give an example how that would work with phoneix pubsub? I have searched the forum but couldn’t find one.

Ideally, I wish I could put phoenix pubsub in it’s own app apps/pubsub and point apps/web to it.

config :web, Test.Web.Endpoint,
  url: [host: "localhost"],
  secret_key_base: "...",
  render_errors: [view: Test.Web.ErrorView, accepts: ~w(html json)],
  pubsub: [app: :pubsub] # where :pubsub is an app in umbrella

The problem I’m having with pubsub being in apps/web is that I can’t be sure that Test.Web.Endpoint is loaded when I make a call to it like Test.Web.Endpoint.subscribe from some other app like apps/user.

And if I am to use Phoenix.PubSub.subscribe from inside apps/user, then that (I think) requires me to add phoenix_pubsub as a dependency to apps/user.

Now that I think about it, it might be better to leave phoenix pubsub to deal with websockets and the like, and either write a global distributed registry/pubsub myself or use something like swarn [0]. Not sure though.

I am also not sure how some things are implemented in phoenix pubsub (I’ve been planning to read it, but I’m afraid I don’t know elixir well enough to understand it fully), for example, does it batch messages when passing between nodes or not? Because it seems it can be somewhat of a bottleneck [1] if every message is sent with send.

[0] https://github.com/bitwalker/swarm
[1] http://www.ostinelli.net/boost-message-passing-between-erlang-nodes/

Something on the following should work:

def MyApp.MyPubSubListener do
  use GenServer
  alias Phoenix.PubSub

  def publish(what), do: PubSub.broadcast SomeName, "channel_to_send_receive_messages", what

  # Callbacks

  def init() do
    PubSub.subscribe SomeName, self, "channel_to_send_receive_messages"
    {:ok, %{}} # The initial state of this GenServer.
  end

  def handle_info(message, state) do
    #  ... do something with your message/state ...

    {:no_reply, state}
  end
end

That should do the trick. Do not forget to start it in a supervision tree of sorts. And remember to create one for each channel/topic you want to subscribe.

So what happen when you want to change the database user schema? Since individual user schemas are all over the contexts and you don’t have a single point of truth, will that become a headache when you introduce changes to these denormalized schemas?

Posted by @michalmuskala earlier:

http://michal.muskala.eu/2017/05/16/putting-contexts-in-context.html

1 Like

I believe that it is important to delineate the difference between a normalized data model, a denormalized data model, a document-oriented data model and a segregated data model.

Normalized: This is the ‘old school’ approach to creating relational database structures, where fields are stored in separate tables and relations between them (such as JOINS) are used to gather the data combination(s) your application might desire.

Denormalized: Opposed to a normalized relational database, certain pieces of data are replicated in multiple tables, making sure that no inter-table queries are necessary, resulting in faster read speeds, but slower write speeds and a more complicated structure for the developer to maintain (making sure the copies of the data all stay up-to-date).

Document oriented: Every ‘user’ might have different fields in the database, and care must be taken to handle missing fields. This is the storage approach that some NoSQL databases take.

Segregated: Here, there is no single point of truth, but this also means that there is no ‘the database user schema’. The different contexts store different aspects of your user, which are referenced by the same symbolic identifier (which might be a UUID, for instance). The outside world does not care about the persistence layer that the different contexts use to store/retrieve their information from. We therefore might be talking about multiple different kinds of databases alltogether!
Of course, when your contexts are separated in a logical fashion (note, YMMV; exactly how to structure your contexts is very application-specific), there will be one context handling general user info, one context handling e.g. the set of products the user manages, one context handling the forum that is part of your app, etc. This means that only the ‘user context’ needs to know about most changes in your user, with the other contexts only knowing about the user’s UUID, and maybe from time to time asking the ‘user context’ if the specific user is authorized to perform a certain action.

So, to answer your question: If your separation of contexts is logical, then this will not become a headache.

Now I look back, I may or may not have expressed myself correctly. My question could be rephrased to “will that become a headache when you introduce changes to these fragmented schemas?”

But your reply kind of also answered this question already. Thanks!

I still have some confusions about contextual schema. Guess it’s quite off topic, so I post a separate question at Contextual schema VS out-of-context schema (in Phoenix 1.3)

1 Like

@chrismccord Thank you and the rest of the contributors for all your work. Elixir / Phoenix is what got me a taste of FP, and I’ve been trying to learn that style of programming ever since.

Now, on to my actual contribution to this thread. I got very lucky when I started learning Phoenix to have happened on Lance’s Phoenix is Not Your Application talk before I even knew what Phoenix was. (Not really but you get what I am saying). I keep harping on Lance’s talk as well as Wojtek’s talk on umbrellas. Because I built my app this way, I can update to Phoenix 1000 without breaking a sweat.

So, personally, I think I would always go the umbrella route. (And I gotta check this --umbrella flag you mentioned). Nonetheless, I commend the Phoenix folks for taking on a lot of work, likely a lot of flak, and nudging end-users to move closer to the umbrella ideal.

1 Like

@karmajunkie Thanks for the example you gave. Can you provide a simple sample in GitHub that less experienced programmers like us in elixir can look at?

For me, the truth is using phx is more difficult than Phoenix.

I have not been able to grasp things like handling relationship across context.

I live in my own world here in Nigeria and I have only found one person other than me using elixir. Samples can make things easier for me

Of importance is how to handle relationships between models.
How do you handle 1:M relationships?
What if a model in a context say A requires certain data available in another model in a different context B?
How do you ensure data integrity, such that changes made to a model A in context A is propagated to model A in context B?

Please your prompt response is highly appreciated

What if a model in a context say A requires certain data available in another model in a different context B?

There’s no correct answer to that question. It’s just the perfect place to ask yourself, why you need to cross those context boundries.

E.g. if one context is the help desk and another one is the sales department then it’s quite strange that one does need to depend on a struct of the other. In that case I’d add a new schema to the help desk requesting just the data the help desk does require from the db (multiple schemas of the same table are no problem in ecto).

If one context is the checkout context and the other your auth context holding the current user and it’s shopping cart, than it’s a relation, which feels quite natural. I personally wouldn’t split that up into multiple context based schemas.

It has to be weighed if it’s better to add more code, but have the ability to evolve both contexts independently (probably the better choice for bigger projects / bigger teams) or to keep things lightweight and use some relations beyond context boundries if they make sense.

How do you ensure data integrity, such that changes made to a model A in context A is propagated to model A in context B?

With ecto it’s first and foremost the db. If you query your tables you’ll get the updated data of them. As mentioned above you can have multiple schemas handle the same tables. Everything beyond that is in your own hands, like it was before contexts as well (e.g. two simultaneous update requests might overwrite the first one).

1 Like

Your first mistake here is in using the database as an integration layer between different contexts—all you’re doing is coupling together two things which are explicitly meant to be decoupled. If you’re going to use contexts as a service boundary, then they should be independent. This means that context B doesn’t borrow the data of context A—context A broadcasts to the world what it thinks the world ought to know, and context B maintains its own store of data based on what context A has to say. If these contexts are actually decoupled there probably isn’t much that B is interested in from A.

That’s the job of context B, listening to messages broadcast from context A.

If you’re dead-set on coupling the data in the way you describe, don’t create artificial boundaries where you’re not enforcing a systemic boundary. Just use context A’s schemas in context B. You can have a separate schema that refers to the same underlying database table from a different context; I have proffered this option in the past, but to be honest I don’t think its a great idea—just a way of fitting a round peg in a square hole.

Putting together a good example application that applies this style of architecture and design is not really trivial, and trivial examples just create more questions when you invariably get away from the happy path.

What I would recommend for you is to ignore the notion of contexts for now. Elixir doesn’t actually care about them, and Phoenix really doesn’t either outside of the generators. Learn all the other aspects of Elixir, OTP, and Phoenix first—at some point, the context stuff will just come naturally. Its also good to do some reading around analysis patterns—there are many ways of approaching architecture, and DDD isn’t the best one in every situation.

Why getter and setter methods are evil (Allen Holub, 2003)

Don’t ask for the information you need to do the work; ask the object that has the information to do the work for you.

Replace “object” in the above statement with “context” and it’s still accurate. These Days Holub phrases this as “Ask for help, not information”.

A little bit earlier Andy Hunt and Dave Thomas talk about the “Tell don’t Ask” principle in:
The Art of Enbugging (pdf, IEEE Software January/February 2003) and
Tell, Don’t Ask

Martin Fowler summarizes his take in TellDontAsk

“Tell Don’t Ask” is a consequence of Responsibility-Driven Design.

Practicing Ruby: Responsibility-centric vs. data-centric design (2012)

  1. Objects tend to be highly cohesive around their behavior, because roles are defined by behavior, not data.

In “context-speak”:

Contexts tend to be highly cohesive around their capabilities.

Essentially, if your thinking is even somewhat rooted in the CRUD world your are going to have a hard time isolating contexts.

5 Likes

Essentially, if your thinking is even somewhat rooted in the CRUD world your
are going to have a hard time isolating contexts.

Spot on! I’ve noticed this design “problem” ever since the first MVC web
frameworks popped up. They are all quite centered around the database
table (the model) and generators and documentation show how to have 1
“Context”/“Api”/“Boundary” per table and then use them directly from
your code.

This goes through in everything and even form generators are done from
the model. Following this forces you into a design that is not good for
you and hard to get out of. It is also hard to change how you reason
about it.

Generally determining contexts is hard unless you know your domain well,
but you should not base it on your database tables, rather think what
your software is trying to solve and what service it provides.

In terms erlang everything is based around modules (which is true for
elixir as well). In erlang the “best practice” is something like this:

A module should be coherent and loosely coupled. A module should have a
small, well defined and documented API.

An OTP application is a set of modules. The OTP application should have
1 module which is exposed to the “outside”. It should have a small,
well defined and documented API. No implementation details should leak
outside of the application. That is, if you return a data-structure that
data-structure should be opaque and only modifiable by using provided
functions.

Modules within an OTP application can depend on each other, they are an
implementation detail.

I see contexts as something similar. A context should have a small, well
defined API and no implementation details shouild leak outside of the
context.

If you have a small, API you should easily be able to swap out the
implementation of a module/context/OTP application without affecting any
other code.

7 Likes

This is an old article (2009) that relates to a very specific use case. AFAIK this does not even apply anymore.

AFAIK this does not even apply anymore.

So the messages are batched by default when sent to other nodes? I thought not, since there are some projects like GitHub - discord/manifold: Fast batch message passing between nodes for Erlang/Elixir. which explore approaches to do that similar to what you’ve described in your article …