Graph Dashboard with Phoenix channels for multiple graphs sources on one "page" - Grafana like

[tomazbracic]

I would like to get some help / guidance on how to build more complex web dashboard something similar to “dashboards” in Grafana. So, for instance, having 1 page (dashboard) with multiple > 20 graphs of different types, like single stats, pie chars, bar chars, etc, etc.

The dilema / questions that I have are related to my newbie approach and understanding. So for instance.
You have various sources of data, pub/sub, remore REST calls, timeseries databases, Ecto (Postgres), etc, etc.

Now in Phoenix router you usually define something like “/somepath, someController, some action”
So lets assume that /somepath could be /dash1, some controller can be dash1_controller.ex, and some action… here I get to a problem. Should and how… it doesn’t seems right to me… but should I get (how) all data for > 20 graphs from various sources and prepare them under 1 controller function/action?
That would mean enormous / huge data structure passing it at once? over the view to a template and then there decompose the data structure to various graphs that would need it? or what?

And I didn’t even mention Channels configuration/setup.

I find really hard to understand this, since all the examples and tutorials are showing more or less just something simple like bar chart with 2 metrics, and one controller, one route… but this is usually far from real life example/situation.

And I assume that having good structure and approach on a project base / file base level is crucial for bigger projects and its maintability.

So , I would really appreciate some guidance, suggestions here.

Thanks in advance.

[kip]

I think I would start this way:

1. What is the right functional core for managing a data source, retrieving data and streaming it to a consumer

2. What is the right model for converting a data stream into a visualisation (ie some kind of chart or graph or map or table or …).

3. What is the right model for defining component layout on a front end - ie where does a visualisation go on the page, how big etc etc

4. How do I define a component that connects to a data source and transforms it into a visualisation

Only then would I worry about how I plumb it all together. In this case I’d probably find my self with a simple HTML page that has several “containers” on it which are each configured to show a visualisation of a given data source. And each container would access that streaming data source itself via Ajax or web socket or whatever.

[tomazbracic]

Thanks for your answer…

Ok, sorry I thought that I was clear in my post.
Let’s assume that questions 1, 2, 3 and bit of 4 are already answered. A bit of 4 because I think this one actually touches my dilema. In regards to how to approach this from the point of Router, Controllers, Actions.

Based on the last paragraph …sure. I though of this “containers” like too. I could transfer everything into Phoenix API and build front-end with Elm or React… and as you suggested treat each of the “graphs” as stand alone entity… and change its data with AJAX or throught Phoenix channels.

But if I would proffer staing within Phoenix template system and without some other frontend technology… and for the sake of debate leave out LiveView for now… what would be the suggested approach.
Or maybe saying this a bit differently… lets assume that my “context” part is done… .and I have from various sources. What would be best approach to “push” this soft realtime data to the page (channels, ajax) in regards to how to pass that data from controller…is it just passing from one action really big number of variables into the template? or doing it differently? And I guess this is connected to the question of how to set this up in Router since I would assume that I would go /use URL like… https://mydomain/dash1.html where all my > 20 graphs would live.

[tomazbracic]

Hey… perhaps for better understanding of my concerns. This admin template is one random from internet. Just for the sake of better understanding.

http://codervent.com/bulona/demo/index.html

So we could say that top 4 squares could or not be taken care by one controller action. Or might even not, if based on the context data sources would be different.
Then you have this “site traffic” section, then “weekly sales” and so one.
I can’t really imagine that one controller action would gather all the data for this whole dashboard.
And there are more graphs on this page.

So in router.ex

I can’t really do this

"/graph1 (of many), Graph1Controller, :“some action”

And then if data would not be related to one specific controller … etc, etc.

OR… this is really not best approach. And better solution would be to convert my Phoenix to Phoenix API and decide on some front end technology like Elm, React… and do the approach as you mentioned already… isolated “containers” and each of those can then use, connect differently to the API?

[Exadra37]

Maybe try to load the entire example.com/dashboard as a skeleton where each graph is a live view container that as an initial state.

So when you hit example.com/dashboard and the browser finishes to render the page the live view containers will call the backend, and in the mount function for each live view you can use Phoenix.PubSub to subscribe to the channel that is responsible to feed the graph with data, and you graphs will become live as soon that channel start broadcasting data.

[kip]

I think you would have one endpoint for each data source - because I wouldn’t design it with the assumption that the data sources are all consolidated through a single server and I would want to decouple that as much as makes sense.

If course that doesn’t mean they couldn’t be on one server, just I would not want that to be a requirement.

Then I’d use the Phoenix router to dispatch however I need it to for each data source. For a “traditional” approach the following would be one way. The alternative “liveview” approach would be similar at the routing layer but different in other ways. And you use channels of course thats different too.

The point being that the client page has one or more components that each access the data they require via whatever endpoint that data lives on. And routing takes care of dispatching as you need.

defmodule HelloWeb.Router do
  use HelloWeb, :router
  pipeline :api do
    plug :accepts, ["json"]
  end

  # send data, and render on the client
  # or use the browser pipeline and send HTML
  scope "/", HelloWeb do
    pipe_through :api

    # Use routing to decide where to dispatch to
    get "/component/data/special_source", SpecialDataController, :show
    get "/component/data/:source", DataController, :show
  end
end

[ondrej-tucek]

I think that kip has right and it’s a mistake avoiding these points. Because from here you can make a choice what it would be used.

I mean, if data in dashboard must be refresh each hour, why not use AJAX/Phoenix Channels/… + Elm/React/… But be aware, this is a difficult road (from my point of view). You have to serialize/deserialize data, routing,…

Currently we are developing sw for fuelcell control . There are here a lot of sensors and they can measure each 1 ms/sec/min/… and also we want to show their aggregated data. We started with Elm on frontend (1 and half year ago) and Elixir (+ Nerves) on backend side. So we were sending these data via websocket. But as I said a lot of work with (de)serialization data, routing…

Now thanks to LiveView you can make a dashboard very fast. See screenshot below. This is after one week work, not finish, still WIP, a few graphs are missing. I’m currently writing a fake sensor data generator via GenServer and I hope that on the end of this week I would deploy it to heroku.

What would be best approach to “push” this soft realtime data to the page (channels, ajax) in regards to how to pass that data from controller

If these data are changed each …/sec/minute, definitely I would choose the websocket.

is it just passing from one action really big number of variables into the template? or doing it differently?

It depends… It’s a pitty that there is no visualisation library in elixir world (there is one, contex ?, but it’s still WIP I think). So you can select from JS, e.g. Plotly (that’s our case, because it’s free) or Highcharts,… and via Hooks join it.
Now you have to look at if there exists some function like "extend" or similar which knows how to keep old data and newly incoming join it. So you haven’t to send whole updated list of data but only one point.

I guess this is connected to the question of how to set this up in Router

In our case it’s just live "/graphs/actual", Graphs.ActualLive where are mounted stream data from fake generator. In real case it will be connect db instead of fake generator here.

[tomazbracic]

Thank you all of you guys for all your help and suggestions. I will try to proceed based on you r suggestions.