Drab: client and server side unified (the idea and the proof-of-concept)

Hi guys,
nowadays building the web-based application (more complicated then the rails scaffolds) is in real building two apps: client side and server side. And even if you are a master of AJAX and JS, there are still some inconveniences, like providing a feedback to the browser on some events, which occurs on the server: long running process or task finished, or wants to communicate to the browser with message “hey! I finished 30%!”. Normally, you need to pull every few seconds to provide the update. Baaad.

There were a few attempts to unify server and client side, like Nagare or Volt Framework. I really liked this, so why don’t we add something like this to Phoenix? Phoenix/Elixir is a dream machine. Let’s give it some more possibilities :slight_smile:

Back to the topic: Drab is the idea of manipulating client-side DOM objects on the server side. It is providing jQuery-like functions directly to the Elixir. Look at the code below - after each step of some process the server updates DOM object on the client side. No need to poll every few seconds to update the process status.

def perform_long_process(socket, dom_sender) do
  steps = MyLongProcess.number_of_steps()
  for i <- 1..steps do
    MyLongProcess.perform_step(i)
    # update the progress bar after each of MyLongProcess steps
    socket 
      |> attr(".progress-bar", "style", "width: #{i * 100 / steps}%")
      |> html(".progress-bar", "#{Float.round(i * 100 / steps, 2)}%")
  end
end

Last, but not least, this approach could eliminate JS as well. Please don’t get me wrong, I am not an enemy of JS :slight_smile: - but wouldn’t it be much easier when you can, for example, disable the button, run long database update and re-enable the button after finish, in the one place? Also it could be killer feature for the new programmers, which are lost in the Ajax and JS world…

There is a not a long way from the idea to the code, so I’ve developed the proof-of-concept page. It contains few working examples with the code samples, and of course it is running on Phoenix+Drab.

Please treat it like a proof-of-concept only. It still supports only a few events and only a few jQuery functions. But if anyone wants to try it yourself, there is a source code on Github with the installation instruction. Installation is not very straight forward yet, please forgive me. Additionally you can find the sources of the proof-of-concept page and the example project - ready-to-go project with Phoenix and Drab already installed, to be used as a sandbox.

Please give me a feedback: what do you think about this idea? Is it worth developing, or the ol’ good AJAX is good enough for everyone and websockets are only for chats?


Grych

8 Likes

So it is basically being able to go back to having the server be stateful control over a single session (but via websocket instead of comet), not bad, but it does not always scale hugely as a server can only be doing so much at once (and holding session information means a lot more data being actively used as you have to hold some form of the client model on the server). I love this programming model personally, but it just does not scale. If the site is expected to always be less than however-much-of-computing-power, then it is fine, but, hmm, nowadays you could probably scale far better than you used to be able to. Need to do research in to it to see how well it works…

Either way, the Intercooler-js library does something similar, but from javascripts side (where the server can send commands over without you needing to javascript up a ton of stuff).

I’d say continue building the library, would be interesting to test these old ideas again. :slight_smile:

1 Like

This reminds me of Armstrong’s ezwebframe. I considered it (though I rolled my own) at some point, but left it because I was skeptical that it’d actually provide all that much of an upside.

I think the real win of it would be if you had a statically typed language that did the change coordination, because then you’d be able to rely on it to not allow stupid changes. All the events would be statically typed, etc., which is what you do get if you go with certain languages on the front-end.

If you’re planning on taking this all the way I’ll be keeping an eye on it, though, because I’m sure it could be more convenient for a lot of stuff.

Thanks for the reply!
The project is in such early stage so I did not think about scaling yet. But why shouldn’t it scale? Does Phoenix+Websockets not scale?

Please notice that Drab does not keep any state or session. Its event handlers are honest, isolated functions. It is very similar to controller actions - but instead of the pipeline connection |> controller |> action we have socket |> commander |> handler.

It does not keep any kind of client model on the server as well. All information are retrieved from the client in a realtime. For example, Drab.Query.val(socket, "#first_name") actually queries the browser and returns the result of $("#first_name").val(). You may think about it as the kind of a database: we can select, update, insert or delete from the DOM tree*.

Of course Phoenix must keep the websocket connection, but if Phoenix scales, Drab scales in the same way :slight_smile:

Anyway, it is always best to check it in practice: now https://tg.pl/drab is working on two isolated Phoenix servers, behind the nginx load-balancer (honest one, without any cheats like sticky cookie). And still works! (it may be stuck for few seconds from time to time, it is running on 8-years old mac mini and sometimes IO blocks everything for a while).

Thanks, I will!


Grych

*) maybe it is a good idea to completely rename DrabQuery functions, so instead of calling Drab.Query.val(socket, "#selector") it would be something like Drab.Query.select(socket, :val, "#selector")? Analogically, Drab.Query.update(socket, :val, "#selector", "new value"), etc etc. It might be more understandable, I believe.

Hi gonz, thanks for the reply!

I am afraid I do not get you. Do you suggest to rewrite it on the top of some statically typed language? Of course, the idea could be implemented in any language/framework, but here and now we are on the Elixir forum :slight_smile:

Essentially, yes, I was saying that I considered implementing this wholesale at some point for my own projects, but decided that it would only really provide much of a benefit if you could guarantee type safety and other things for your changes.

Of course, it’s very comfortable being able to orchestrate changes from your server and it really meshes with the erlang philosophy of “pretend everything else is just erlang (processes)”, so I still think it brings a lot of value, which is why I’ll keep an eye on it. I think some kind of contract system on top of it would be very neat, though, and it’s something you could very likely do with macros, if you wanted to.

That would help guarantee some kind of compile-time checking that would enable some guarantees as to what’s being done with the dom manipulation.

1 Like

Well. We are heading to the old discussion what is better, Xmas or Easter :slight_smile:

Manipulating DOM objects is simply processing the strings*. CSS selectors are strings, values, classes, properties are strings. Notice that Drab.Query does not send the DOM object itself to the server, it sends only the result of corresponding jQuery method (val(), html(), etc) - so it is always the list of the strings.


Grych
*) with the exception of attr("enabled/hidden", true/false), when jQuery expects boolean.

Can you guarantee at compile-time that you are sending reasonable values to your JS client?

No. You can always send a bubble instead of proper HTML. It depends on you, as a developer.

BTW I think we are going off-topic.

My point is simply that with contracts of some kind you can enforce correct values being sent to your JS client. In statically typed languages you get this for free, but it should be possible to implement a layer that verifies stuff at compile-time in Elixir.

If this is not a priority for you I understand that; I’m just trying to propose an improvement on the concept. It’s hardly off-topic, being that it’s an attribute the protocol/application can have or not.

1 Like

In past experience of using styles just like this, it means the server socket process has to hold the state of the client as well, so it knows what to update, where, when, how, etc… It ends up being a lot more data that has to stick around. It scales fine in speed, especially with phoenix sure, but you can run in to two issues. The first of which is memory usage, unless the user of your library goes through pains to ensure to minimize the storage overhead (which can be impossibly difficult in some complex client hierarchy cases), it will eat a lot more memory than otherwise. Second is that you can only have 65k open connections per IP, so you need to buy more IP’s if you intend to have more connections than that or some people’s connections will just not work, but this second issue is a generic websocket issue too, if you plan for more than 65k concurrent connections, you need more IP’s (and probably more computers if the memory usage is too high anyway ^.^). :slight_smile:

Correct, but the users of the library will have to hold state, the style encourages extra state holding.

Yep, but the user of the library will often have to do that.

Hmm, that is different, that would indeed not have to hold ‘data’ the client has, but you would still have to know what to query, which on single pages is not bad, but on SPA’s that would be a lot more to hold then. I could see this style working better than the old one. :slight_smile:

The one thing that is hard to deal with though is round-trip time, that is why client-side SPA’s got popular, to minimize round-trip latency. I’ve made a few little sites in Wt, a C++ (yes) web programming framework that did SPA’s on server-side, basically pretending the browser was just a GUI front-end to send and receive messages from like normal native GUI programming. It was fantastic, but even in C++ it was memory heavy, but that is easily because being a server-side SPA it absolutely had to hold the entire client interface on the server, which is significantly worse (oh but it was a glory to work in, never touched javascript, just pretended it was like a normal GUI window). ^.^

EDIT: Wow! Wt still exists! This was a decade ago that I last used it! Looks like it has a few more web-specific features like built-in bootstrap and such now for theming and both server-side and client-side OpenGL/WebGL, but otherwise still about the same and still as powerful, and still being developed. :slight_smile:

2 Likes

@OvermindDL1, thanks for the update.

Again, the scalability is not in my concern yet, but I think Phoenix can handle this. Remember, we are speaking about the websockets, not the TCP/IP sockets, so there is no limit of 65536 ports by IP. All the connections are coming to the single IP:port.
Actually some guys managed to have 2 million active websocket connections on a single machine. Read here about it.

I don’t think it would be necessary and Drab will never support such programming style.

Not everything has to scale. If you are building internal backends, administration panels, etc, you are expecting at most thousands of user and you can then afford to use programming models that wouldn’t be available at large scale.

In a way, it is great that Erlang, Elixir and Phoenix scales because you can afford to build things that do not and they will work quite well. If your foundation doesn’t scale though then it is always an uphill battle.

3 Likes

They may come ‘in’ to a single port, but when the connection establishes they get assigned a new port. Still the limit. :slight_smile:

Yep I know, they assigned multiple IP’s to their test machine (if you notice the article they had to change phoenix to accept connections coming in to it regardless of the IP of the machine).

That is precisely where things like Wt were useful, fully stateful but few areas. That is also why I keep poking to continue this Drab library, it would save me work in a few backend areas. ^.^

This is really neat!

I created Phoenix originally to tackle ideas along these lines. I was working on my Sync ruby gem, which allowed push-updates from the server w/out js:

So I’m very interested in efforts along these lines. Channels provides great plumbing to enable these kinds of libraries to be built, at scale.

5 Likes

Why? Single process opens listen port and don’t have to open another connection. It knows where to send the return package.
I remember that model from, AFAIR, “Programming UNIX” book, where server process forked to handle the connection while master still listen on the main port. But it was in the XX century :slight_smile: Nowadays you don’t have to do it. And it is good, because no webserver behind the firewall with opened only port 80 would work!

I can’t see any evidence of that in the article. There is no mention that they had to run Phoenix listening on multiple IPs (is it even possible?) or if they had to run multiple Phoenix servers.
They were running out of the ports, but on the client machines (the ones running Tsung - simulating websocket connections). And if you see to the phoenix.xml file they provided, they all connects to some_ip:4000.

Precisely, it listens on one port, when it accepts the incoming connection the accept call spins off and returns a new socket with the 5-tuple set of connection information. My prior posts information was not entirely accurate (I just had my coffee I’m blaming it on) and unsure what all I was speaking. :slight_smile:

The clients most definitely had to have multiple IP’s though, and this part is what is hinted in the article.

It is possible to run it on multiple IP’s (my dev phoenix instance does precisely this). And yes, they needed multiple IPs on the ‘clients’ (which they used multiple servers for anyway since they were low cpu systems), not servers, sorry. :slight_smile:

/me still needs more coffee, woke up to 60F/15.5C degrees this morning, still not warmed up… >.>

1 Like

Hi Chris, thanks for the answer.

I fully agree, and it is very natural to work with channels.

But actually there is one inconvenience which I had to workaround - there is no synchronous message from the server to the client, which I need for querying DOM objects. I had to emulate it with pair of async messages - first Drab sends the query to the client, with its PID tokenized in the message, then it goes to sleep with receive; after this client grabs the result and sends it back to the server with the PID, so channel knows where to send the return value.

After I wrote it works great, but I am still wondering if there is no more elegant way to accomplish the synch message from the server to the client. Maybe should I step down from channels level to the websockets?

Huh, that’s an idea that could maybe be PR’d to Phoenix… It already has a ‘sync’-like call from client to server, perhaps it should have one the other way too. Would just require one more call on public client API and one more call on the server api.

Nah, channels are perfect. For now I’d emulate sync messages be just passing a reference to the client and having it pass it back, sadly have to handle timeout and such manually if the client does not respond and for now, but if something gets added to Phoenix… ^.^