Is anyone interested in writing general forum / CMS software in Elixir + Phoenix? Do you by chance know of any existing projects? Thanks.
but it has not had any github updates in 9 months
3 Likes
I thought about writing a CMS in Elixir but there is a not so trivial problem:
Every CMS should have a plugin system (my opinion) and that plugin system should be manageable during runtime so the admin user can search, install, update and delete plugins while the application is running. But how do you do this in a language that uses a long running process and also needs to compile code? It’s tricky. You also have to take care of database migrations too (because plugins might need to store data). I am sure with a bit of magic trickery, you will be able to get something together, but how well does it work in the end?
There is a reason most CMS and even eCommerce systems are written in PHP. PHP makes those dynamic plugin systems quite easy because there is no long running process and no compilation involved.
Now you can say that people should just add a mix dependency and restart the Elixir process, but that is far from enduser friendly.
5 Likes
I was super optimistic about Firestorm in contrast to the incumbent for modern-day forum tech, Discourse, which is what Elixir Forum uses. Discourse is fairly pleasant and snappy as an end user, but it is a chore from an operator’s perspective because it’s Rails trying to do soft-realtime, and with a strange, bespoke deployment workflow that manages to undermine the value/repeatability/standardization of Docker-based deployment almost entirely. So much of the hand-wringing in that ancillary project is unnecessary in a BEAM environment, not that we don’t have our own wrinkles around deployment.
I’m disappointed but not surprised that Firestorm has fallen idle. Most crowdfunded, change-the-face-of-niche-software projects seem to do that. It never reached an especially usable state, the last time I checked in on it, although kudos to them for at least attempting to dogfood it.
4 Likes
It’s on my long list of things to do, fork firestorm and finish the todo list 
3 Likes
Wouldn’t it work well if the CMS let those plugins communicate through hooks, which would allow the core application to run without a care in the world while the admin would add and remove plugins. It’s could be way safer as well, depending on how it is done.
As for the database, it would be a huge pain to let the app and plugins to share the same DB. IMHO each plugin should have its own DB so that removing them is always painless, and when plugins need to VIEW some data from the app DB it should do so via the available hooks.
3 Likes
But how does the plugin get installed? It needs to be pulled from somewhere, compiled, and started.
Have you looked at existing solutions? That’s how it works.
That’s not enduser friendly or manageable at all. Imagine you have 20 plugins installed, you would need 20 databases.
IMHO, what you suggest is even worse than adding a dependency and restarting the app, from an enduser perspective.
1 Like
Plugins could either be coded in a scripting language to remove such inconveniences, or have a standard way to do what you said that they need to conform to (could be one plugin <> one docker container), in which case it could be made automatic as well.
I have not, but working on a project in PHP I am not surprised that everything is spaghetti code, including the way the DB is handled.
In which case can a few tens of DBs be a problem?
It’s much simpler to manage as each plugin creator has to manage his own DB, without side effects, without conflicts with other plugins, and etc.
It’s also much simpler for the website admin, as removing a plugin won’t fuck up anything EVEN IF the plugin creator created a mess. Updating would be a breeze, and downgrading wouldn’t break the DB half of the time.
With the usual plugin systems, any plugin installing, upgrade and downgrade can break everything, especially in the DB, and they can steal all the info they want too. I dunno, but as a end user I wouldn’t feel safe (and I guess that’s part of the reason why I have always avoided CMS).
For a forum I’d probably do it as a set of elixir/erlang/etc files, the BEAM can compile directly (and so can elixir) so you can hot-load easily, they should just be their own applications that register hooks to be called. With that you could implement scripting languages as well if you wanted.
Agree, it’s not hard to have them share.
And they wouldn’t be able to share data effectively.
Well look at Discourse, this forum, it is a pluggable hook-based style.
1 Like
That doesn’t make any sense, it’s not the right approach. I’m not a database expert, but I think this will hurt performance so badly, as database’s connections are expensive. It’s totally fine if you need a table for each plugin (like most of the existing solutions), but not a different database. A typical structure may look like this:
- CMS
|- Plugins
|-- Id
|-- Name
|-- Description
|-- Active
|-- Roles
|-- ...
|
|- Plugin_News
|-- Id
|-- Title
|-- Body
|-- Plugin_Id
|-- ...
Another way is to use a “manifest” file to define the metadata for a plugin, so you won’t need to store its information in the database. However I’d say that the database approach is much better, in the end most of the plugins may need a table to store some kind of information, you can also mix both approaches like Orchard CMS does.
Ignoring remote installation of plugins (ugh, that should be very complex), but an alternative solution instead is to distribute a set of plugins with the app and enable or disable them through the admin panel using the database to store its state, that should be enough as a starting point. Next you can think how to implement a more complex solution, but your project doesn’t need to start as a full-featured WordPress, have in mind that it has more than a decade of development.
Firestorm is a very interesting piece of software and the concept itself is worth exploring, but I’m not sure this is what I’m after.
@Phillipp I’m still relatively new to both Elixir and Phoenix but I don’t think a plugin system is something that would be hard to implement. No matter the language, I would design a production grade CMS to accept signed packages from the admin system with the optional override with a huge warning. Each package would have a standard format for installation, migration, upgrading, and deletion whether it be local or remote.
As for handling plugin compilation, I need to explore OTP more, but I don’t see why this would be a problem. Could you be more specific as to why you couldn’t use a newly installed plugin?
Related: Does hot code-swapping come for free?
You raise a good point about PHP. Maybe I’m just too naive to the drawbacks of Elixir + Phoenix, but I really don’t see the issue here. I’ll have to explore some strategies and will hopefully get back to you soon.
@shanesveller Discourse is certainly better than most if not all forum software but it’s not something I would wish to install and run myself. A conversation organizer might be a more accurate description of what I might attempt to build.
In my opinion, very few people need Docker, and I also don’t like the idea of downloading entire environments. Install Nix package manager, install it on an Arch machine, and call it a day.
Related: https://nixos.org/nixos/nix-pills/
I wasn’t aware of Firestorm being crowdfunded. Do you feel the team delivered or was it a pump-and-dump of dev that you normally see on Kickstarter?
@frigidcode With the latest updates to Elixir, Phoenix, and LiveView, do you think the effort of a fork is worth it? Sometimes re-inventing the wheel helps you understand the pain points that eventually lead to a better solution.
@Cochonours I’m not so sure about the idea of hooks. I understand that method is popular with Wordpress but I don’t like the concept of anything from anywhere being able to alter anything. I would try to keep it more consistent with concept of Phoenix pipelines if possible. Hopefully you would agree that a good plugin system wouldn’t require any code modifications outside of maybe a configuration file. Are you aware of lesser known alternatives to the hook-style system?
Although it sounds super cool to allow plugins to communicate with each other, I would consider this an anti-pattern. Unless you’re designing a set of plugins that need to communicate with each other, there’s no way this isn’t a debug nightmare. Do you know of a scenario where this would be considered a good idea?
One DB per-plugin doesn’t sound ideal to me. I’ve been a fan of database.schema.table for a good decade or two. In your production grade applications, do you often need an entire database for a single feature? Normally you would only do this if the DB was shared with other applications and those applications existed before your requirements. I can certainly see why there might be headaches in terms of permissions. I’m not saying it’s the most practical thing to do, but you could probably have two different DB users, being a general CMS user, and a plugin CMS user. Basically the plugin CMS user wouldn’t be able to touch certain schemas or would have read-only access. I’m also not too worried about the setup and cleanup of a plugin because I would design a system that would only accept a single standard that addressed both migrations and table ownership. I would go as far as programmatically creating a new DB user for each plugin and lock that user into a schema created just for the plugin. The only real hurdle is being able to get the permissions correct and being reasonable about how to adjust them for a non-technical user.
A plugin being coded in a different language sounds like a nightmare to me but I’m open to new ideas if you have some candidates? You’re basically describing FFI from what I understand. If I were to take this route, all plugins would have to be written in this language. I would only build a system that supported a single language simply from a maintenance perspective.
Related: https://github.com/clojerl/clojerl
@OvermindDL1 A hook-based system might’ve worked for Discourse, but it’s also written in what I believe to be a less powerful language. I’m not sure what system I’m going to come up with or if it’s even going to be successful, but for sure I think we can do better than hooks and events.
@adrianrl I agree with you about DB performance. You also have to think about connection pools and I don’t think having a connection pool for an example of 20 plugin-specific DBs is going to be something I’m willing to do. Maybe there’s a good reason to do this?
2 Likes
I wasn’t meaning to add hooks every where so the code becomes a nightmare to understand, but indeed taking advantage of pipelines. One still needs to define where the plugins can inject their functions (in the pipeline, before rendering, after rendering, …), I don’t see a way around that, and every new hook like this makes the whole more difficult to understand when a bug appears.
I’m with you on this. It’s a trade-off but making them orthogonal would prevent a lot of headaches and nasty bugs. There is always be someone who wants to alter the functionality of another plugin in a way that wasn’t designed, but you could scratch that.
One DB per plugin is just the easiest/quickest way to make it work while ensuring total isolation and programming ease, but any other ways like you described would be acceptable. Your app will have to handle all that new user in DB + setup all rights + creating the table and vice versa when the user pauses or uninstalls the plugin. My idea is only to be able to ensure that plugins cannot alter in any way data that do not belong to them, and let the plugin writers in charge of their own little migrations orthogonally. Non technical users shouldn’t have to care about the how.
I’m used to this kind of thing and it’s not so bad, but indeed as other said it seems to be easy to just keep it all in elixir. Keep in mind that allowing several languages with a (strict) API allows people from various backgrounds to write or import plugins, which could be a plus. It’s also a minus as the bar of entry will be lower, so…
We are talking about a few DBs, at most a few tens of DBs (which would already be a lot of plugins installed), so keeping those connections open is peanuts. It’s better performance wise to have one table per plugin and let the plugins use joins to other tables from the app DB, but in that case it’s quite a lot more complicated to handle data access and security. If you don’t allow joins and let the plugins only use their own table the performance cannot be better than one DB per plugin (as in this case each plugin can use the type of DB that is the best for the job).
Is that really a plugin system anymore? The app devs have to carry the burden ad vitam, and the probability of spaghetti code becomes much higher, making it impossible to remove/extract a plugin a few years down the line.
I don’t know if I understand what you’re trying to say, but querying another database implies joins too, you may want some kind of relationship between the plugin and the app.
Why not? It just works and it’s more straightforward to implement. The problem here is recompiling the whole application in real-time for every installed or upgraded plugin, think about a new installation where you may install a bunch of plugins to check if they are useful for your project, not only plugins but also themes, I think it’s not worth it. What would happen if an installed plugin tries to call obsolete APIs or isn’t working correctly? Big disaster here.
Why spaghetti? Think about this structure:
CMS_web/
plugins/
plugin_news/
plugin_seo/
plugin_stripe/
And then:
defmodule CMSWeb.Plugins.PluginNews do
# ...
end
As you can see each plugin lives in their own module, so this kind of organization should be enough to avoid repetition. I worked on a custom CMS system for two years or so in PHP, and we never consider that we had spaghetti code, even using OOP that sometimes is totally a mess of abstractions.
1 Like
Now that LiveView exists and will be improved even further in the near future, I believe the time is right for a Phoenix-based forum software.
It will be no small task however and realistically I don’t think anyone will tackle it without several years of guaranteed salary.
1 Like
Hm… Milestone bounty?
Wouldn’t that just be retreading ground covered by incumbent solutions already? Seems the current opportunities for innovation are on the client side (web), e.g.:
- Store recent, bookmarked topics and posts for offline viewing
- Queue posts in offline mode for later dispatch when connectivity is available.
- Updating of offline topics with background-fetch
- etc.
Of course that is going to require a server side API that can support that style of distribution.
1 Like
In my opinion it has more to do with shared hosting. PHP (and MySQL) have been included with just about every shared hosting plan on the planet and is IMHO the main reason behind the popularity of PHP in general. I used PHP (and still need to) for years to that very reason. It isn’t until you actually need a long running process that you move on to greener pastures, like Python, node.js, and (now for me) Elixir.
There is a project that was started a while back but it hasn’t been updated in a good while: Contento
4 Likes
Technically yes, in practice no. Older versions of PHP – those still dominating usage on I’d bet most hosting services on the planet – have a number of well-known security vulnerabilities, not to mention that some of their “frameworks” allow SQL injection to this day.
IMO we need a second wave of commoditization of the web for non-technical users. Not sure there are technical stacks that do this perfectly but Erlang / Elixir are very close – although the BEAM doesn’t support sandboxing and built-in rate-limiting and/or slices of computing resources (that can depend on which user is on which paid plan, for example).
And websites die or get removed from Google’s index every day. That definitely wasn’t sir Tim Berners-Lee’s idea for the web and he said as much many times.
The average website needs to become very lean and mean if we want not to end up with the big 5 corporations hosting all the internet (which many people like myself would work against with all our will and skills… if we had the free time, or were paid for it).
Which brings us to…
That’s very true. I personally I’m unwilling to participate there though. The JS ecosystem is still juvenile at best – the committee cannot even agree on a standard library – and my time is better spent with smart people recognizing where evolution is needed.
1 Like
Here is another forum with GraphQL:
