On "Why Elixir?"

[shanesveller]

Thanks for your attention here!

@josevalim

I would love to know what is missing because the only aspect of Distillery we haven’t brought over at this point is hot code upgrades, which you have mentioned multiple times you don’t care for.

Looking at the latest documentation from the Mix hexdocs, I think the only outstanding bits are things that you could have in spirit with native releases, just not in identical forms. The configuration providers have excellent parity already, and the tarball support came not too much later.

• I get a lot of mileage out of custom subcommands over eval and rpc calls, to reduce cognitive overhead during operational or triage tasks. This is also a place where documentation and runbooks could supplement about as well.
• I’ve seen them used for significant evil but book hooks are a nice model sometimes. I expect that the guideline for this is to use configuration providers as much as possible, and for anything else just roll them into your supervision tree directly, perhaps with an :ignore result as appropriate.
• I academically miss the prospect of release plugins for a way for Hex packages to incorporate into your release build more intrusively without a lot of copypasta, but have not actually seen them used in practice so far. Like the boot hooks, I think this probably just evolves into supervision tree entries as well, plus documentation.

I hope it doesn’t seem petty that I say I miss these things in this specific form, but also the Mix.Release module docs and mix task docs leave them out as topics thus far. For most folks, they’d need to be aware of the existence of these features as prior art in order to synthesize the native solutions above from first principles, and part of my stance is that deployments are already hard to teach good instincts for. Not having a canonical reference for these things in the first-party material means I need to say things like “also check out the Distillery docs even though you may not use it, there’s some good ideas to steal from”.

@josevalim

Improved in Elixir v1.11 which will require those to be at least listed in your application dependencies. This will address issues such as poor boundaries between (umbrella) applications.

I’m just as excited for this as I am for runtime.exs!

@josevalim

Even if we consider your three leading entries, I am struggling to come up with a language that currently excels on all three.

Very true, I won’t get everything I mentioned from any one place. I don’t want to turn this into an advertisement thread for Rust so I will just say it already has some traction in most of the spaces I’ve mentioned having an unfulfilled interest in, with ML being probably the weakest and webdev having some ergonomic concerns in its current form.

[josevalim]

@shanesveller

I get a lot of mileage out of custom subcommands over eval and rpc calls, to reduce cognitive overhead during operational or triage tasks. This is also a place where documentation and runbooks could supplement about as well.

That’s something we left out on purpose because it adds a lot of complexity and many people reported Distillery scripts complexity as one of its biggest downsides. I encourage to build regular scripts on top of the main script, which is more Unix-like.

Similar for boot scripts. If you do need to hook into a particular command, then you can use the env.sh file, but I would avoid these particular papercuts.

If there is something that is hard to achieve with those approaches, please let us know, and we can see how to tackle the individual problems. I would prefer to move the discussion from “we don’t have this” to “we can’t do this”.

Plugins are done with custom steps. That’s something we had since day one because they are required for projects like Nerves. All of those are documented and used by the community, maybe what is lacking is actual guides to help people migrate from Distillery.

[LostKobrakai]

@shanesveller

I get a lot of mileage out of custom subcommands over eval and rpc calls, to reduce cognitive overhead during operational or triage tasks. This is also a place where documentation and runbooks could supplement about as well.

I use this for a project since switching to ˋmix releaseˋ from distillery:

GitHub - LostKobrakai/doist

But to be fair for new projects I use eval as in my case CI is the only place to call those commands, so the benefit is minor.

[josevalim]

@shanesveller

with ML being probably the weakest and webdev having some ergonomic concerns in its current form.

I honestly don’t see Rust displacing any of the major players on web dev. It is not about framework capability but rather the level you want to sit at when writing business/domain logic. To be clear, it is definitely doable, but far from first choice.

Similar thoughts apply for distributed systems. Doable from pretty much everywhere but if your goal is to start something, Erlang removes many decisions from the process and it has proven over and over again its scalability and operability.

This is not a Rust critique, just trying to drive home the point that languages won’t check all of those boxes. Rust is probably yet to pass the peak of inflated expectations until it settles into certain domains.

Other than that, I definitely hope Rust succeeds in domains such as ML because its interoperability story means that it will make ML directly available to many other communities.

[henriquefernandez]

Are C, Java, Node or Rust ecossystem “productive and fun” to code as Elixir?

IMO this is one of the greatest advantages of Elixir and maybe we are missing here. I never found before a tech stack that brings me Joy of development and nice performance (in its niche) like Elixir.

Elixir syntax and the language itself is a very nice built.

Do you like “[] === []” to be evaluated as “false” like node? Or to write “public system main args out print wharever pom.xml” like Java? Or to don’t have a REPL/interative shell like Rust? Or to work with pointers while you are building a simple CRUD web app in C?

I really respect all of those. But for Web Apps or Distributed Systems i found nothing “”“better”"" than Elixir.

IMO expressiveness are very very very important thing while modeling business rules. And “it’s a thing” work with a language that not few like you are hiting your head in your keyboard writing a code thats seems like visual noise.

Sorry for my bad english.

[ityonemo]

I work with ML professionals, and my opinion is that are very smart people, and like many very smart people they don’t want to put up with a lot of things on the way to getting what they really want to do. Even if Rust manages to abstract a lot of the most challenging parts into macros, they’re still relatively uncomposable… In this sense: Fighting the borrow checker just to parse a poorly structured CSV that came out of a shell script dropped from an ETL pipeline, into a high performance pipeline of matrix-matrix multiplication on a GPU is, I believe, not something they are going to want to do in general.

[ityonemo]

Maybe a “best practices” guidelines for eval and rpc might be useful. One of my eval statements in my shell script looks like this, which I think is maybe evocative of some of the struggles:

#/bin/sh
my_app/bin/my_app eval "import IP; Application.ensure_all_started(:my_app); :some_atom1; :some_atom2; MyApp.run_file('~/configuration.toml', ~i/$1/);" 

• I’m not sure there’s guidance about sometimes needing to do Application.ensure_all_started/1 in an eval statement; I only knew immediately to put that in from lots of experience, and this may be frustrating for beginners.
• the :some_atoms are there because i have been maybe overly cautious about String.to_existing_atom.
• the first parameter to run_file/2 is a charlist to avoid having to escape double quotes, and is a bit of a hack because I happened to remember run_file just calls File.read!, which admits charlists.
• the second parameter uses this sigil_i that I happened to remember I’d written because that lets me avoid a quotation mark in IP.from_string!/1. If I had a function that took a String, I would probably want to use ~s// or something, but guidance on that as a best practice would be nice.

It might also be nice to be able to package shell/batch scripts into the release relatively easily in the Release pipeline (like how we have :assemble and :tar directives) I don’t do this currently (and just copy/paste from my source directory into vi).

That said, I love releases. I did a release on a personal project - persistent websocket datascraping into Sqlite last night directly onto an free-tier aws. Smooth as a baby.

[dimitarvp]

I think the main appeal towards Rust for Elixir devs is the static typing (and one really useful product of it: exhaustive pattern matching).

I claim that the experience of making a web app in Rust is miles behind Phoenix.

[dimitarvp]

@ityonemo

That said, I love releases. I did a release on a personal project - persistent websocket datascraping into Sqlite last night directly onto an free-tier aws. Smooth as a baby.

Write a guide in a blog post, please!

[konstantine]

@henriquefernandez

IMO expressiveness are very very very important thing while modeling business rules. And “it’s a thing” work with a language that not few like you are hiting your head in your keyboard writing a code thats seems like visual noise.

That. The practicality and aesthetics of Elixir code.

[sasajuric]

You raised some valid points, and I can see how various issues combined can lead to frustration. I’ve been using BEAM languages for the past decade, and while I’m in general a happy user, I agree that there’s a lot of room for improvement

In particular, I think that the lack of strong typing is a big deficiency, and I’m hoping that some of the ongoing initiatives will address this. If projects such as Gleam reach enough maturity I could see myself migrating to them, at least partially.

I also agree that ecosystem is far from perfect, in terms of size, as well as support. It is indeed worrying that some of the prominent libraries are developed as a private effort of a few individuals, with lot of the work probably done outside of working hours.

All that being said, I’m still in general a very happy user of BEAM languages, and I believe that they are by far the most suitable options for building fault-tolerant soft real-time backends of any size and complexity. Projects such as WhatsApp have demonstrated that BEAM can take us very far, but at the same time, in my experience BEAM languages, especially Elixir, excel at building small scale simpler systems. I’ve worked on a couple of such systems which were implemented completely in a single BEAM language, as a single project, running as a single standalone OS process in production, requiring no external dependency at all. One interesting example was a proprietary CI server, a sort of hard-coded Circle/Travis/Jenkins, which had to deal with all of the standard CI challenges, such as monitoring changes in a remote repo, running multiple concurrent builds, managing load and concurrency, dealing with docker containers, caching, running scheduled jobs, persisting state, etc. All of that was implemented as a single standalone Elixir OTP app, using nothing else on the side. I’m not aware of any other technology that would allow me to reduce the operational complexity so much.

As a smaller-scale example of the kind of simplification we can get with BEAM, take a look at my site_encrypt library, which I’ve also showcased in this blog post. Again, I’m not certain that such level of operational simplification can be achieved outside of BEAM, at least not with the similar set of guarantees.

It’s probably impossible to asses objectively, but I personally believe that these benefits are much more important than the downsides I’ve experienced. For example, when I was building a CI, I needed to interact with GitHub graphql API, and had to implement the client from scratch. It took me about a day to research the docs and get a working prototype, and then a few more days to write a proper solution. In a richer ecosystem I might be able to find a library and solve this in a matter of an hour or so. This seems like a radical time overhead, but in the grand scheme of things it was insignificant, because the bulk of the time was spent on the essential domain logic where no library could help me. Such has been my general experience in the past decade of working with BEAM. Sure, I occasionally had to reimplement some wheel manually, such as a basic client for an external service. But usually most of the work was spent on the actual domain logic, and so this occasional overhead didn’t add up to anything significant.

In the end, it comes down to how each of us values given pros & cons, and it depends on what are the challenges we’re trying to solve. For example, I agree that BEAM is not a good fit for some domains, such as GUI apps, CLIs, fast numerical processing, etc, and I usually advise people to look for something else for such domains. But, like @josevalim, I’m not sure that any language/runtime will be a good at everything. I believe that BEAM is a great fit for fault-tolerant soft real-time systems precisely because this is the thing it focuses on.

In any case, while I may disagree with some of your points, I still think you raised valid concerns, and that’s always a good thing.

One final minor comment to the point from your gist:

Umbrellas as a project structure are an extremely permeable form of “isolation” and in my mind provide neutral or negative value to one’s architecture. Elixir has no concept of module-level privacy or hierarchy, only public and private functions

I share your sentiments about umbrellas, and I’ve never used them myself. The boundary project is my attempt to tackle this issue in a different way. Feedback is welcome

[cnck1387]

@sasajuric

It took me about a day to research the docs and get a working prototype, and then a few more days to write a proper solution. In a richer ecosystem I might be able to find a library and solve this in a matter of an hour or so.

I think this is an important note to make. You mentioned you have a decade’s worth of experience using the BEAM. This puts you into the same category as Jose himself, or someone who is at the extreme end of being super proficient and capable of writing pretty much anything you want without too much of a headache.

As someone who doesn’t have that type of experience, your few days of coding a custom client becomes an impossible hurdle to cross or involves asking for help from external sources and potentially never reaching a solution. When this happens multiple times during a project on every corner, that’s when you think about dropping out of the language.

It’s very much a deterrent when you want to integrate with something (such as Stripe) and seeing an officially supported library for Python, Ruby, Node, PHP, Go, .NET and Java but no Elixir. Almost every service I want to integrate with doesn’t have an Elixir client.

Also seeing libraries like ex_aws stop being maintained because the sole author who maintains it doesn’t use AWS anymore is a bummer. The forum post was around for months with an ask for help to find a new maintainer but no one replied, so now one of the most popular back-ends for storing file uploads becomes a problem you need to solve as an individual developer.

In a lot of cases, to develop a web app you’re on the hook for having to become a library creator instead of an application developer just to begin your project. Some folks might want that, but it’s not exactly a productive environment where the goal is to go from no app to launching an app.

[LostKobrakai]

If you’re judging the usefulness of a programming language by its ecosystem that maybe should’ve been a (bigger) factor in choosing it as the platform for whatever you’re building in the first place. You’re by no means wrong in your assessment, but elixir is still a quite niche language. It’s to be expected that you might not find whatever you need in third party libraries and even less so in official libraries by companies, which are often provided by a mixture of popularity and what people within the company can program. If you’d go just by the metric of ecosystem I guess ruby, php or js are likely unbeatable.

@sasajuric’s point as I understand it is that the tradeoff for not having that ecosystem might not matter as soon as elixir/the beam does provide (greater) benefits in other places. And one doesn’t need a decade in experience to see or benefit from them. It depends on many more factors besides experience like how complex/big a project you’re working on, how stateful your service is, the “devops story” (being able to connect to the vm at runtime and observe/debug), how much moving pieces you want to deal with, …. And one still might come to the conclusion of elixir not being the correct choice.

[amnu3387]

Although there isn’t an “official” lib for Stripe (I know it’s just an example) but since you mentioned it, the stripity_stripe is pretty much ok and well maintained - it also allows you to pass additional params into all requests or generate custom requests, so basically even if Stripe’s API is incremented accepting new key-values or new endpoints, you can use them even if stripity_stripe hasn’t been updated.

I also understand the value of having libraries for interacting with common API’s, but on the other hand I think it’s a bit overrated - if I was not asked specifically to use libraries for those interactions, I personally would just use the actual HTTP API. In the case of the most used/relevant ones they’re pretty much well designed and you’ll only be using a very small subset of their functionality - every time I have to use a lib, I have to read the docs of the official HTTP API of the provider and the docs of the lib - and this is not to say they’re not valuable - it’s just they can get out of sync and then there you go hunting for what there’s a mismatch.

I think yours are valid concerns, if you’re developing something you don’t have to worry about writing the interface for the HTTP API and just use something that “no one was fired for using this lib” (in the sense it’s the official, sanctioned one), and writing it might take a toll because handling the API’s responses is not your domain problem. I still think though that in the overall scheme of things it’s a minor part, and eventually after you do it once, it shouldn’t be that time consuming doing it twice or thrice.

Just my opinion, not sure it’s worth even a $0.01 but that’s how I see it.

(and this in the context of the other things the BEAM gives you almost for free, of course all things being equal, without such runtime, there would be no reason to go with a language that has a smaller ecosystem)

[sasajuric]

To be clear, I’m not suggesting that libraries are bad. All other things being equal (or similar enough), it’s of course better to have the library available than to not have it . My position is that as a backend developer I get some important benefits from BEAM that I personally value more than the amount of available libraries, and that even with the lack of libraries, the end solution often seems significantly simpler to me.

To reiterate, I worked on a couple of systems that were implemented completely in a single BEAM language, without anything else used on the side. With many other languages I’d need to run multiple OS processes (i.e. microservices) tied together with other 3rd party tools such as Redis, message queue, reverse proxy, cronjob. This is a huge amount of technical complexity that is rarely, if ever, mentioned when comparing languages.

In my view, ecosystem can always be grown on top of good foundations, but it doesn’t work the other way around. You can’t fix some fundamental deficiencies at the runtime layer, like no support for fault-tolerance or stable latency, by adding more libraries. You can only work around such deficiencies outside of the language, e.g. by going down the microservices path, maybe reaching for k8s to assist you with that. This can certainly work, but I’d argue that it’s much harder than hand-coding a couple of REST or GraphQL requests and interpreting the responses

In particular, when it comes to manually integrating with a 3rd party service, my experience is not very extensive, but in the few cases I had it was relatively straightforward and didn’t require any advanced knowledge of BEAM. It boiled down to reading the API docs, picking an http client library, issuing the requests, and interpreting the responses. I’d usually need to invoke only a couple of different actions, so it wasn’t a lot of work. I agree that this is still far from perfect, and that it can seem unsurmountable to junior, but I don’t think it’s rocket science

There’s definitely a lot of room for improvement. Things can and should be simpler, but we can gradually get to that point, given time and effort. Which is why in general I place more value at the foundational layer (in this case BEAM) than at the ecosystem.

[StefanHoutzager]

Great post!

Fairly minor in the list above, but I think we made deals with devils in courting the amount of Blockchain/Crypto orgs that are present in the community. I personally find that approach unconscionably wasteful ecologically and also almost universally solving the wrong problem technically.

Haskell has this elephant in the room too. Personally about 50% of the invites I got for speaking about contractor jobs for elixir were blockchain related. Technically interesting, but ethically?

The Haskell Elephant in the Room

[derek-zhou]

Technology frontiers are usually pushed by lucrative and questionable motives. Streaming video was pioneered by the porn industry and big data analysis was driven by the privacy invading social medias. In the end, we all benefit.

[cnck1387]

@sasajuric

To reiterate, I worked on a couple of systems that were implemented completely in a single BEAM language, without anything else used on the side. With many other languages I’d need to run multiple OS processes (i.e. microservices) tied together with other 3rd party tools such as Redis, message queue, reverse proxy, cronjob. This is a huge amount of technical complexity that is rarely, if ever, mentioned when comparing languages.

I don’t want to push this thread OT but I don’t see those things as being complex in the grand scheme of things and some of them are shared with Elixir in most web apps.

Most web apps written in any language will likely want:

• At least a persistent database, such as postgres
• A reverse proxy to properly handle things like SSL termination, redirects, static file caching, country detection, basic load balancing and many other features that nginx supports
• A way to execute tasks in the background and periodically.

Popular languages like Python and Ruby have battle hardened tools to solve background tasks in all shapes and forms (Celery in Python and Sidekiq in Ruby).

In Elixir, chances are you’d still want to use Oban or another library because in a realistic app you’d want a bunch of features like separate queues, retries, cancelling, uniqueness and scheduled tasks along with a dozen other things a robust background / que library will offer.

The only extra added complexity you may encounter in another tech stack is using Redis, but Redis is one of the most least complicated things to manage from an infrastructure POV. You can leave it running untouched for months or years and it’ll run like a champion. Lots of cloud providers also provide fully managed Redis servers too if you’re into using services like that.

Throw in a bit of Docker and suddenly that Redis complexity kind of goes away even if you decide to self manage it. You can literally add 5 lines of YAML to 1 file and now you have Redis up and running along with the rest of your stuff.

Redis also doubles as a cache back-end and has great support in popular web frameworks.

I haven’t directly used cron in years on any web app because periodic tasks are solved problems with background tools like Celery. It’s also especially nice because it’s distributed too and you can keep your web server pretty much entirely stateless.

Everyone has their preferences but personally I would rather use tools that thousands of folks have been using for many many years because they have a ton of edge cases ironed out and are super well explored and supported. I prefer working in an environment where I can solve the business needs of my apps without having to reinvent a new library from scratch every step of the way.

It’s especially nice when more opinionated frameworks also take care of common things too, because at least you can be confident the library or functionality won’t stop being supported over night because one person decides they are not using it anymore.

[sasajuric]

So supposedly making a REST request is complicated and requires advanced technical expertise, but dealing with nginx, sidekiq (which requires running an external worker), and redis isn’t? I’m gonna have a very hard time accepting that

The (admittedly not fully realized) potential of BEAM is the fact that you need to only learn the programming language and that can take you very far. This reduces the amount of technologies that needs to be mastered, and simplifies the life for everyone on the team. But beyond that, less external technologies reduce the disconnect between dev, test, and prod. Perhaps I was on wrong teams, but none of us usually ran nginx or sidekiq locally. This led to occasional production bugs b/c the stuff not running locally or on CI is the stuff untested.

In contrast, Phoenix can handle many of the reverse proxy features you mentioned. It is used equally on all machines (dev, staging, prod), requires minimum extra operational overhead, and is easily testable.

If you want to obtain an SSL certificate via Let’s Encrypt, take a look at site_encrypt. You can get it working in a matter of minutes and it won’t require installing anything else on the side. And you use the language you normally use to work with it, so no need to learn a special flavour of yaml, ini, or anything like that. That language is compiled, so syntax errors are immediately detected during compilation, while semantic errors (e.g. misspelling of the domain name) can be detected in tests. Since the interface is a programming language, there’s a lot of flexibility (like e.g. fetching input parameters from OS env or some secure store), and the stuff works equally well in local dev and test without requiring anything running on the side. Since site_encrypt periodically renews the certificate it runs a periodic job without requiring any extra OS process or external component. Using vanilla OTP supervision tree, the job scheduler binds itself to the Phoenix endpoint, so if the endpoint is stopped, the job will be stopped too, thus avoiding running a certification that is bound to fail. For more details see this post on site_encrypt and this post on periodic jobs.

Compared to using nginx + certbot this gives me simpler usage, simpler operation (less moving parts), reduced dev/prod mismatch, better testability, and more flexibility. To me these are the very important benefits, and this is where I see a huge potential of BEAM. I’ve had enough first hand practical evidence of it to be convinced. I’ve also had the pleasure of working on systems powered by a bunch of moving parts, and I don’t feel like going back to that

Just to be clear I don’t hesitate to use external components where it makes more sense. I agree that external database is frequently needed (though oddly enough I managed to get away without it on a few occasions), but other than that I think that many projects can do just fine without needing reverse proxies, redises, external message queues, cronjobs, & such. I’m not saying that these tools are bad per se, but I prefer exploring more lightweight built-in options and move to these tools only when there’s justified need.