On "Why Elixir?"

shanesveller · August 27, 2020, 7:03pm

This relates to a discussion thread we’ve had at work back in July and I’ve gisted my answer for that here. It contains more of my positive sentiments about Elixir/BEAM, and there’s a decent amount of those, but it’s semi-off-topic.

For the thread at hand: I’ve stopped working with Elixir outside of my day job, and mostly stopped promoting it to others or participating in user communities in my spare time. I no longer feel the drive I once had to help others be successful with the language.

I feel that there are too many papercuts compared to what I personally value in software development, and most of them are inflexible aspects of the experience of working with Elixir. I began learning Elixir in mid-2013 and have worked with it professionally in some capacity or another since 2016, and full-time as my primary focus since late 2018. I still work with it daily and have no plans to change that for the foreseeable future.

Most of what I am feeling jaded about falls under one of these larger topics, and I’m possibly going to ruffle some feathers and/or summon some pedants to tell me how I’m mistaken:

I want an efficient, sustainable, and graceful on-ramp to deploying software. Otherwise, what’s the point? I know this part of the domain extremely well, care about it very/too much, and have invested many hours in making it better personally and professionally, but it basically only gets us to the same table-stakes that any other language enjoys. Runtime vs compile-time configuration continues to be a massive attention/energy sink and teaching hurdle. Teaching good instincts for Elixir deployment has not gotten any easier than it was in 2018, IMO.

More personally, I don’t live in a BEAM-only world and I can’t tolerate niche features and tools that can’t be standardized with the other languages I write/deploy. I don’t value hot upgrades and I assert that most businesses don’t need them, only pursue them from vanity.
I increasingly value provably correct software, which for me means an actual type system to go with your good testing practices. Dialyzer is an insufficient tool here because it is so easy to ignore/misinterpret/bypass. It has non-zero value but the ROI on its usage is very poor even as a willing participant. I can count on my hands the number of meaningful bugs it has caught at my professional roles, although they were often extremely nefarious and would never have been caught by tests.
It’s hard for me to pin this part down past half-baked sentiments, but I feel that Elixir is a language that is reasonably good for experts but far too flexible, vague and undirected for neophytes. There is not enough guardrails in the core tooling, and few idioms being prescribed from first-party sources. As many people try to claim about C, it’s easily possible to write very clean, well-designed and effective code in Elixir. However, the language gives you plenty of rope to hang yourself with if you are not working with or at that expert-level experience. I’m not arguing for something like Golang, where someone with two weeks of experience and someone with 15 years of development experience write similar code because they have no other choice. I don’t know what I am arguing for here, but I feel a lack. rustc and clippy do absolute wonders here as pedagogical tools.
We still don’t have much adoption by mainstream companies and providers - it is a night and day difference when comparing to a language that is used more broadly. Our library ecosystem often feels like this XKCD comic, and our language tooling (i.e. elixir-ls) are maintained by underappreciated and diligent volunteers who are spread way too thin. I don’t think more than a handful of people are paid to work on this space, and I still encounter areas a few times a year where I would have no choice but to write an integration library myself.

ExAWS is still looking for maintainers last I saw, despite it probably being extremely crucial to many businesses, and AFAICT Elixir is basically the ghost of a dream within FAANG, Dropbox, Spotify, etc… I don’t chase these companies out of me-too attitudes or vanity, but I feel quite powerfully that they generally disregard Elixir as a realistic choice for serious projects and so we are excluded from the kinds of contributions that their additional engineering power could make, to the benefit of us all.
There are some use-cases that we are quasi-permanently not suitable for. Some of those happen to be areas of work that I really want to participate in. We are very good for web development and for network daemons, and Nerves is very compelling for embedded, but Elixir is a poor fit for desktop software, CLI applications, game development, machine-learning or other compute-heavy tasks, or FFI coming from other languages.
There is some noticeable brain-drain occurring, especially towards Rust and WASM. Losing the active participation of some of the folks who have moved on really set us back, IMO.
Fairly minor in the list above, but I think we made deals with devils in courting the amount of Blockchain/Crypto orgs that are present in the community. I personally find that approach unconscionably wasteful ecologically and also almost universally solving the wrong problem technically.

My gist above has more (not particularly refined) thoughts here as well. If I sound like a hater, I had a lot of good to say about Elixir too. I am undoubtedly a pessimist and a cynic, though.

I don’t think libraries of end-user macros or other non-breaking changes are the way out for most of my problems. I also don’t think some ill-defined “craftsmanship” movement about professionalism is what’s missing either. My problems aren’t because the community isn’t comprised of talented folks.

I wish projects like Gleam, Hamler, and whatever Facebook is cooking up the absolute very best, but I don’t think I am likely to stick around long enough to see their dreams fully realized. They would probably resolve a lot of my frustrations if they can deliver on the premise.

My current front-runner as a replacement is Rust. That’s probably fairly obvious from some of my phrasing above.

krstfk · August 27, 2020, 7:38pm

You raise some good points but

maximum' :: (Ord a) => [a] -> a  
maximum' [] = error "maximum of empty list"  
maximum' [x] = x  
maximum’(x:xs) = maximum’(xs)

In Haskell is logically wrong, and type checks. If you want probably correct software you need a theorem prover and it’s a whole new level of difficulty, that most of us can’t afford.

While true, that with some assumptions and a type system on par with the Haskell’s type system you can go a long way when refactoring ( I would highly recommend the https://www.cs.ox.ac.uk/publications/books/functional/
book to get an idea of what it’s about), it’s not a way to prove your software is correct.

We also have some pretty great tools to insure correctness : dialyzer + prop check + unit tests can go a long way.

Not to say that you’re wrong to want a strong/static type system. I wish I had it when working with elixir or erlang, but then again, the benefits of the os_like ecosystem and the simplicity of these languages compensate the lack of static typing for me.

shanesveller · August 27, 2020, 9:24pm

Sure, your point is well-made. I don’t believe passing type-checking means software is inherently correct anymore than I believe than dynamic types means software can’t be correct. It’s just a tool in the belt and one measurement of several that matter, but I find it a very useful one even so.

krstfk · August 27, 2020, 10:06pm

I don’t disagree, and that’s actually why I don’t feel this thread is very useful.

When I started working in my current position I was tasked with looking for an alternative to the tech stack (LAMP) for our web apps. I did not want to go on with the p, I hate it, irrationally. I’ve considered a few languages including ocaml, haskell, scala, rust, erlang, elixir. For each of those I’ve evaluated the costs and benefits for my use case and settled on elixir although there are things I don’t like here either (eg: lack of static typing, syntax, variable shadowing by default and pin operator…).

I’m not concerned because it’s niche. The beam is not.

What I’m trying to express is that companies/individuals may have valid reasons to ditch Elixir/erlang, but these won’t necessarily be reproducible.

Also, it’s fairly easy to interact with the outside world with nifs, ports, c nodes, Java nodes, or message queues

Of course you wouldn’t write a cli unless in elixir unless you had to, nor a desktop app (although observer and wings3d would be counter examples).

sodapopcan · August 27, 2020, 10:55pm

If this thread isn’t useful (and I certainly understand why you say it is not), as a newcomer I certainly find it interesting.

Not having built anything other than a toy app so far, I’m getting a little nervous about deployment. Libraries struggling to find maintainers is another thing that concerns me. Both these points I have come across before this thread, though, so can’t say I learned about them in this thread.

Personally, I couldn’t care less about static typing (Elixir being dynamic was a big attractor), syntax, or any of that. The most annoying thing about the syntax for me was that coming from Ruby, I kept thinking I was in Ruby which would slow me down, but I got over that. Otherwise, I’m pretty into the syntax—even the pin operator! Maybe going with prime and no re-binds would have been nicer, but it took me very little time to get used to pinning and the amount of times I rebind I single var in a function is far more than I pin.

ANYWAY

@shanesveller, I’m interested to hear more of your thoughts on hot code reloading if you would. I understand what it is, but I don’t fully understand its implications and why it wouldn’t be desirable (again, I’ve never deployed anything Elixir or Erlang). I’ve always thought of the pursuit of vanity as throwing as many services and acronyms into your tech stack as possible… deploying with just your language and getting zero-downtime-for-free seems like a very noble pursuit to me. I do understand that not everything can be done with OTP, but how far it can seemingly get you, especially in the greenfield phase, is very attractive to me.

Of course, I’m also the type of person who will read about a new tech craze and 9/10 times think, “Damn, that’s really interesting, but I hope I never have to use it!!! …and oh lord I hope my employer doesn’t find an excuse to start using it…”

I might be a little off base here, but I’m very curious about all of this. As a beginner I would love to see deployments mix release right up front and centre in the language’s marketing material. I mean, Phoenix is already spouting NO JAVASCRIPT why not NO DOCKER! I dunno, again, I may be off base here.

dorgan · August 28, 2020, 2:02am

It’s a fairly new addition to the language, though. There was distillery before(and still is!), but before 1.9 releases weren’t so “blessed” by the language itself. Now reading the docs to run mix release with a :tar step configured is all I need, really.

Considering the latest updates to elixir that focused exactly on configuration and releases, why is it the case now?

I don’t quite get what you mean with that, but for this part:

The core team seems to agree with you.

However, the language is designed to be extremely good at solving a family of problems that made them sacrifice static typing, so it’s a matter of tradeoffs. Considering that, I doubt there is something to be done as it’s a non solved issue and there’s active research going on. Sure, you can interop Gleam with Elixir for the pure, not message related parts, but I feel it as strange as when I try to mix ReasonML with JavaScript, in those cases type-safety is still very hard to achieve. ~~And don’t get me started on TypeScript unsoundness and HORRIBLE web api bindings.~~

However, while possible for the adventurers, many of those fields are non-goals for the language. This reminds me of the bad usage of the language/libraries that hurts marketing, such as people trying to develop games with LiveView(that I can’t even play because I live at the other end of the world and latency is of at least 300-500ms) and shadowing the usecases that make it unique and tempting.

But Rust is far from being a 1:1 replacement for Elixir, at least not out-of-the box(and I’m not sure there’s something like the BEAM for rust. Emphasis on for as afaik theres a beam implemented in rust).
Elixir and Erlang are all about the guarantees they offer, not only about memory safety or “async capabilities”.

In your gist you mention:

Error handling with tagged tuples is fairly verbose, and we didn’t always have with as a special form, which helped a lot. Similar to my comments above, it’s something that needs intent and attention to handle correctly, because “forgetting to check for nils and errors” is not something that the compiler will call you out for.

But I must say that’s also true for Rust, OCaml, Haskell, Gleam… with the Option/Maybe and Result/Either types. You may also see tagged tuples as ADTs. Sadly, I feel implementing proper abstractions from category theory like those seem odd even in this functional language and such formal ideas are a barrier for people coming from non functional languages. Also point-free style is odd with capture syntax.

It is not my intention to question your reasons, you’ve invested several years on it and you clearly know why you do what you do, moreover I agree with the points I haven’t quoted, but as mentioned in this reply there are many hidden stories behind the opinions we form over time and it’s hard to extract a general gist of where and why are we failing, as it is needed to make case-by-case studies.

This other article from Fred Hebert is a good take on why people come and leave from the beam ecosystem and where value really is.

PS: I rewrote this reply several times and I hope it came out as something productive and not fueling a circular discussion.

sodapopcan · August 28, 2020, 2:26am

You may have missed my point… I’m not saying that I was expecting to see release mix up front and centre NOW—I know it’s relatively new. I’m saying that I hope beefing it up is a priority and that one magical day in the foreseeable future we’ll seeing, “Elixir is a dynamic, functional language designed for building scalable and maintainable applications with a kick-ass, brain-dead deployment process—no Docker or Kubernetes required!” (there is definitely some hyperbole in there). If there is signal out there that this is on the roadmap, please point me to it! I’m very much talking in terms of attracting newcomers here (as well as wanting it for myself, of course).

Otherwise, when I do go to release, I’ll be pinging you for pointers

sodapopcan · August 28, 2020, 2:27am

And now that I’ve written that reply and read the rest of your post, thanks for the link to the core teams take on hot reloads!

sodapopcan · August 28, 2020, 4:04am

I’ve digested more of your post its various links (I’ve worked my way through a bit of Fred H’s blog but had yet to get to that one) but in particular I’d like to respond to the aforementioned link about the core team’s take on hot code reloading.

I understand their rationalization but the problems associated with hot code reloading seem on par with—if not even easier to reason about—zero-downtime-deployments involving structural DB changes in any language’s ecosystem. It seems like a hot code deploy is max two deploys whereas some DB migrations could be get to five or more deploys with a larger blast radius codewise.

Again, I’m new and might not have the whole picture.

I would also like to acknowledge that I do realize I’ve been conflating hot code reloading and general deployment wishes in my posts.

Edited to add: Wellllll ok, if you had to treat every deploy this way, I could see it becoming a burden.

Nicd · August 28, 2020, 7:57am

I agree with this and have seen it many times on this forum and various community chats, which is actually why I wrote a blog post about it to try and help. That said, with config/runtime.exs coming in Elixir 1.11, I’m hopeful we’ll finally have a unified configuration system. I just hope we can get generators of popular projects like Phoenix to embrace it quickly, or it will be like string formatting in Python – too many choices for newcomers.

rvirding · August 28, 2020, 8:27am

I just want to be a little picky here about release handling. It is NOT a language or libraries feature, there has been support in OTP from very early for releases. Support both for building them, see the :systools module in the SASL application, and for doing configuration at startup time. It is the mix tool which in the beginning chose not to provide support for building releases. I don’t know why.

And I really agree Fred Hebert has written some really interesting and good stuff, both books and articles, which are also extremely readable. His Learn You Some Erlang also describes releases .

hauleth · August 28, 2020, 9:11am

That is why I am working on libraries like systemd to provide better integration with systemd, so the Docker will not be needed (and even troublesome). I still need to write article about how to deploy Elixir application with systemd, as it could make the process clearer. Not only about running the application itself, but also it will provide tooling for simpler configuration management (as you can run script to generate sys.config before start of the application instead of relying on “hacks” in the core, like config/runtime.exs).

dimitarvp · August 28, 2020, 9:43am

Because now we have several slightly differently working compile-time / boot-time / run-time configuration policies and it becomes a legacy support burden. When you get into a new project as a consultant / freelancer / employee you’ll have to take into account if they use distillery or releases and what Elixir version. A lot of project’s configs are very specific and people are understandably unwilling to change them because of language or library version upgrades. I faced this often and the result was an ugly pile of .exs files that nobody except people in the team could decipher and modify.

I am with you here and hence I adopted a policy to mainly use Elixir for glue and orchestration and iteration speed but reach for Rust when things need to be faster / better type-checked / venture into an area that’s a non-goal for the BEAM.

Not exactly, Rust and OCaml enforce exhaustive pattern matching.

dimitarvp · August 28, 2020, 10:22am

I applaud you for the good summary and the courage to post it. The linked GitHub gist is also an awesome read!

There are aspects where Elixir seems to not be able to have a strong opinion (configuration being a prominent example). And those papercuts indeed do build up.

Without this, any language is eventually doomed. Platforms like Render and Gigalixir have emerged to make this easier – but they only manage to herd you into their own garden where deployment is still a service-specific affair. I’d like one Deployfile that can be used for all deployments! How hard can it be? Maybe Dockerfile-s can be transparently translated into actual scripts for Docker-less deployments?

This and the rest of the paragraph is spot on. As a guy who was a noob (and still is) in many technologies, it’s really not helpful to have many ways of doing things and relying on people to discover for themselves. Some direction and “blessed” ways that must be very front-and-center-and-impossible-to-miss is a very much needed change! It’s one thing to play with a language, but a completely different story when you are working with it. Newbies need direction.

I’ve entertained the idea of making a website with brutally short snippets for common scenarios (sort of like StackOverflow but much more minimal and to the point, and without comments too) but life, work and health takes precedence and such a project might never come to fruition.

Yep. Good practice advices baked right into the language’s tooling I found to be an amazing way to train newcomers and experienced veterans alike.

I work with Rust more and more lately (it’s my new job) and I have to say that beyond OTP, brief syntax and fantastic libraries like Phoenix and Ecto, I don’t feel much pull back to Elixir. OTP is definitely unbeatable for the moment and that’s reason enough but I wonder how long will that last.

Apologies to everyone for dumping 3 posts in a row.

cpgo · August 28, 2020, 12:09pm

I agree with this.
elixir-ls is awesome when it works (most of the time), but when it doesnt the experience is not very good.

Usualy deleting the build directories fix the issue, but still I feel we should give more love and support to the language server maintaners (thanks @JakeBecker, @axelson and the other contributors).

On a side note, its interesting to see that the Go team has officially stepped up as the new maintainer of the Go extension.
Im not saying that the elixir core team should maintain the vscode (or any) extensions, just pointing out that editor support is very important for the language.

derek-zhou · August 28, 2020, 1:20pm

While I agree that the “blessed” ways should be promoted more, I like to have choices (I used to program in perl). For example I don’t like eex (or any template in general) and elixir/phoenix gives me the option to hand roll html in render/2

hauleth · August 28, 2020, 2:45pm

My problem is that even the “blessed” approach seems a little bit “unnatural” for me. The main problem there is that there are some options that cannot be set via config/config.exs but I think (but I am not sure) that these options can be set via config/runtime.exs. And to be able to configure these options via config/runtime.exs the VM needs to be started twice during startup of the application… Additionally while config/*.exs files looks like code, these cannot contain all BEAM types (for example functions, I am not sure how about PIDs or refs) nor it can use dependencies, at least not all of them. This makes the configuration here quite confusing, in my humble opinion.

shanesveller · August 28, 2020, 3:18pm

Several other people have alluded to this in the further replies, but hot upgrades do not come for free despite being readily available in the technical sense. You need to be very intentional in how you plan and execute such transitions, and you often need to write specific support code to make it successful, and these do not have much in the way of prior art for folks coming from other languages, so this is often a new set of skills and instincts to nurture. Even some of the strongest proponents of hot upgrades will readily admit that they don’t use it 100% of the time for every revision - sometimes they still just do a full restart of each BEAM node.

Most businesses’ uptime expectations are about the customer impact, not about individual runtime instances, and you can use bog-standard horizontal scaling and load balancers to achieve this in a language-agnostic way that still feels available to the user while you’re doing your shuffle behind the scenes. This skillset is more readily available in the broader industry with lots of prior art and tooling. Most products don’t actually have the strict tolerances that call for true hot upgrades as a strict necessity like a telecom would have in the original use-case for Erlang.

All of that is also just dealing with the BEAM implications, and you yourself acknowledged down-thread that a lot of the difficulty actually comes from how you manage your off-BEAM stateful database content during evolutions. That bit doesn’t really change one way or the other with hot upgrades and it’s the piece that most orgs struggle with IME, and I think it’s a more productive place to spend your time, energy, and educational efforts.

shanesveller · August 28, 2020, 3:36pm

I think releases.exs has not moved the state of the art forward from what we already had with Distillery, but I am quite looking forward to runtime.exs when Elixir 1.11 ships in ~October. That will really drive the conversation forward and introduce the required learnings earlier in a project’s lifetime, which is all to the good. The version of releases introduced natively with Elixir 1.9 does a minimal subset of what Distillery provides, and some of the things that were omitted are things I still want for every project today. To be honest, I think most people switched to native releases not because Distillery’s documentation was hard or its features were overwrought, but because it has every appearance of being unmaintained. I hope that’s not true, but there’s been relatively little communication around it since Paul mostly left the Elixir ecosystem. I’m grateful to his incredible work on Distillery and libcluster, but it feels like we’re on shaky ground where the new solution is incomplete and the incumbent is stagnating.

Hot code upgrades have no analogue in other languages, and edeliver is basically a Capistrano clone, which in hindsight requires a sliver of Ansible to reimplement if you enjoy that deployment model. I emphatically don’t. These tools and techniques cannot be adapted to a second or third language at a polyglot organization, or at least I feel attempting to do so is not a good use of time. If you are at an org that allows that fluidity of language use, you have to optimize your deployment practices for the lowest common denominator and start communicating and working with a standardized process and vocabulary around deploys. There’s no space for bespoke single-language stuff, given my belief that hot upgrades are not actually necessary for delivering business value in most products/orgs.

I don’t have the goal of arriving at something that is like-for-like with the BEAM. If I valued all of its designs and principles that highly, I would stick with Elixir and be happy. I value languages that let me do meaningful work, do it well and comfortably, and avoid needing superfluous hardware to power it. I value code that is resilient to human and technical failure. The BEAM does not have a monopoly on those properties.

I think most of the rest of your post is actually supporting my own stance, highlighting that I value things and want to pursue areas of work that are implicitly or explicitly non-goals for Elixir. It’s okay for those things to be non-goals for Elixir, but it’s also okay for me to recognize that disconnect, and that the values driving the design decisions are not my own. That means I probably have to look elsewhere to avoid suffocating as a developer.

Totally happy with your points and your tone, thought it was all in the spirit of respectful challenge and I appreciate that. Lots of great links, too.

josevalim · August 28, 2020, 4:29pm

I would love to know what is missing because the only aspect of Distillery we haven’t brought over at this point is hot code upgrades, which you have mentioned multiple times you don’t care for.

That’s my impression from reading your post too. And, to be fair, it is not necessarily about those being non-goals for Elixir. Some of your concerns are really only addressed in the most popular languages (or the heavily backed ones), and even then, not all of them would check all of the boxes.

Edit: Comments on the gist.

if a module is loaded in your copy of the BEAM you can call it day long, even if you shouldn’t be.

Improved in Elixir v1.11 which will require those to be at least listed in your application dependencies. This will address issues such as poor boundaries between (umbrella) applications. It is a direction we are actively exploring.

There’s a lot of use-cases it’s frankly just plain unsuitable for. CLIs, game development, and machine learning applications are some common examples.

To be honest, very few languages would be suitable for CLIs, game development, and machine learning, in addition to the areas Elixir already provides a good foundation for, such as data ingestion, web apps, embedded systems, and distributed systems.

Even if we consider your three leading entries, I am struggling to come up with a language that currently excels on all three.

Node and Java beat us handily in a lot of scenarioso, and anything in the C- family or Rust can do much more and with way less resources required.

Agreed on C, Rust, and Java, but I can’t think of where Node would handily beat us and we are generally being used at.