Distillery + Windows 10 + hot upgrade questions

Questions / Help
1

We have been working on creating distillery releases with hot updates targeting Windows 10, and there have been a few unexpected differences. I was hoping someone could fill in a few details, since I’m not sure if the differences are required or if they’re an artifact of an intermediate state (or unfinished iterations) of Windows deployments.

I’m pretty unfamiliar with Windows development, so I apologize if some of this seems like it should be apparent. We would target Linux if possible, because of our better familiarity with the platform, but we’re dependent on 3rd party software that only works on Windows.

The service name is <rel_name>_<rel_version> instead of just <rel_name>

When a hot update is applied, the old service is removed and a new service (with the new version number) is added. Couldn’t this just use a less specific service name? Is there something in the windows service management framework that necessitates the different service names?

One thing that we’ve noticed is that during the hot update, the erl processes crash. The new service comes back up, but it does kill our channel connections during the upgrade.

Visual C++ Redistributable Packages must be installed, even if erts ships with the release

Is there no way to ship the required dlls with the release? If not, we’ll just need to add that to our initial installer (or install docs). Apologies if this is something that we should already know when working with Windows.

I’m happy to issue a pull request adding this info to the documentation. If someone has an opinion on what page that should be (or if there should be a new page), that would be helpful.

Sometimes REPLACE_OS_VARS works, sometimes not

I wish I could isolate the factors for when it would not work, but to get around it we rewrote our app to pull the problematic configuration out of a yml file that we can edit on the initial install. It seemed to work fine on the Win10 VMs that we have set up in VirtualBox, and it seemed fine on some hardware, but we ran into this on an older physical machines (running Win10, for what it’s worth).

This one was really weird.

Distillery creates a tar.gz file, even for Windows

How do people tend to untar releases? To ease iteration during development, we’ve installed git-bash even in the otherwise-barebone VMs that we’ve been installing the service into. I’ve played around a little bit with 7Zip4Powershell, but it seems like you need to do multiple passes to fully extract the archive.

Do people usually install extra software to untar archives, or write powershell scripts for the initial install, or is there an alternative I’m missing?

Services are started with Automatic

We’ve been running into issues where our services fail to start up on boot when set to Automatic. We need to manually switch them to Automatic (Delayed). This is not a great experience, since the delay is 2 minutes. It’s better than the service not starting at all, though… Does anyone have a better solution to this?

I wonder if there might be some other system service dependency that we need to list, to ensure that the service waits just a bit longer to start (but less than 2 minutes).

–

Thank you in advance to anyone who takes a look at this and can point me in the right direction… or any direction, for that matter. I’m happy to issue pull requests or file issues if any of the above merits it, but I know that right now I don’t know enough about why things are they way they are to do so in a valid fashion.

1 Like
2

Windows does not hot-upgrade well, the issue is the file system, NTFS does not support concurrent read/write access of files thus the files cannot be updated, your only real choice is to manually inject the updated files from elsewhere (brittle and can bloat the VM) or do a rolling update, blame Windows here.

This is common for about any windows program in existence…

Services on windows tend to run under the, hmm, I think it was the SYSTEM user if I recall right, it’s environment variables are significantly different than a user’s.

You should not need to extract it yourself as the runtime can generally handle that for you, however yeah 7Zip4Powershell is perfectly fine, and yes 7zip does not see tar.gz files as a single archive but rather it sees it as a tar file (a collection of files, ‘basically’ concatenated together with metadata) that is then gzipped (which operates on single files for maximum dictionary reuse, hence why it often compresses better than zip in almost all cases), hence why you have ‘2 steps’. :slight_smile:

This is just some of the usual things when working on Windows. Do note, I’ve never needed to extract a tar.gz when I ran BEAM services on windows.

Now if only Windows followed POSIX standards (or even pretended to). :wink:

3

First of all, thank you for the reply. This gives me a lot more to research.

The runtime seems fine for updates, but not for the initial install. We’ve been playing around with WiX toolset to create an installer with all the necessary files bundled, avoiding the need for tar. It’s a lot more complicated than downloading a tar and extracting it, though.

Do you rely on the erlang runtime to extract the release in the initial install, or is there another way that you wind up getting the code onto a system in the first place? We were hoping to ship erts with our code.

4

When I ran mix release and it made the release in the rel directory, part of the release script then copied that entire release directory inside the rel directory to the remote windows server and ordered the server to run the old service uninstall then run the new service install. Thankfully we managed to finally toss out that windows server at work and are now on linux. ^.^;

We did not hot code swap on windows, that is a recipe for disaster because NTFS really sucks as a file system. But we had rolling outages once a week to do such updates in anyway.