Do you use hot-reload/recompilation in production?
Please note that this poll is only about BEAM’s internal features of runtime upgrades and recompilation. External tools for 0-downtime upgrade and deploys are out of the scope of this poll
- Yes, I regularly upgrade my system in production using relup/appup
- Yes, I regularly upgrade my system in production using some in-house solution which uses BEAM’s internal hot upgrade feature in runtime.
- Yes, I regularly upgrade my system using plain git pull && recompile
- Yes, when some bug occurs I prefer to fix the bug using recompile
- Yes, but only when a critical bug occurs
- No, I always upgrade running system using external tools
If you do not use runtime recompilation, I’d like to discuss why and what do you use instead.
Personally, I used to work in a project where we used
relup for publising patch version hot upgrades. However, it required a special technique to write code and relup testing pipeline was exotic
Whoa, I can see that someone voted for the inhouse hot-upgrade system. Could you please share any details about it?
Voted “No”. I don’t care about the hot-upgrade thing and I pray I never need it.
But even if you end up needing it, IMO just having blue-green deployments with a load balancer is good enough.
No, but I am interested. I have an app that have internal state that taking many network loads and significant time to recover.
Well, sometimes blue-green is not possible, because not all servers are stateless and QoS does not allow to fix every bug with replacing deploy. I think it depends on the requirements of an application
Yeah I agree f.ex. LiveView projects would really benefit from painless and transparent state handoff mechanism that does NOT involve silent reconnecting.
Session handoff can be done with blue-green deployment, but it requires some extra code to do such thing (for example,
horde?) solves this problem right away).
But the problem is that this handoff mechanism can have a bug in it, and that’s where hot-reload can be used. However, hot-reloading requires a lot of resources and manpower to maintain. And it requires a ton of time to write a universal solution for this
I am not experienced in hot upgrades but I’ve heard same thing you said repeated a number of other times and I admit this made me develop a remote (as in: not personally confirmed) bias to it.
IMO I think our current model of deploying apps is super outdated anyway e.g. if you need persistent WebSockets connections they absolutely must be managed by something else like a load balancer or a CDN-like service in front of your “real” app so as people never actually disconnect – and when you are redeploying your app the service on top should employ the blue-green deployment model and transparently hand-off the state and connections to the new app.
Same idea goes for DB connections and all other stateful components. Some cloud providers are doing this really well with attachable HDD / SDD volumes were you have literal zero downtime even if you issue a command like “please give me 50GB more”. Your app will never find out that it happened, it will just have access to the bigger storage medium right away.
…But that’s going in the territory of advanced computing R&D activities that nobody wants to champion, sadly.
I used hot deploys extensively at my prior job. I wrote up my thoughts on it a while back (Why Hot Deploys are Cool | Greg Vaughn’s Musings). The landscape has changed in the past couple of years with distillery no longer being the primary tool to build releases though.
No I use blue/green deployment but I’m interested to try hot-reload/recompilation.
AWS EBS does this… and when fronted by Kubernetes volumes, it’s incredibly easy to work with.
Sorry, the only other thing I gush about besides Elixir is Kubernetes (yes this is more an EBS feature, but I love how the Koob EBS volume driver completely abstracts the feature and just works)…