Curious what people think about where to host moderately complicated applications these days. (To scope that, I’ve got 2 multi-tenant phoenix apis, 3 postgres db servers, 2 small independent little services, rabbitmq, an opentelemetry collector and log shipper at the moment) Deployed into 3 environments.
Infrastructure as Code is a requirement, which unfortunately rules out fly, which is where we are now. I’m thinking that something like DigitalOcean on Kubernetes, or taking (another) risk on a platform like Render.
Anyone have experience with either of these platforms? Are there other simple options (ahem, not AWS) out there?
I have looked quite a lot at Vultr but I have not used them. There offering seems competitive and their network was fast when testing all the cloud providers.
Logflare is my pick either hosted or self hosted for logging using google bigquery, very economical.
Other than that I use fly for some things and getting projects up and running in non production quickly. I also host my own gitea on fly.io using sqlite backend rather than using Gitlab or GitHub. It’s very fast.
If you can qualify for the AWS startup $100k credit I wouldn’t ignore AWS.
My biggest challenge for security compliance is that only the major clouds like AWS, Azure or Google meet the requirements. So ultimately for anything of value it has to be those three.
Considering there is now Kubernetes backend for FLAME, I’d go with Kubernetes, just to have the option. I don’t think anyone has written a DigitalOcean backend yet.
We run our app in Kubernetes (AWS EKS) at work. Clustering with libcluster works great.
Vultr I was unaware of. I’ll definitely throw them in the mix.
And yes, when I said digital ocean, I meant for kubernetes support. I didn’t realize Flame could be used for FLAME directly. Intriguing.
Fly also just launched FKS ( Fly Kubernetes Service) if kubernetes is a hard requirement for you. Also note that folks run kubernetes themselves on fly already so we’re probably lower level than you’re thinking. We can run anything in a container. My guess is we tick all the IaC boxes for you already.
I don’t trust, the best I can do is verify the independent audit reports and compliance certifications and make risk based decisions. The security posture of the AWS, Azure and Google clouds is sufficient for most government systems and meeting privacy requirements of most countries (non defence and national security use cases) but it is something you have to consider on a service by service basis as not all services are created equal or they contain elements that are not managed at the same level.
A classic one is Microsoft Purview, a data governance solution which crawls all over your entire digital estate and yet doesn’t meet government compliance standards. Several times the resulting risk assessment is a handbrake on the plans of a newly established business unit tasked with implementing enterprise data governance, and of course like most corporate lemmings they reach for Microsoft. “Not happy” directors.
Like that even means anything. They’re very happy to outsource liability, like most governments. I can bet a good dollar that exactly zero proper auditing was done by government(-affiliated) entities.
Back on topic, I was wondering about Vultr for a long time. I’ll need some extra backup servers and some distributed workers sometime in 2024 so it’s either going to be them or Hetzner.
Vultr has good global coverage including Australia and SE Asia and a reasonable level of security compliance as well which is also improved recently.
Vultr are also good for HA setups with floating IP which avoids the cost and upper limits on throughout of cloud service load balancers and additional latency.
Vultr also actively support FreeBSD which I prefer for servers due to better networking with Erlang BEAM as FreeBSD kqueue is more efficient and scalable than anything Linux does. The total observaility of every subsystem is something I could not pass up.
AWS also provide a good option with their Firecracker VMs which boot FreeBSD in 25ms. With such a fast boot time you can instantly demand scale and failover without paying for headroom or contingency.
For deployment I also prefer FreeBSD jail’s which are also supported by Nomad.
I would not trust hetzner with anything serious, at the end of the day there is a price for their dirt-cheap servers. Dev servers and pipelines I would go for hetzner all day long, especially with their new ARM servers.
One of my previous employer had some serious problems back in the day hosting hardware in their datacenters, it was so bad that at some point they were planning on taking legal action.
Well in Australia at least the cloud providers and the entire supply chain do get to enjoy a rigorous assurance process if they want to participate.
Datacenters must be certified strategic, which involves multiple Agency assessments including intelligence agencies. Similarly cloud service providers are also subject to specific audits. Liability for data service providers is unlimited as is any important cloud service…
.
That’s an excellent insight on FreeBSD. Thanks! I recently started looking around to escape the mess the Linux is, and apart from a bit chaotic firewall frontends story, it seems like a breath of fresh air!
I’m curious if it would be possible to apply Kamal’s approach on FreeBSD with Elixir releases. That is, build and pack release and do blue/green deployment. That would be nice!
(Given that FreeBSD doesn’t support OCI containers as far as I know).
For deployment I also prefer FreeBSD jail’s which are also supported by Nomad .
I switched to FreeBSD - Have no experience in deploying elixir/freebsd to other services as I host on my own hardware.
So far, I’m so happy with the simplicity given with ZFS, jails and amazed how far I’m able to go without all the complexity added by docker.
I sure like the features given by containers like docker - and so far I’m able to solve most of the problems with jails that dock helped me with in the past. Instead of config & docker files - I use shell-scripts.
For me - it has for sure been worth it to really explore the freebsd alternative.
I’m a long-standing BSD fan, even though I never had a chance to use it in production because of business reasons (time to market, cost of maintaining own infra, etc.)
For me, it’s much more sane and straightforward than Linux, even if it falls short in some respects.
I have a product that each customer run their own instance of.
The planned “load” is perhaps best described as - 50-60 instances running a elixir-service in a container. To that Im using Haproxy and postgresql.
I have no data of interest when it comes to network load or I/O load.
My current setup is
Ampere Altra - 128core (ARM Neoverse-N1 r3p1)
384Mb of ram.
a Zpool of 6disk SSD’s running ZFS raidz2.
My containers are run in jails.
At the moment I have written my own very simple scripts to run tests and builds of new containers when new commits arrives to the repo.
The deployment is “semi” automated at the moment. Evaluating available tools for assisting blue/green deployments.
Running on my hardware has so far been very straight forward and simple.
As the load Im having is most likely to be very static. I have no need to achieve elastic scaling of resources.
Time to market - doesnt really sound to be relevant factor when it comes to selection of OS.
When it comes to cost. sure. There is a cost of running my own infra. So far Im more agile deploying new instances with the tools at hand then doing the same operation using google-cloud. So for me - It has been cost reduction in both time and “rent”.
The scripts are so simple that I wouldnt consider it as adding technical debt - its on the level that Im actually considering it as documentation of my test/build/deploy process.
When it comes to stats handling loads, I believe Netflix has some presentations on that topic more interesting then what Im having.
