Hey there,
I wondering if WhatsApp folks or some of you have had some Packer images that you use in your production clusters.
Can you give more context on your question? And I might be ignorant here but what has this to do with Erlang at all?
I am referring to https://www.packer.io/ tool
My intention is to create/use some OS Image that is preconfigured and optimize to run Erlang VM.
That way I could use that OS Image as the baseline for the clustering and tweak things if I need to, but at least it is starting from something. Personally, I am not that good at infrastructure things and OS stuff, so it would be super helpful.
Does that make sense?
2 Likes
Iām not sure if that pairs well with using FreeBSD. Things will eventually go wrong somewhere and if you donāt know a lot (yet), then Iād optimize for the most external knowledge available. In the server world this likely means using ubuntu.
1 Like
Like that idea, that is why I put (or not) in the title since I am interested in starting the conversation about it and hopefully end up with some shared resources that we could collaborate on
1 Like
I think it pairs very well with FreeBSD. I use it for my personal projects and have never touched Linux at all. The idea of having to finding out which linux distribution I should take, packages I should installā¦ is keeping me from using linux. FreeBSD is very simple, well documented and battle proofed. People on this Forum are wondering why Elixir and Erlang are not adopted more widely despite hugely successfull companies like WhatsApp are or were using it. Iām wondering the same about FreeBSD.
2 Likes
Even 1 hour ago I was thinking what could be the most stripped OS I could run Elixir on, thus I am really interest in seeing were this discussions goes 
I think the goal would be to have an OS base image with the small foot print possible to run Erlang, aka less stuff installed less attack vectors that can be exploited to break into your production server.
1 Like
Yeah that is why I am curious to know if Erlangers have some resources that I could leverage, and hopefully use Packer (or something alike) to automate a bit
1 Like
After trying out a whole bunch of Linux and BSD distros in Virtual-Box this week, I wish there was a leaner, simpler way to just start building in Beam/Elixir/Phoenix without messing with Linux or BSD configuration.
Beam+Elixir+Phoenix running directly in a VM or Unikernel would be ideal. I have heard of LING (Erlang on Xen), but the project is no longer active.
May I suggest changing the title of this thread to - āMinimalist, Deterministic OS Builds for Elixirā to better reflect the concerns already raised here.
2 Likes
Well we can build Linux from scratch:
http://www.linuxfromscratch.org/
Linux From Scratch (LFS) is a project that provides you with step-by-step instructions for building your own custom Linux system, entirely from source code.
Currently, the Linux From Scratch organization consists of the following subprojects:
- LFS :: Linux From Scratch is the main book, the base from which all other projects are derived.
- BLFS :: Beyond Linux From Scratch helps you extend your finished LFS installation into a more customized and usable system.
- ALFS :: Automated Linux From Scratch provides tools for automating and managing LFS and BLFS builds.
- CLFS :: Cross Linux From Scratch provides the means to cross-compile an LFS system on many types of systems.
- Hints :: The Hints project is a collection of documents that explain how to enhance your LFS system in ways that are not included in the LFS or BLFS books.
- Patches :: The Patches project serves as a central repository for all patches useful to an LFS user.
Who as the skills for it?
An alternative is to pick-up Debian and strip it down:
https://wiki.debian.org/ReduceDebian
Reducing the size of the Debian Installation Footprint
It may be useful to reduce the size of the installation footprint on Embedded systems, or on older computers or laptops with limited drive space, or in cases where a small installation is preferred. Minimal systems in general also carry security benefits because fewer packages means that there are fewer security exploits available.
This seems more easy to be achieved 
Or maybe just use Alpine?
Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.
In terms of security it has a very good track record when compared with Debian and Ubuntu in the scan results for known vulnerabilities on docker images.
The use of musl libc can be or not an issue, but maybe someone with more experience in the BEAM can tell us more?
1 Like
Iād like to see a general thread about FreeBSD being used with Elixir/Erlang too - thereās an older thread here:
But since we have more Erlangers (and WhatsApp team members) here now perhaps we could start a new thread. Would you like to do the honours @alchemist_ubi? 
Maybe something like:
Title: FreeBSD and Elixir/Erlang in 2021
Body:
Does anyone use FreeBSD with Elixir or Erlang? Do you have any tips or advice? Any potential issues or pitfalls we might need to look out for? Perhaps youāve considered FreeBSD? Why? And did you end up using it? If so, why? If not, why not!?
2 Likes
For the OS image you may came up with, you can then try to use some of the harden techniques from this bash script:
#!/bin/sh
set -x
set -e
#
# Docker build calls this script to harden the image during build.
#
# NOTE: To build on CircleCI, you must take care to keep the `find`
# command out of the /proc filesystem to avoid errors like:
#
# find: /proc/tty/driver: Permission denied
# lxc-start: The container failed to start.
# lxc-start: Additional information can be obtained by \
# setting the --logfile and --logpriority options.
# Remove existing crontabs, if any.
rm -fr /var/spool/cron
rm -fr /etc/crontabs
rm -fr /etc/periodic
# Remove all but a handful of admin commands.
find /sbin /usr/sbin ! -type d \
-a ! -name exim \
-a ! -name nologin \
-delete
# Remove world-writable permissions.
# This breaks apps that need to write to /tmp,
# such as ssh-agent.
find / -xdev -type d -perm +0002 -exec chmod o-w {} +
find / -xdev -type f -perm +0002 -exec chmod o-w {} +
# Remove unnecessary user accounts, including root.
sed -i -r '/^(exim)/!d' /etc/group
sed -i -r '/^(exim)/!d' /etc/passwd
# Disable interactive login for everybody.
sed -i -r 's#^(.*):[^:]*$#\1:/sbin/nologin#' /etc/passwd
sysdirs="
/bin
/etc
/lib
/sbin
/usr
"
# Remove apk configs.
find $sysdirs -xdev -regex '.*apk.*' -exec rm -fr {} +
# Remove crufty...
# /etc/shadow-
# /etc/passwd-
# /etc/group-
find $sysdirs -xdev -type f -regex '.*-$' -exec rm -f {} +
# Ensure system dirs are owned by root and not writable by anybody else.
find $sysdirs -xdev -type d \
-exec chown 0:0 {} \; \
-exec chmod 0755 {} \;
# Create spool dir so that exim user doesn't have to.
# Dockerfile treats this as a volume.
mkdir -p /var/spool/exim
chown -R exim:exim /var/spool/exim
chmod 0755 /var/spool/exim
# Remove suid bit from exim.
chmod u-s /usr/sbin/exim
# Remove all suid files.
find $sysdirs -xdev -type f -a -perm +4000 -delete
# Remove other programs that could be dangerous.
find $sysdirs -xdev \( \
-name hexdump -o \
-name chgrp -o \
-name chmod -o \
-name chown -o \
-name ln -o \
-name od -o \
-name strings -o \
-name su \
\) -delete
# Remove init scripts since we do not use them.
rm -fr /etc/init.d
rm -fr /lib/rc
rm -fr /etc/conf.d
rm -fr /etc/inittab
rm -fr /etc/runlevels
rm -fr /etc/rc.conf
# Remove kernel tunables since we do not need them.
rm -fr /etc/sysctl*
rm -fr /etc/modprobe.d
rm -fr /etc/modules
rm -fr /etc/mdev.conf
rm -fr /etc/acpi
# Remove root homedir since we do not need it.
rm -fr /root
# Remove fstab since we do not need it.
rm -f /etc/fstab
# Remove broken symlinks (because we removed the targets above).
find $sysdirs -xdev -type l -exec test ! -e {} \; -delete
Found a good resource for hardening Ubuntu, that can be used as a good base resource for hardening any Debian based server:
Backup
Backup! Do not forget about backup. This can save previous time!
[ 1 articles ]
Disabled Services
Harden Ubuntu by disabling unnecessary services
[ 16 articles ]
Hardening
Hardening Ubuntu softwares and services
[ 11 articles ]
Initial Setup
Hardening Ubuntu Initial Setup - What are the first steps
[ 5 articles ]
Monitoring Tools
Use monitoring tools to figure out if something or someone do something illegal on your >server>
[ 6 articles ]
Optimization
Improve latency, optimize, speed up and improve number of request per seconds
[ 5 articles ]
Server Setup
Setup your Ubuntu Server - How to setup your server
[ 12 articles ]
Software
Software - Install useful software to help harden your Ubuntu server
[ 5 articles ]
Useful Commands
Useful commands you can use on your server
[ 4 articles ]
4 Likes
Amazing resources, man. Thanks.
3 Likes
I am more or less in the same path as @alchemist_ubi 
I want to find the most bare-bone Linux server distro from where I can build an image to easily deploy into VPS or bare-metal servers in order to run the BEAM in production, otherwise I will just resort to use a docker OS like RancherOS:
And f you fancy some more:
https://suchprogramming.com/barebones-linux-system/
In this post, I will describe the process Iāve used to create a very minimal Linux build. It is minimal for the sake of learning, to get a clearer picture of how barebones a operating system could be put together. The build will consist of an operating system kernel and an initial ram disk
that is followed by another post to build the ISO:
https://suchprogramming.com/barebones-linux-iso/
My post on Building a Barebones Linux System has been the most popular post on my blog so far. Last week a reader left a comment asking how the system described in that post could be loaded onto an ISO file along with GRUB to make it into a bootable image. I think thatās a pretty good follow-up so today I will share the process on how to do just that!
Then I remembered @NobbZ and others have advocate, here in the forum and/or at devtalk.com, for NixOS :
Find out why so many developers, system administrators and other enthusiasts use Nix and cannot imagine a world without it.
And I was surprised to find that they also have:
https://releases.nixos.org/nixops/nixops-1.6.1/manual/manual.html
NixOps is a tool for deploying NixOS machines in a network or cloud. It takes as input a declarative specification of a set of ālogicalā machines and then performs any necessary steps or actions to realise that specification: instantiate cloud machines, build and download dependencies, stop and start services, and so on.
So, I think I now need to spend some time learning it 
Debian can be very bare-bone, if you install the system with depootstrap or cdebootstrap
https://wiki.debian.org/Debootstrap
1 Like
I saw deboostrap yesterday on my research but I got the impression that was just a way to install Debian without an installation media, but seems I misunderstood it. Thanks for elucidating me
Some months ago I remember to to have found a Debian core that was the starting point for building other distros, but cannot find it anymore, thus now I am wondering if it was something done with this debootstrap 
GitHub - neeraj9/hello-elixir-rump: Say hello to ElixirRump an Elixir Microkernel powered by Rumprun unikernel is about as slim as you can get and still have a practical / shippable system, but it needs to be updated a lot for latest erlang + current netbsd base.
osv-apps/elixir at master Ā· cloudius-systems/osv-apps Ā· GitHub is another example of a similar strategy.
FWIW I love running Elixir on FreeBSD, with haproxy, pf + jails + zfs its perfect.
3 Likes
Thanks for the links, I will check them out
Do you have any write-up on it?
My concern with FreeBSD is that is an OS with all batteries included, thus lots of packages doing nothing in the system but that may open an attack vector into it. Do you know if is possible to strip down FreedBSD to itās bare-bones, aka to only have the packages you care about?
1 Like
WhatsApp no longer uses FreeBSD, due to moving away from bare-metal provisioning to Facebook-managed infrastructure.
In the bare-metal times, WhatsApp did not use Packer or similar tools. OS (FreeBSD) was simply installed and then configured. Since there were almost no tools/packages installed, there was no real need to constantly update or maintain anything. No software - no maintenance problems.
