Out of memory error when deploying Geolix config when deploying to fly.io

Phoenix Forum Questions / Help
1

I’ve upgraded my Pheonix app to 1.6 so I can test deployment with https://fly.io. Almost everything is up and running. However, when I re-add my Geolix config:

config :geolix, databases: [%{
  id: :dynamic_city,
  adapter: Geolix.Adapter.MMDB2,
  source: nil,
  init: {Zoinks.Geolix, :database}
}]

I get the following error message after the new image is scaled up:

2021-12-29T03:30:30.000 [info] Configuring virtual machine
2021-12-29T03:30:30.000 [info] Pulling container image
2021-12-29T03:30:31.000 [info] Unpacking image
2021-12-29T03:30:31.000 [info] Preparing kernel init
2021-12-29T03:30:31.000 [info] Configuring firecracker
2021-12-29T03:30:32.000 [info] Starting virtual machine
2021-12-29T03:30:32.000 [info] Starting init (commit: 7943db6)...
2021-12-29T03:30:32.000 [info] Preparing to run: `/bin/sh -c /app/bin/server` as nobody
2021-12-29T03:30:32.000 [info] 2021/12/29 03:30:32 listening on [fdaa:0:32ca:a7b:2984:8cc8:6b30:2]:22 (DNS: [fdaa::3]:53)
2021-12-29T03:30:33.000 [info] Reaped child process with pid: 547, exit code: 0
2021-12-29T03:30:36.000 [info] [    3.980196] Out of memory: Killed process 515 (beam.smp) total-vm:1890308kB, anon-rss:168232kB, file-rss:0kB, shmem-rss:82096kB, UID:65534 pgtables:672kB oom_score_adj:0
2021-12-29T03:30:36.000 [info] Main child exited normally with code: 137
2021-12-29T03:30:36.000 [info] Seaped child process with pid: 568 and signal: SIGUSR1, core dumped? false
2021-12-29T03:30:36.000 [info] Starting clean up.
2021-12-29T03:30:43.000 [info] Starting instance
2021-12-29T03:30:43.000 [info] Configuring virtual machine
2021-12-29T03:30:43.000 [info] Pulling container image
2021-12-29T03:30:44.000 [info] Unpacking image
2021-12-29T03:30:44.000 [info] Preparing kernel init
2021-12-29T03:30:44.000 [info] Configuring firecracker
2021-12-29T03:30:45.000 [info] Starting virtual machine
2021-12-29T03:30:45.000 [info] Starting init (commit: 7943db6)...
2021-12-29T03:30:45.000 [info] Preparing to run: `/bin/sh -c /app/bin/server` as nobody
2021-12-29T03:30:45.000 [info] 2021/12/29 03:30:45 listening on [fdaa:0:32ca:a7b:2984:8cc8:6b30:2]:22 (DNS: [fdaa::3]:53)
2021-12-29T03:30:46.000 [info] Reaped child process with pid: 547, exit code: 0
2021-12-29T03:30:47.000 [error] Health check status changed 'passing' => 'critical'
2021-12-29T03:30:48.000 [info] Reaped child process with pid: 568 and signal: SIGUSR1, core dumped? false
2021-12-29T03:30:48.000 [info] [    3.230681] Out of memory: Killed process 515 (beam.smp) total-vm:1890712kB, anon-rss:167988kB, file-rss:4kB, shmem-rss:82264kB, UID:65534 pgtables:680kB oom_score_adj:0
2021-12-29T03:30:49.000 [info] Main child exited normally with code: 137
2021-12-29T03:30:49.000 [info] Starting clean up.

1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total]
v12 failed - Failed due to unhealthy allocations - not rolling back to stable job version 12 as current job has same specification

So I assume this is happening because Geolix runs on startup and the memory required exceeds that of the basic firecracker. Is there something I can do to load Geolix differently - such that I can use the default firecracker?

2

There’s a startup_sync option in Geolix 1.1 and higher, but that may just delay the problem until a little later in the boot process.

3

I have tried startup_sync: true and you are correct. I’m now thinking that my only option is to delay loading the geoip database until much later in the process. However, at that point in time I’m assuming that the firecracker will terminate and a new instance will be created?

If that’s true, my only option would be to reduce the data set. My city database is 55MB - I’m guessing that there isn’t much I can do about that. Maybe I could base my data on country and ask the user to choose the correct time zone manually…

4

If you think you don’t need that much more memory, you can try deploy in 32 bit. I have a blog post about it: Deploy a 32bit application to fly.io

5

Oh, very cool. I’ll take a look at this later today!

6

There’s a lot more to building a 32-bit docker image than I thought!

While I was contemplating the process you came up with, I realised that I could insert the geoip data (MaxMind) into the database.

I’m retrieving the time zone once when the user is created, so a hit to the database (in my case) is negligible.

Thanks for the advice - it could be handy if I get into a jam later!