Deploying app in Fly.io: bom [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

Questions / Help
1

I have a backend chat app that I have been trying to deploy on fly.io

No matter how much I scale the app it keeps crashing. How do I debug this?

2025-01-20T13:00:33Z app[2874de0a105d68] bom [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-01-20T13:00:33Z app[2874de0a105d68] bom [info][    4.949595] reboot: Restarting system
2025-01-20T13:00:33Z runner[2874de0a105d68] bom [info]Successfully prepared image registry.fly.io/app-name:deployment-01JHZ2AKKB98PD1HJ03NNVVSFM (4.310267752s)
2025-01-20T13:00:33Z app[2874de0a105d68] bom [info]Out of memory: Killed process
2025-01-20T13:00:34Z runner[2874de0a105d68] bom [info]Configuring firecracker
2025-01-20T13:00:39Z app[2874de0a105d68] bom [info]2025-01-20T13:00:39.902743399 [01JJ1WSQPYFXRPCK61Q9FCY09A:main] Running Firecracker v1.7.0
2025-01-20T13:00:39Z app[2874de0a105d68] bom [info]2025-01-20T13:00:39.902743399 [01JJ1WSQPYFXRPCK61Q9FCY09A:main] Running Firecracker v1.7.0
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO Starting init (commit: 3a6e10d0)...
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO Preparing to run: `/app/bin/server` as nobody
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO [fly api proxy] listening at /.fly/api
2025-01-20T13:00:40Z runner[2874de0a105d68] bom [info]Machine created and started in 11.638s
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO Starting init (commit: 3a6e10d0)...
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO Preparing to run: `/app/bin/server` as nobody
2025-01-20T13:00:40Z app[2874de0a105d68] bom [info] INFO [fly api proxy] listening at /.fly/api
2025-01-20T13:00:40Z runner[2874de0a105d68] bom [info]Machine created and started in 11.638s
2025-01-20T13:00:41Z app[2874de0a105d68] bom [info]2025/01/20 13:00:41 INFO SSH listening listen_address=[fdaa:c:af60:a7b:177:87f8:270e:2]:22 dns_server=[fdaa::3]:53
2025-01-20T13:00:41Z app[2874de0a105d68] bom [info]2025/01/20 13:00:41 INFO SSH listening listen_address=[fdaa:c:af60:a7b:177:87f8:270e:2]:22 dns_server=[fdaa::3]:53
2025-01-20T13:00:43Z app[2874de0a105d68] bom [info]🚀 Server running at: http://localhost:4000
2025-01-20T13:00:43Z app[2874de0a105d68] bom [info] WARN Reaped child process with pid: 686 and signal: SIGUSR1, core dumped? false
2025-01-20T13:07:28Z proxy[2874de0a105d68] bom [info]App app-name has excess capacity, autostopping machine 2874de0a105d68. 0 out of 1 machines left running (region=bom, process group=app)
2025-01-20T13:07:28Z app[2874de0a105d68] bom [info] INFO Sending signal SIGTERM to main child process w/ PID 629
2025-01-20T13:07:28Z proxy[2874de0a105d68] bom [info]App app-name has excess capacity, autostopping machine 2874de0a105d68. 0 out of 1 machines left running (region=bom, process group=app)
2025-01-20T13:07:28Z app[2874de0a105d68] bom [info] INFO Sending signal SIGTERM to main child process w/ PID 629
2025-01-20T13:07:29Z app[2874de0a105d68] bom [info] WARN Reaped child process with pid: 688 and signal: SIGUSR1, core dumped? false
2025-01-20T13:07:29Z app[2874de0a105d68] bom [info] WARN Reaped child process with pid: 688 and signal: SIGUSR1, core dumped? false
2025-01-20T13:07:30Z app[2874de0a105d68] bom [info] INFO Main child exited normally with code: 0
2025-01-20T13:07:30Z app[2874de0a105d68] bom [info] INFO Starting clean up.
2025-01-20T13:07:30Z app[2874de0a105d68] bom [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-01-20T13:07:30Z app[2874de0a105d68] bom [info][  410.429669] reboot: Restarting system
2

Hi, smells like memory leak. What do you do in Application.start method?

3

this is your main hint. the default server in fly only has 256mb, so if you do anything more than the most basic stuff, you’re going to need at least 512mb. usually 1gb is actually where “safe” starts. that applies for the DB also.

1 Like
4

Turns out that Fly was just killing the instance on zero activity.

Facing a different challenge now. When i try to redeploy the app using fly deploy it keeps running into below error. Any idea why?

2025-01-24T07:08:52Z proxy[2874de2c396308] bom [error][PC01] instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)

2025-01-24T07:08:52Z app[2874de2c396308] bom [info] INFO Main child exited with signal (with signal 'SIGKILL', core dumped? false)

2025-01-24T07:08:52Z app[2874de2c396308] bom [info] INFO Process appears to have been OOM killed!

2025-01-24T07:08:52Z app[2874de2c396308] bom [info] INFO Starting clean up.

2025-01-24T07:08:52Z app[2874de2c396308] bom [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2025-01-24T07:08:52Z app[2874de2c396308] bom [info][ 7.133147] reboot: Restarting system

2025-01-24T07:08:52Z app[185750da7143e8] bom [info] INFO Starting init (commit: 3a6e10d0)...

2025-01-24T07:08:53Z app[185750da7143e8] bom [info] INFO Preparing to run: `/app/bin/server` as nobody

2025-01-24T07:08:53Z app[185750da7143e8] bom [info] INFO [fly api proxy] listening at /.fly/api

2025-01-24T07:08:53Z app[2874de2c396308] bom [info]Out of memory: Killed process

2025-01-24T07:08:53Z runner[185750da7143e8] bom [info]Machine started in 1.186s

2025-01-24T07:08:53Z app[185750da7143e8] bom [info]2025/01/24 07:08:53 INFO SSH listening listen_address=[fdaa:c:af60:a7b:177:2b82:db6:2]:22 dns_server=[fdaa::3]:53

2025-01-24T07:08:54Z proxy[185750da7143e8] bom [error][PC01] instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)
5

It still looks like a memory issue, and that’s there in the logs:

2025-01-24T07:08:53Z app[2874de2c396308] bom [info]Out of memory: Killed process

How much memory does the machine have? Does it need to be increased?

1 Like
6

I have tried 256, 512 & 1024MB. In every instance it has same issue. Works fine when running it. When I try to redploy a new version, it throws error.

7

did you scale the memory from command line? that scaling command defaults back when deploying. you have to change the memory in fly.toml.

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1
3 Likes
8

Yes that was the issue… updated the toml file to 512MB, it was 256MB & works like a charm. Thanks!

1 Like