Out of memory , kernel cannot allocate memory

joserp93

Hello everyone. I am trying to use Task.async stream to upload files by S3.put_object. Like in this example:

upload_file = fn {src_path, dest_path} ->
  S3.put_object("my_bucket", dest_path, File.read!(src_path))
  |> ExAws.request!
end

paths = %{"path/to/src0" => "path/to/dest0", "path/to/src1" => "path/to/dest1"}

paths
|> Task.async_stream(upload_file, max_concurrency: 10)
|> Stream.run

But I have a problem. If there are many files, the kernel starts to run out of memory and the rpi restarts, I have the log. If anyone has an idea of how to increase the memory or was able to solve this, it would be very helpful.

[ 1163.448679] 2_scheduler invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
[ 1163.459534] 2_scheduler cpuset=/ mems_allowed=0
[ 1163.464301] CPU: 2 PID: 104 Comm: 2_scheduler Tainted: G         C        4.19.118 #2
[ 1163.472332] Hardware name: BCM2835
[ 1163.475831] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1163.483768] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1163.491173] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1163.498719] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1163.506828] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1163.515201] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1163.524195] [<c02ef534>] (__alloc_pages_nodemask) from [<c0682760>] (skb_page_frag_refill+0xe4/0xfc)
[ 1163.533507] [<c0682760>] (skb_page_frag_refill) from [<c06841b8>] (sk_page_frag_refill+0x14/0x6c)
[ 1163.542590] [<c06841b8>] (sk_page_frag_refill) from [<c06f022c>] (tcp_sendmsg_locked+0x43c/0xc18)
[ 1163.551670] [<c06f022c>] (tcp_sendmsg_locked) from [<c06f0a30>] (tcp_sendmsg+0x28/0x3c)
[ 1163.559873] [<c06f0a30>] (tcp_sendmsg) from [<c06800c4>] (sock_sendmsg+0x14/0x24)
[ 1163.567510] [<c06800c4>] (sock_sendmsg) from [<c068016c>] (sock_write_iter+0x98/0xf4)
[ 1163.575534] [<c068016c>] (sock_write_iter) from [<c0350250>] (do_iter_readv_writev+0x118/0x194)
[ 1163.584437] [<c0350250>] (do_iter_readv_writev) from [<c0350be4>] (do_iter_write+0x84/0x190)
[ 1163.593075] [<c0350be4>] (do_iter_write) from [<c0350d68>] (vfs_writev+0x78/0xe4)
[ 1163.600708] [<c0350d68>] (vfs_writev) from [<c0350e3c>] (do_writev+0x68/0xf0)
[ 1163.608023] [<c0350e3c>] (do_writev) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1163.615741] Exception stack(0xc1851fa8 to 0xc1851ff0)
[ 1163.620903] 1fa0:                   00000010 b6bc00e0 00000031 b6bc00e0 00000010 00000000
[ 1163.629274] 1fc0: 00000010 b6bc00e0 b403c920 00000092 0005ad95 b6bc00d0 a9b42a0c 00082903
[ 1163.637610] 1fe0: 00000000 b403b9b0 b403c920 b6da3218
[ 1163.643240] Mem-Info:
[ 1163.645594] active_anon:19493 inactive_anon:2 isolated_anon:0
[ 1163.645594]  active_file:1477 inactive_file:9296 isolated_file:0
[ 1163.645594]  unevictable:0 dirty:0 writeback:5 unstable:0
[ 1163.645594]  slab_reclaimable:1130 slab_unreclaimable:3224
[ 1163.645594]  mapped:2121 shmem:3 pagetables:200 bounce:0
[ 1163.645594]  free:755040 free_pcp:143 free_cma:15788
[ 1163.679473] Node 0 active_anon:79156kB inactive_anon:8kB active_file:2800kB inactive_file:34808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8484kB dirty:0kB writeback:20kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1163.702629] DMA free:64628kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:112kB inactive_file:20kB unevictable:0kB writepending:20kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:1000kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1163.729648] lowmem_reserve[]: 0 0 3008 3008
[ 1163.734061] HighMem free:2958876kB min:512kB low:19644kB high:38776kB active_anon:79040kB inactive_anon:8kB active_file:2856kB inactive_file:34612kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:800kB bounce:0kB free_pcp:1556kB local_pcp:0kB free_cma:0kB
[ 1163.762704] lowmem_reserve[]: 0 0 0 0
[ 1163.766577] DMA: 87*4kB (UEHC) 55*8kB (UEHC) 15*16kB (UEHC) 21*32kB (UEHC) 8*64kB (UEC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 65316kB
[ 1163.782491] HighMem: 126*4kB (UM) 120*8kB (UM) 151*16kB (UM) 52*32kB (UM) 48*64kB (UM) 22*128kB (UM) 21*256kB (UM) 19*512kB (UM) 12*1024kB (UM) 10*2048kB (UM) 708*4096kB (UM) = 2959272kB
[ 1163.799477] 2754 total pagecache pages
[ 1163.799480] 0 pages in swap cache
[ 1163.799483] Swap cache stats: add 0, delete 0, find 0/0
[ 1163.799485] Free swap  = 0kB
[ 1163.799487] Total swap = 0kB
[ 1163.799490] 802816 pages RAM
[ 1163.799492] 770048 pages HighMem/MovableOnly
[ 1163.799494] 9473 pages reserved
[ 1163.799496] 16384 pages cma reserved
[ 1163.799499] Tasks state (memory values in pages):
[ 1163.799502] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1163.799516] [     88]     0    88      418       48    28672        0             0 nbtty
[ 1163.854315] [     95]     0    95      502       19    28672        0             0 rngd
[ 1163.854321] [     96]     0    96      483      274    32768        0             0 nbtty
[ 1163.870991] [     97]     0    97    66114    27965   585728        0             0 beam.smp
[ 1163.870996] [    102]     0   102      433      255    24576        0             0 erl_child_setup
[ 1163.871001] [    127]     0   127      399       43    20480        0             0 heart
[ 1163.871005] [    131]     0   131      454      244    32768        0             0 uevent
[ 1163.871009] [    134]     0   134      398       41    20480        0             0 kmsg_tailer
[ 1163.871012] [    140]     0   140      455       51    24576        0             0 if_monitor
[ 1163.871016] [    153]     0   153      400       45    24576        0             0 muontrap
[ 1163.871020] [    154]     0   154      577      374    28672        0             0 ntpd
[ 1163.871025] [    176]     0   176      400       45    24576        0             0 muontrap
[ 1163.871029] [    177]     0   177      489       58    28672        0             0 udhcpc
[ 1163.871036] Out of memory: Kill process 97 (beam.smp) score 34 or sacrifice child
[ 1163.871049] Killed process 102 (erl_child_setup) total-vm:1732kB, anon-rss:56kB, file-rss:964kB, shmem-rss:0kB
[ 1163.871251] oom_reaper: reaped process 102 (erl_child_setup), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1163.892263] heart: Erlang is crashing .. (waiting for crash dump file)
[ 1163.999468] init invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=2, oom_score_adj=0
[ 1164.017310] init cpuset=/ mems_allowed=0
[ 1164.021332] CPU: 3 PID: 1 Comm: init Tainted: G         C        4.19.118 #2
[ 1164.028555] Hardware name: BCM2835
[ 1164.032055] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1164.039989] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1164.047394] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1164.054938] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1164.063044] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1164.071415] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1164.080370] [<c02ef534>] (__alloc_pages_nodemask) from [<c0311380>] (kmalloc_order+0x20/0x30)
[ 1164.089094] [<c0311380>] (kmalloc_order) from [<c03113ac>] (kmalloc_order_trace+0x1c/0xf4)
[ 1164.097555] [<c03113ac>] (kmalloc_order_trace) from [<c026b8a4>] (devkmsg_open+0x50/0xf0)
[ 1164.105928] [<c026b8a4>] (devkmsg_open) from [<c03567d8>] (chrdev_open+0xb0/0x19c)
[ 1164.113651] [<c03567d8>] (chrdev_open) from [<c034e4c4>] (do_dentry_open+0x114/0x3bc)
[ 1164.121673] [<c034e4c4>] (do_dentry_open) from [<c035fe9c>] (path_openat+0x2bc/0x109c)
[ 1164.129780] [<c035fe9c>] (path_openat) from [<c0360cf0>] (do_filp_open+0x74/0xe8)
[ 1164.137411] [<c0360cf0>] (do_filp_open) from [<c034fd24>] (do_sys_open+0x17c/0x200)
[ 1164.145254] [<c034fd24>] (do_sys_open) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1164.153181] Exception stack(0xc1c33fa8 to 0xc1c33ff0)
[ 1164.158341] 3fa0:                   0000001b b6f20d10 ffffff9c 00016118 000a0001 00000000
[ 1164.166710] 3fc0: 0000001b b6f20d10 0000001b 00000142 befddd9c befddd98 00015d24 00015d3c
[ 1164.175078] 3fe0: 0002707c befdd660 000132c8 b6e71248
[ 1164.180301] Mem-Info:
[ 1164.182818] active_anon:75 inactive_anon:2 isolated_anon:0
[ 1164.182818]  active_file:396 inactive_file:183 isolated_file:0
[ 1164.182818]  unevictable:0 dirty:1 writeback:0 unstable:0
[ 1164.182818]  slab_reclaimable:1131 slab_unreclaimable:3144
[ 1164.182818]  mapped:372 shmem:3 pagetables:29 bounce:0
[ 1164.182818]  free:785910 free_pcp:4 free_cma:15788
[ 1164.215521] Node 0 active_anon:300kB inactive_anon:8kB active_file:1584kB inactive_file:732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1488kB dirty:4kB writeback:0kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1164.238133] DMA free:68076kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:72kB inactive_file:40kB unevictable:0kB writepending:4kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:728kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1164.264834] lowmem_reserve[]: 0 0 3008 3008
[ 1164.269122] HighMem free:3075648kB min:512kB low:19644kB high:38776kB active_anon:300kB inactive_anon:8kB active_file:1664kB inactive_file:1120kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:116kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB
[ 1164.297194] lowmem_reserve[]: 0 0 0 0
[ 1164.300981] DMA: 140*4kB (UEHC) 154*8kB (UEHC) 68*16kB (UEHC) 34*32kB (UEHC) 17*64kB (UEHC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 68160kB
[ 1164.317231] HighMem: 188*4kB (UM) 145*8kB (UM) 129*16kB (UM) 121*32kB (UM) 111*64kB (UM) 88*128kB (UM) 72*256kB (UM) 48*512kB (M) 26*1024kB (M) 11*2048kB (M) 722*4096kB (UM) = 3075688kB
[ 1164.334124] 584 total pagecache pages
[ 1164.337873] 0 pages in swap cache
[ 1164.341271] Swap cache stats: add 0, delete 0, find 0/0
[ 1164.346646] Free swap  = 0kB
[ 1164.349601] Total swap = 0kB
[ 1164.352562] 802816 pages RAM
[ 1164.355517] 770048 pages HighMem/MovableOnly
[ 1164.359917] 9473 pages reserved
[ 1164.363179] 16384 pages cma reserved
[ 1164.366839] Unreclaimable slab info:
[ 1164.370532] Name                      Used          Total
[ 1164.376062] RAWv6                     15KB         15KB
[ 1164.381431] UDPv6                     15KB         15KB
[ 1164.386774] TCPv6                     31KB         31KB
[ 1164.392151] mqueue_inode_cache         15KB         15KB
[ 1164.397578] UNIX                      63KB         63KB
[ 1164.402926] RAW                       31KB         31KB
[ 1164.408296] UDP                       61KB         61KB
[ 1164.413638] tw_sock_TCP                3KB          3KB
[ 1164.419009] TCP                      121KB        121KB
[ 1164.424359] eventpoll_pwq             15KB         15KB
[ 1164.429731] request_queue             62KB         62KB
[ 1164.435073] blkdev_ioc                16KB         16KB
[ 1164.440442] biovec-max               300KB        300KB
[ 1164.445791] biovec-128               126KB        126KB
[ 1164.451162] biovec-64                 63KB         63KB
[ 1164.456503] biovec-16                165KB        165KB
[ 1164.461879] skbuff_head_cache        283KB        283KB
[ 1164.467215] configfs_dir_cache          3KB          3KB
[ 1164.472652] file_lock_cache           15KB         15KB
[ 1164.478024] shmem_inode_cache        108KB        108KB
[ 1164.483367] proc_dir_entry            56KB         56KB
[ 1164.488737] pde_opener                15KB         15KB
[ 1164.494088] seq_file                  15KB         15KB
[ 1164.499458] sigqueue                  15KB         15KB
[ 1164.504800] kernfs_node_cache       1346KB       1346KB
[ 1164.510170] filp                     263KB        263KB
[ 1164.515520] names_cache              128KB        128KB
[ 1164.520888] key_jar                   16KB         16KB
[ 1164.526233] vm_area_struct           154KB        154KB
[ 1164.531609] mm_struct                 32KB         32KB
[ 1164.536945] files_cache               16KB         16KB
[ 1164.542294] signal_cache             109KB        109KB
[ 1164.547664] sighand_cache            220KB        220KB
[ 1164.553006] task_struct              284KB        284KB
[ 1164.558377] cred_jar                  36KB         36KB
[ 1164.563725] anon_vma_chain            28KB         28KB
[ 1164.569095] anon_vma                  31KB         31KB
[ 1164.574436] pid                       16KB         16KB
[ 1164.579804] trace_event_file          35KB         35KB
[ 1164.585152] ftrace_event_field         84KB         84KB
[ 1164.590609] pool_workqueue            16KB         16KB
[ 1164.595952] task_group                83KB        121KB
[ 1164.601326] kmalloc-8192             160KB        160KB
[ 1164.606674] kmalloc-4096            1504KB       1504KB
[ 1164.612017] kmalloc-2048             480KB        480KB
[ 1164.617387] kmalloc-1024             304KB        304KB
[ 1164.622729] kmalloc-512              452KB        488KB
[ 1164.628098] kmalloc-256               76KB         76KB
[ 1164.633447] kmalloc-192               86KB         86KB
[ 1164.638818] kmalloc-128              184KB        184KB
[ 1164.644161] kmalloc-64              4569KB       4584KB
[ 1164.649533] kmem_cache_node            8KB          8KB
[ 1164.654883] kmem_cache                24KB         24KB
[ 1164.660252] Tasks state (memory values in pages):
[ 1164.665064] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1164.673889] [     95]     0    95      502       19    28672        0             0 rngd
[ 1164.682154] [    127]     0   127      399       43    20480        0             0 heart
[ 1164.690526] Out of memory: Kill process 127 (heart) score 0 or sacrifice child
[ 1164.697939] Killed process 127 (heart) total-vm:1596kB, anon-rss:28kB, file-rss:144kB, shmem-rss:0kB
[ 1164.707331] erlinit: Erlang VM exited

[ 1164.707435] oom_reaper: reaped process 127 (heart), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1164.711202] erlinit: Sending SIGTERM to all processes

[ 1164.720354] watchdog: watchdog0: watchdog did not stop!
[ 1165.745917] erlinit: Sending SIGKILL to all processes

[ 1165.881653] erlinit: umount 0 failed: No such file or directory

[ 1166.387850] reboot: Restarting system

6 comments

/nerves #nerves-system

0 1354 6

2022-11-29 16:02:01 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

LostKobrakai

I’d suggest streaming to S3 instead of reading the whole file into memory, e.g. using ExAWS.S3.upload/4

Post #1

outlog

+1 ExAws.S3 — ExAws.S3 v2.5.9 looks like put_object is not streaming the file..

Post #2

LostKobrakai

File.read!/1 for sure isn’t.

Post #3

joserp93

ok, It would be using this example:

"path/to/big/file"
|> S3.Upload.stream_file
|> S3.upload("my-bucket", "path/on/s3")
|> ExAws.request! #=> :done

Post #4

joserp93

It didn’t work for me, the same thing keeps happening to me.

Post #5

al2o3cr

The code in the initial post will start up to 10 Tasks that are each doing an S3.Upload.stream_file.

Each S3.upload also uses Task.async_stream to upload chunks in parallel:

https://github.com/ex-aws/ex_aws_s3/blob/34bdff67da81835c4bd5b1361ec81b6765c462ed/lib/ex_aws/s3/upload.ex#L122-L124

With that default of 4 that means, if there are enough files and they’re big enough, you’ll end up with 40 Tasks that each want to read a 5MB chunk of data before sending it to S3. That’s 200MB of memory just for the chunks!

Post #6

— All posts loaded —