Hello everyone. I am trying to use Task.async stream to upload files by S3.put_object. Like in this example:
upload_file = fn {src_path, dest_path} ->
S3.put_object("my_bucket", dest_path, File.read!(src_path))
|> ExAws.request!
end
paths = %{"path/to/src0" => "path/to/dest0", "path/to/src1" => "path/to/dest1"}
paths
|> Task.async_stream(upload_file, max_concurrency: 10)
|> Stream.run
But I have a problem. If there are many files, the kernel starts to run out of memory and the rpi restarts, I have the log. If anyone has an idea of how to increase the memory or was able to solve this, it would be very helpful.
[ 1163.448679] 2_scheduler invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
[ 1163.459534] 2_scheduler cpuset=/ mems_allowed=0
[ 1163.464301] CPU: 2 PID: 104 Comm: 2_scheduler Tainted: G C 4.19.118 #2
[ 1163.472332] Hardware name: BCM2835
[ 1163.475831] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1163.483768] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1163.491173] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1163.498719] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1163.506828] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1163.515201] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1163.524195] [<c02ef534>] (__alloc_pages_nodemask) from [<c0682760>] (skb_page_frag_refill+0xe4/0xfc)
[ 1163.533507] [<c0682760>] (skb_page_frag_refill) from [<c06841b8>] (sk_page_frag_refill+0x14/0x6c)
[ 1163.542590] [<c06841b8>] (sk_page_frag_refill) from [<c06f022c>] (tcp_sendmsg_locked+0x43c/0xc18)
[ 1163.551670] [<c06f022c>] (tcp_sendmsg_locked) from [<c06f0a30>] (tcp_sendmsg+0x28/0x3c)
[ 1163.559873] [<c06f0a30>] (tcp_sendmsg) from [<c06800c4>] (sock_sendmsg+0x14/0x24)
[ 1163.567510] [<c06800c4>] (sock_sendmsg) from [<c068016c>] (sock_write_iter+0x98/0xf4)
[ 1163.575534] [<c068016c>] (sock_write_iter) from [<c0350250>] (do_iter_readv_writev+0x118/0x194)
[ 1163.584437] [<c0350250>] (do_iter_readv_writev) from [<c0350be4>] (do_iter_write+0x84/0x190)
[ 1163.593075] [<c0350be4>] (do_iter_write) from [<c0350d68>] (vfs_writev+0x78/0xe4)
[ 1163.600708] [<c0350d68>] (vfs_writev) from [<c0350e3c>] (do_writev+0x68/0xf0)
[ 1163.608023] [<c0350e3c>] (do_writev) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1163.615741] Exception stack(0xc1851fa8 to 0xc1851ff0)
[ 1163.620903] 1fa0: 00000010 b6bc00e0 00000031 b6bc00e0 00000010 00000000
[ 1163.629274] 1fc0: 00000010 b6bc00e0 b403c920 00000092 0005ad95 b6bc00d0 a9b42a0c 00082903
[ 1163.637610] 1fe0: 00000000 b403b9b0 b403c920 b6da3218
[ 1163.643240] Mem-Info:
[ 1163.645594] active_anon:19493 inactive_anon:2 isolated_anon:0
[ 1163.645594] active_file:1477 inactive_file:9296 isolated_file:0
[ 1163.645594] unevictable:0 dirty:0 writeback:5 unstable:0
[ 1163.645594] slab_reclaimable:1130 slab_unreclaimable:3224
[ 1163.645594] mapped:2121 shmem:3 pagetables:200 bounce:0
[ 1163.645594] free:755040 free_pcp:143 free_cma:15788
[ 1163.679473] Node 0 active_anon:79156kB inactive_anon:8kB active_file:2800kB inactive_file:34808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8484kB dirty:0kB writeback:20kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1163.702629] DMA free:64628kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:112kB inactive_file:20kB unevictable:0kB writepending:20kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:1000kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1163.729648] lowmem_reserve[]: 0 0 3008 3008
[ 1163.734061] HighMem free:2958876kB min:512kB low:19644kB high:38776kB active_anon:79040kB inactive_anon:8kB active_file:2856kB inactive_file:34612kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:800kB bounce:0kB free_pcp:1556kB local_pcp:0kB free_cma:0kB
[ 1163.762704] lowmem_reserve[]: 0 0 0 0
[ 1163.766577] DMA: 87*4kB (UEHC) 55*8kB (UEHC) 15*16kB (UEHC) 21*32kB (UEHC) 8*64kB (UEC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 65316kB
[ 1163.782491] HighMem: 126*4kB (UM) 120*8kB (UM) 151*16kB (UM) 52*32kB (UM) 48*64kB (UM) 22*128kB (UM) 21*256kB (UM) 19*512kB (UM) 12*1024kB (UM) 10*2048kB (UM) 708*4096kB (UM) = 2959272kB
[ 1163.799477] 2754 total pagecache pages
[ 1163.799480] 0 pages in swap cache
[ 1163.799483] Swap cache stats: add 0, delete 0, find 0/0
[ 1163.799485] Free swap = 0kB
[ 1163.799487] Total swap = 0kB
[ 1163.799490] 802816 pages RAM
[ 1163.799492] 770048 pages HighMem/MovableOnly
[ 1163.799494] 9473 pages reserved
[ 1163.799496] 16384 pages cma reserved
[ 1163.799499] Tasks state (memory values in pages):
[ 1163.799502] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1163.799516] [ 88] 0 88 418 48 28672 0 0 nbtty
[ 1163.854315] [ 95] 0 95 502 19 28672 0 0 rngd
[ 1163.854321] [ 96] 0 96 483 274 32768 0 0 nbtty
[ 1163.870991] [ 97] 0 97 66114 27965 585728 0 0 beam.smp
[ 1163.870996] [ 102] 0 102 433 255 24576 0 0 erl_child_setup
[ 1163.871001] [ 127] 0 127 399 43 20480 0 0 heart
[ 1163.871005] [ 131] 0 131 454 244 32768 0 0 uevent
[ 1163.871009] [ 134] 0 134 398 41 20480 0 0 kmsg_tailer
[ 1163.871012] [ 140] 0 140 455 51 24576 0 0 if_monitor
[ 1163.871016] [ 153] 0 153 400 45 24576 0 0 muontrap
[ 1163.871020] [ 154] 0 154 577 374 28672 0 0 ntpd
[ 1163.871025] [ 176] 0 176 400 45 24576 0 0 muontrap
[ 1163.871029] [ 177] 0 177 489 58 28672 0 0 udhcpc
[ 1163.871036] Out of memory: Kill process 97 (beam.smp) score 34 or sacrifice child
[ 1163.871049] Killed process 102 (erl_child_setup) total-vm:1732kB, anon-rss:56kB, file-rss:964kB, shmem-rss:0kB
[ 1163.871251] oom_reaper: reaped process 102 (erl_child_setup), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1163.892263] heart: Erlang is crashing .. (waiting for crash dump file)
[ 1163.999468] init invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=2, oom_score_adj=0
[ 1164.017310] init cpuset=/ mems_allowed=0
[ 1164.021332] CPU: 3 PID: 1 Comm: init Tainted: G C 4.19.118 #2
[ 1164.028555] Hardware name: BCM2835
[ 1164.032055] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1164.039989] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1164.047394] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1164.054938] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1164.063044] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1164.071415] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1164.080370] [<c02ef534>] (__alloc_pages_nodemask) from [<c0311380>] (kmalloc_order+0x20/0x30)
[ 1164.089094] [<c0311380>] (kmalloc_order) from [<c03113ac>] (kmalloc_order_trace+0x1c/0xf4)
[ 1164.097555] [<c03113ac>] (kmalloc_order_trace) from [<c026b8a4>] (devkmsg_open+0x50/0xf0)
[ 1164.105928] [<c026b8a4>] (devkmsg_open) from [<c03567d8>] (chrdev_open+0xb0/0x19c)
[ 1164.113651] [<c03567d8>] (chrdev_open) from [<c034e4c4>] (do_dentry_open+0x114/0x3bc)
[ 1164.121673] [<c034e4c4>] (do_dentry_open) from [<c035fe9c>] (path_openat+0x2bc/0x109c)
[ 1164.129780] [<c035fe9c>] (path_openat) from [<c0360cf0>] (do_filp_open+0x74/0xe8)
[ 1164.137411] [<c0360cf0>] (do_filp_open) from [<c034fd24>] (do_sys_open+0x17c/0x200)
[ 1164.145254] [<c034fd24>] (do_sys_open) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1164.153181] Exception stack(0xc1c33fa8 to 0xc1c33ff0)
[ 1164.158341] 3fa0: 0000001b b6f20d10 ffffff9c 00016118 000a0001 00000000
[ 1164.166710] 3fc0: 0000001b b6f20d10 0000001b 00000142 befddd9c befddd98 00015d24 00015d3c
[ 1164.175078] 3fe0: 0002707c befdd660 000132c8 b6e71248
[ 1164.180301] Mem-Info:
[ 1164.182818] active_anon:75 inactive_anon:2 isolated_anon:0
[ 1164.182818] active_file:396 inactive_file:183 isolated_file:0
[ 1164.182818] unevictable:0 dirty:1 writeback:0 unstable:0
[ 1164.182818] slab_reclaimable:1131 slab_unreclaimable:3144
[ 1164.182818] mapped:372 shmem:3 pagetables:29 bounce:0
[ 1164.182818] free:785910 free_pcp:4 free_cma:15788
[ 1164.215521] Node 0 active_anon:300kB inactive_anon:8kB active_file:1584kB inactive_file:732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1488kB dirty:4kB writeback:0kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1164.238133] DMA free:68076kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:72kB inactive_file:40kB unevictable:0kB writepending:4kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:728kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1164.264834] lowmem_reserve[]: 0 0 3008 3008
[ 1164.269122] HighMem free:3075648kB min:512kB low:19644kB high:38776kB active_anon:300kB inactive_anon:8kB active_file:1664kB inactive_file:1120kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:116kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB
[ 1164.297194] lowmem_reserve[]: 0 0 0 0
[ 1164.300981] DMA: 140*4kB (UEHC) 154*8kB (UEHC) 68*16kB (UEHC) 34*32kB (UEHC) 17*64kB (UEHC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 68160kB
[ 1164.317231] HighMem: 188*4kB (UM) 145*8kB (UM) 129*16kB (UM) 121*32kB (UM) 111*64kB (UM) 88*128kB (UM) 72*256kB (UM) 48*512kB (M) 26*1024kB (M) 11*2048kB (M) 722*4096kB (UM) = 3075688kB
[ 1164.334124] 584 total pagecache pages
[ 1164.337873] 0 pages in swap cache
[ 1164.341271] Swap cache stats: add 0, delete 0, find 0/0
[ 1164.346646] Free swap = 0kB
[ 1164.349601] Total swap = 0kB
[ 1164.352562] 802816 pages RAM
[ 1164.355517] 770048 pages HighMem/MovableOnly
[ 1164.359917] 9473 pages reserved
[ 1164.363179] 16384 pages cma reserved
[ 1164.366839] Unreclaimable slab info:
[ 1164.370532] Name Used Total
[ 1164.376062] RAWv6 15KB 15KB
[ 1164.381431] UDPv6 15KB 15KB
[ 1164.386774] TCPv6 31KB 31KB
[ 1164.392151] mqueue_inode_cache 15KB 15KB
[ 1164.397578] UNIX 63KB 63KB
[ 1164.402926] RAW 31KB 31KB
[ 1164.408296] UDP 61KB 61KB
[ 1164.413638] tw_sock_TCP 3KB 3KB
[ 1164.419009] TCP 121KB 121KB
[ 1164.424359] eventpoll_pwq 15KB 15KB
[ 1164.429731] request_queue 62KB 62KB
[ 1164.435073] blkdev_ioc 16KB 16KB
[ 1164.440442] biovec-max 300KB 300KB
[ 1164.445791] biovec-128 126KB 126KB
[ 1164.451162] biovec-64 63KB 63KB
[ 1164.456503] biovec-16 165KB 165KB
[ 1164.461879] skbuff_head_cache 283KB 283KB
[ 1164.467215] configfs_dir_cache 3KB 3KB
[ 1164.472652] file_lock_cache 15KB 15KB
[ 1164.478024] shmem_inode_cache 108KB 108KB
[ 1164.483367] proc_dir_entry 56KB 56KB
[ 1164.488737] pde_opener 15KB 15KB
[ 1164.494088] seq_file 15KB 15KB
[ 1164.499458] sigqueue 15KB 15KB
[ 1164.504800] kernfs_node_cache 1346KB 1346KB
[ 1164.510170] filp 263KB 263KB
[ 1164.515520] names_cache 128KB 128KB
[ 1164.520888] key_jar 16KB 16KB
[ 1164.526233] vm_area_struct 154KB 154KB
[ 1164.531609] mm_struct 32KB 32KB
[ 1164.536945] files_cache 16KB 16KB
[ 1164.542294] signal_cache 109KB 109KB
[ 1164.547664] sighand_cache 220KB 220KB
[ 1164.553006] task_struct 284KB 284KB
[ 1164.558377] cred_jar 36KB 36KB
[ 1164.563725] anon_vma_chain 28KB 28KB
[ 1164.569095] anon_vma 31KB 31KB
[ 1164.574436] pid 16KB 16KB
[ 1164.579804] trace_event_file 35KB 35KB
[ 1164.585152] ftrace_event_field 84KB 84KB
[ 1164.590609] pool_workqueue 16KB 16KB
[ 1164.595952] task_group 83KB 121KB
[ 1164.601326] kmalloc-8192 160KB 160KB
[ 1164.606674] kmalloc-4096 1504KB 1504KB
[ 1164.612017] kmalloc-2048 480KB 480KB
[ 1164.617387] kmalloc-1024 304KB 304KB
[ 1164.622729] kmalloc-512 452KB 488KB
[ 1164.628098] kmalloc-256 76KB 76KB
[ 1164.633447] kmalloc-192 86KB 86KB
[ 1164.638818] kmalloc-128 184KB 184KB
[ 1164.644161] kmalloc-64 4569KB 4584KB
[ 1164.649533] kmem_cache_node 8KB 8KB
[ 1164.654883] kmem_cache 24KB 24KB
[ 1164.660252] Tasks state (memory values in pages):
[ 1164.665064] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1164.673889] [ 95] 0 95 502 19 28672 0 0 rngd
[ 1164.682154] [ 127] 0 127 399 43 20480 0 0 heart
[ 1164.690526] Out of memory: Kill process 127 (heart) score 0 or sacrifice child
[ 1164.697939] Killed process 127 (heart) total-vm:1596kB, anon-rss:28kB, file-rss:144kB, shmem-rss:0kB
[ 1164.707331] erlinit: Erlang VM exited
[ 1164.707435] oom_reaper: reaped process 127 (heart), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1164.711202] erlinit: Sending SIGTERM to all processes
[ 1164.720354] watchdog: watchdog0: watchdog did not stop!
[ 1165.745917] erlinit: Sending SIGKILL to all processes
[ 1165.881653] erlinit: umount 0 failed: No such file or directory
[ 1166.387850] reboot: Restarting system
