joserp93
Out of memory , kernel cannot allocate memory
Hello everyone. I am trying to use Task.async stream to upload files by S3.put_object. Like in this example:
upload_file = fn {src_path, dest_path} ->
S3.put_object("my_bucket", dest_path, File.read!(src_path))
|> ExAws.request!
end
paths = %{"path/to/src0" => "path/to/dest0", "path/to/src1" => "path/to/dest1"}
paths
|> Task.async_stream(upload_file, max_concurrency: 10)
|> Stream.run
But I have a problem. If there are many files, the kernel starts to run out of memory and the rpi restarts, I have the log. If anyone has an idea of how to increase the memory or was able to solve this, it would be very helpful.
[ 1163.448679] 2_scheduler invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
[ 1163.459534] 2_scheduler cpuset=/ mems_allowed=0
[ 1163.464301] CPU: 2 PID: 104 Comm: 2_scheduler Tainted: G C 4.19.118 #2
[ 1163.472332] Hardware name: BCM2835
[ 1163.475831] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1163.483768] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1163.491173] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1163.498719] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1163.506828] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1163.515201] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1163.524195] [<c02ef534>] (__alloc_pages_nodemask) from [<c0682760>] (skb_page_frag_refill+0xe4/0xfc)
[ 1163.533507] [<c0682760>] (skb_page_frag_refill) from [<c06841b8>] (sk_page_frag_refill+0x14/0x6c)
[ 1163.542590] [<c06841b8>] (sk_page_frag_refill) from [<c06f022c>] (tcp_sendmsg_locked+0x43c/0xc18)
[ 1163.551670] [<c06f022c>] (tcp_sendmsg_locked) from [<c06f0a30>] (tcp_sendmsg+0x28/0x3c)
[ 1163.559873] [<c06f0a30>] (tcp_sendmsg) from [<c06800c4>] (sock_sendmsg+0x14/0x24)
[ 1163.567510] [<c06800c4>] (sock_sendmsg) from [<c068016c>] (sock_write_iter+0x98/0xf4)
[ 1163.575534] [<c068016c>] (sock_write_iter) from [<c0350250>] (do_iter_readv_writev+0x118/0x194)
[ 1163.584437] [<c0350250>] (do_iter_readv_writev) from [<c0350be4>] (do_iter_write+0x84/0x190)
[ 1163.593075] [<c0350be4>] (do_iter_write) from [<c0350d68>] (vfs_writev+0x78/0xe4)
[ 1163.600708] [<c0350d68>] (vfs_writev) from [<c0350e3c>] (do_writev+0x68/0xf0)
[ 1163.608023] [<c0350e3c>] (do_writev) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1163.615741] Exception stack(0xc1851fa8 to 0xc1851ff0)
[ 1163.620903] 1fa0: 00000010 b6bc00e0 00000031 b6bc00e0 00000010 00000000
[ 1163.629274] 1fc0: 00000010 b6bc00e0 b403c920 00000092 0005ad95 b6bc00d0 a9b42a0c 00082903
[ 1163.637610] 1fe0: 00000000 b403b9b0 b403c920 b6da3218
[ 1163.643240] Mem-Info:
[ 1163.645594] active_anon:19493 inactive_anon:2 isolated_anon:0
[ 1163.645594] active_file:1477 inactive_file:9296 isolated_file:0
[ 1163.645594] unevictable:0 dirty:0 writeback:5 unstable:0
[ 1163.645594] slab_reclaimable:1130 slab_unreclaimable:3224
[ 1163.645594] mapped:2121 shmem:3 pagetables:200 bounce:0
[ 1163.645594] free:755040 free_pcp:143 free_cma:15788
[ 1163.679473] Node 0 active_anon:79156kB inactive_anon:8kB active_file:2800kB inactive_file:34808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8484kB dirty:0kB writeback:20kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1163.702629] DMA free:64628kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:112kB inactive_file:20kB unevictable:0kB writepending:20kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:1000kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1163.729648] lowmem_reserve[]: 0 0 3008 3008
[ 1163.734061] HighMem free:2958876kB min:512kB low:19644kB high:38776kB active_anon:79040kB inactive_anon:8kB active_file:2856kB inactive_file:34612kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:800kB bounce:0kB free_pcp:1556kB local_pcp:0kB free_cma:0kB
[ 1163.762704] lowmem_reserve[]: 0 0 0 0
[ 1163.766577] DMA: 87*4kB (UEHC) 55*8kB (UEHC) 15*16kB (UEHC) 21*32kB (UEHC) 8*64kB (UEC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 65316kB
[ 1163.782491] HighMem: 126*4kB (UM) 120*8kB (UM) 151*16kB (UM) 52*32kB (UM) 48*64kB (UM) 22*128kB (UM) 21*256kB (UM) 19*512kB (UM) 12*1024kB (UM) 10*2048kB (UM) 708*4096kB (UM) = 2959272kB
[ 1163.799477] 2754 total pagecache pages
[ 1163.799480] 0 pages in swap cache
[ 1163.799483] Swap cache stats: add 0, delete 0, find 0/0
[ 1163.799485] Free swap = 0kB
[ 1163.799487] Total swap = 0kB
[ 1163.799490] 802816 pages RAM
[ 1163.799492] 770048 pages HighMem/MovableOnly
[ 1163.799494] 9473 pages reserved
[ 1163.799496] 16384 pages cma reserved
[ 1163.799499] Tasks state (memory values in pages):
[ 1163.799502] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1163.799516] [ 88] 0 88 418 48 28672 0 0 nbtty
[ 1163.854315] [ 95] 0 95 502 19 28672 0 0 rngd
[ 1163.854321] [ 96] 0 96 483 274 32768 0 0 nbtty
[ 1163.870991] [ 97] 0 97 66114 27965 585728 0 0 beam.smp
[ 1163.870996] [ 102] 0 102 433 255 24576 0 0 erl_child_setup
[ 1163.871001] [ 127] 0 127 399 43 20480 0 0 heart
[ 1163.871005] [ 131] 0 131 454 244 32768 0 0 uevent
[ 1163.871009] [ 134] 0 134 398 41 20480 0 0 kmsg_tailer
[ 1163.871012] [ 140] 0 140 455 51 24576 0 0 if_monitor
[ 1163.871016] [ 153] 0 153 400 45 24576 0 0 muontrap
[ 1163.871020] [ 154] 0 154 577 374 28672 0 0 ntpd
[ 1163.871025] [ 176] 0 176 400 45 24576 0 0 muontrap
[ 1163.871029] [ 177] 0 177 489 58 28672 0 0 udhcpc
[ 1163.871036] Out of memory: Kill process 97 (beam.smp) score 34 or sacrifice child
[ 1163.871049] Killed process 102 (erl_child_setup) total-vm:1732kB, anon-rss:56kB, file-rss:964kB, shmem-rss:0kB
[ 1163.871251] oom_reaper: reaped process 102 (erl_child_setup), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1163.892263] heart: Erlang is crashing .. (waiting for crash dump file)
[ 1163.999468] init invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=2, oom_score_adj=0
[ 1164.017310] init cpuset=/ mems_allowed=0
[ 1164.021332] CPU: 3 PID: 1 Comm: init Tainted: G C 4.19.118 #2
[ 1164.028555] Hardware name: BCM2835
[ 1164.032055] [<c021042c>] (unwind_backtrace) from [<c020bca0>] (show_stack+0x10/0x14)
[ 1164.039989] [<c020bca0>] (show_stack) from [<c077fc98>] (dump_stack+0x94/0xb4)
[ 1164.047394] [<c077fc98>] (dump_stack) from [<c02ea174>] (dump_header+0x70/0x24c)
[ 1164.054938] [<c02ea174>] (dump_header) from [<c02e92b0>] (oom_kill_process+0xb8/0x350)
[ 1164.063044] [<c02e92b0>] (oom_kill_process) from [<c02e9e20>] (out_of_memory+0x11c/0x378)
[ 1164.071415] [<c02e9e20>] (out_of_memory) from [<c02ef534>] (__alloc_pages_nodemask+0xbe8/0x11c0)
[ 1164.080370] [<c02ef534>] (__alloc_pages_nodemask) from [<c0311380>] (kmalloc_order+0x20/0x30)
[ 1164.089094] [<c0311380>] (kmalloc_order) from [<c03113ac>] (kmalloc_order_trace+0x1c/0xf4)
[ 1164.097555] [<c03113ac>] (kmalloc_order_trace) from [<c026b8a4>] (devkmsg_open+0x50/0xf0)
[ 1164.105928] [<c026b8a4>] (devkmsg_open) from [<c03567d8>] (chrdev_open+0xb0/0x19c)
[ 1164.113651] [<c03567d8>] (chrdev_open) from [<c034e4c4>] (do_dentry_open+0x114/0x3bc)
[ 1164.121673] [<c034e4c4>] (do_dentry_open) from [<c035fe9c>] (path_openat+0x2bc/0x109c)
[ 1164.129780] [<c035fe9c>] (path_openat) from [<c0360cf0>] (do_filp_open+0x74/0xe8)
[ 1164.137411] [<c0360cf0>] (do_filp_open) from [<c034fd24>] (do_sys_open+0x17c/0x200)
[ 1164.145254] [<c034fd24>] (do_sys_open) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[ 1164.153181] Exception stack(0xc1c33fa8 to 0xc1c33ff0)
[ 1164.158341] 3fa0: 0000001b b6f20d10 ffffff9c 00016118 000a0001 00000000
[ 1164.166710] 3fc0: 0000001b b6f20d10 0000001b 00000142 befddd9c befddd98 00015d24 00015d3c
[ 1164.175078] 3fe0: 0002707c befdd660 000132c8 b6e71248
[ 1164.180301] Mem-Info:
[ 1164.182818] active_anon:75 inactive_anon:2 isolated_anon:0
[ 1164.182818] active_file:396 inactive_file:183 isolated_file:0
[ 1164.182818] unevictable:0 dirty:1 writeback:0 unstable:0
[ 1164.182818] slab_reclaimable:1131 slab_unreclaimable:3144
[ 1164.182818] mapped:372 shmem:3 pagetables:29 bounce:0
[ 1164.182818] free:785910 free_pcp:4 free_cma:15788
[ 1164.215521] Node 0 active_anon:300kB inactive_anon:8kB active_file:1584kB inactive_file:732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1488kB dirty:4kB writeback:0kB shmem:12kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1164.238133] DMA free:68076kB min:636kB low:792kB high:948kB active_anon:0kB inactive_anon:0kB active_file:72kB inactive_file:40kB unevictable:0kB writepending:4kB present:131072kB managed:93180kB mlocked:0kB kernel_stack:728kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:63152kB
[ 1164.264834] lowmem_reserve[]: 0 0 3008 3008
[ 1164.269122] HighMem free:3075648kB min:512kB low:19644kB high:38776kB active_anon:300kB inactive_anon:8kB active_file:1664kB inactive_file:1120kB unevictable:0kB writepending:0kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:116kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB
[ 1164.297194] lowmem_reserve[]: 0 0 0 0
[ 1164.300981] DMA: 140*4kB (UEHC) 154*8kB (UEHC) 68*16kB (UEHC) 34*32kB (UEHC) 17*64kB (UEHC) 5*128kB (UC) 2*256kB (C) 3*512kB (C) 1*1024kB (C) 1*2048kB (C) 14*4096kB (C) = 68160kB
[ 1164.317231] HighMem: 188*4kB (UM) 145*8kB (UM) 129*16kB (UM) 121*32kB (UM) 111*64kB (UM) 88*128kB (UM) 72*256kB (UM) 48*512kB (M) 26*1024kB (M) 11*2048kB (M) 722*4096kB (UM) = 3075688kB
[ 1164.334124] 584 total pagecache pages
[ 1164.337873] 0 pages in swap cache
[ 1164.341271] Swap cache stats: add 0, delete 0, find 0/0
[ 1164.346646] Free swap = 0kB
[ 1164.349601] Total swap = 0kB
[ 1164.352562] 802816 pages RAM
[ 1164.355517] 770048 pages HighMem/MovableOnly
[ 1164.359917] 9473 pages reserved
[ 1164.363179] 16384 pages cma reserved
[ 1164.366839] Unreclaimable slab info:
[ 1164.370532] Name Used Total
[ 1164.376062] RAWv6 15KB 15KB
[ 1164.381431] UDPv6 15KB 15KB
[ 1164.386774] TCPv6 31KB 31KB
[ 1164.392151] mqueue_inode_cache 15KB 15KB
[ 1164.397578] UNIX 63KB 63KB
[ 1164.402926] RAW 31KB 31KB
[ 1164.408296] UDP 61KB 61KB
[ 1164.413638] tw_sock_TCP 3KB 3KB
[ 1164.419009] TCP 121KB 121KB
[ 1164.424359] eventpoll_pwq 15KB 15KB
[ 1164.429731] request_queue 62KB 62KB
[ 1164.435073] blkdev_ioc 16KB 16KB
[ 1164.440442] biovec-max 300KB 300KB
[ 1164.445791] biovec-128 126KB 126KB
[ 1164.451162] biovec-64 63KB 63KB
[ 1164.456503] biovec-16 165KB 165KB
[ 1164.461879] skbuff_head_cache 283KB 283KB
[ 1164.467215] configfs_dir_cache 3KB 3KB
[ 1164.472652] file_lock_cache 15KB 15KB
[ 1164.478024] shmem_inode_cache 108KB 108KB
[ 1164.483367] proc_dir_entry 56KB 56KB
[ 1164.488737] pde_opener 15KB 15KB
[ 1164.494088] seq_file 15KB 15KB
[ 1164.499458] sigqueue 15KB 15KB
[ 1164.504800] kernfs_node_cache 1346KB 1346KB
[ 1164.510170] filp 263KB 263KB
[ 1164.515520] names_cache 128KB 128KB
[ 1164.520888] key_jar 16KB 16KB
[ 1164.526233] vm_area_struct 154KB 154KB
[ 1164.531609] mm_struct 32KB 32KB
[ 1164.536945] files_cache 16KB 16KB
[ 1164.542294] signal_cache 109KB 109KB
[ 1164.547664] sighand_cache 220KB 220KB
[ 1164.553006] task_struct 284KB 284KB
[ 1164.558377] cred_jar 36KB 36KB
[ 1164.563725] anon_vma_chain 28KB 28KB
[ 1164.569095] anon_vma 31KB 31KB
[ 1164.574436] pid 16KB 16KB
[ 1164.579804] trace_event_file 35KB 35KB
[ 1164.585152] ftrace_event_field 84KB 84KB
[ 1164.590609] pool_workqueue 16KB 16KB
[ 1164.595952] task_group 83KB 121KB
[ 1164.601326] kmalloc-8192 160KB 160KB
[ 1164.606674] kmalloc-4096 1504KB 1504KB
[ 1164.612017] kmalloc-2048 480KB 480KB
[ 1164.617387] kmalloc-1024 304KB 304KB
[ 1164.622729] kmalloc-512 452KB 488KB
[ 1164.628098] kmalloc-256 76KB 76KB
[ 1164.633447] kmalloc-192 86KB 86KB
[ 1164.638818] kmalloc-128 184KB 184KB
[ 1164.644161] kmalloc-64 4569KB 4584KB
[ 1164.649533] kmem_cache_node 8KB 8KB
[ 1164.654883] kmem_cache 24KB 24KB
[ 1164.660252] Tasks state (memory values in pages):
[ 1164.665064] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1164.673889] [ 95] 0 95 502 19 28672 0 0 rngd
[ 1164.682154] [ 127] 0 127 399 43 20480 0 0 heart
[ 1164.690526] Out of memory: Kill process 127 (heart) score 0 or sacrifice child
[ 1164.697939] Killed process 127 (heart) total-vm:1596kB, anon-rss:28kB, file-rss:144kB, shmem-rss:0kB
[ 1164.707331] erlinit: Erlang VM exited
[ 1164.707435] oom_reaper: reaped process 127 (heart), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1164.711202] erlinit: Sending SIGTERM to all processes
[ 1164.720354] watchdog: watchdog0: watchdog did not stop!
[ 1165.745917] erlinit: Sending SIGKILL to all processes
[ 1165.881653] erlinit: umount 0 failed: No such file or directory
[ 1166.387850] reboot: Restarting system
Most Liked
LostKobrakai
File.read!/1 for sure isn’t.
1
joserp93
ok, It would be using this example:
"path/to/big/file"
|> S3.Upload.stream_file
|> S3.upload("my-bucket", "path/on/s3")
|> ExAws.request! #=> :done
1
Popular in Questions
Hi All,
I set a environment variables in dev.exs , like below code.
when i start server, how can i set the ${enable} value?
thanks.
d...
New
can someone please explain to me how Enum.reduce works with maps
New
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
I will often find my self writing things similar to:
case some_value do
nil -> something()
"" -> something()
_ -> somethi...
New
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
I have VueJS GUIs with the project generated using Webpack.
I have Elixir modules that will need to be used by the VueJS GUIs.
I forese...
New
Hi all,
Trying to get some more clarity over utc_datetime and naive_datetime for Ecto:
The documentation above suggests that while ...
New
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
Other popular topics
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
Hello, how can I check the Phoenix version ?
Thanks !
New
Hi all,
I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage.
I’m trying to use Postgres...
New
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Hello guys,
I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Hi folks,
Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
New
Hi everyone,
I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
Latest Nerves Threads
Latest on Elixir Forum
Sponsor Spotlight
Real-time error tracking, performance insights, and observability for devs.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"