solanav
Hello everyone!
I’m currently working on a web spider/metadata extractor and because writing parsers for many file types would be too much work, I’m using libextractor.
I wrote a NIF which I learned only allows for functions that return in less than ~1ms. So I just changed this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract }
};
into this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract, ERL_NIF_DIRTY_JOB_CPU_BOUND }
};
The problem: it seems like the scheduler gets to 100% utilization which is bad because the spider downloads a lot of files in parallel so it becomes impossible to do things like launch the observer if I’m calling the NIF. Any idea if this is normal? Is there a way I can prevent the NIF from blocking the rest of the system?
Here is a picture of the observer while I test the calls to my main function:
Thanks!
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
mpope
Is it possible to run
erlang:system_infoto checkschedulers_state,scheduling_statistics,dirty_cpu_schedulers_online,dirty_cpu_schedulersAnd the most useful would be to check schedulers using microstate accounting?
msacc:start(1000), msacc:print(), msacc:stop().Might not be wise to use it in a production env…ityonemo
Can you set the number of dirty nif schedulers to number of cores - 1? Probably easiest way to test this is to do the following at startup:
NobbZ
Are you sure that your NIF is CPU bound and not IO bound?
solanav
I don’t know how useful this is, I’m just running iex -S mix on my NIF, it’s not running in production or anything, but I hope it helps.
Here I did :msacc.start(1000), ran the NIF, printed and then stopped.
solanav
So I tried doing that and it seems like the result is the same, probably because the number of dirty schedulers was 12 so I guess not much changed.
solanav
Well I am not. Not really. I know libextractor (the library I am wrapping with this NIF) does not wait for user input and does not connect to the internet. But I think it may be running some code stored on dynamic libraries.
But I have tried changing it to IO scheduler and even though the peak was of less duration, it still peaked at 80-100%. The graph was almost the same but no dotted line, just a normal one.
nonblockio
It seems that the performance surge only least for one or two seconds from the chart.
:msacc.start(1000)is kinda too short to catch the vibration since it only analyze scheduler in 1000ms. Could you increase the time of:msacc.start/1and try to fit the surge in that period?nonblockio
And could you provide the OTP version?
solanav
I’m using version 23.1 (tried both on WSL and on a ubuntu 20.04 virtual machine with the same results).
solanav
I can’t get it to work, the extractor won’t start running until :msacc.start() has finished. Is this code alright?