solanav
Hello everyone!
I’m currently working on a web spider/metadata extractor and because writing parsers for many file types would be too much work, I’m using libextractor.
I wrote a NIF which I learned only allows for functions that return in less than ~1ms. So I just changed this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract }
};
into this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract, ERL_NIF_DIRTY_JOB_CPU_BOUND }
};
The problem: it seems like the scheduler gets to 100% utilization which is bad because the spider downloads a lot of files in parallel so it becomes impossible to do things like launch the observer if I’m calling the NIF. Any idea if this is normal? Is there a way I can prevent the NIF from blocking the rest of the system?
Here is a picture of the observer while I test the calls to my main function:
Thanks!
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
mpope
Is it possible to run
erlang:system_infoto checkschedulers_state,scheduling_statistics,dirty_cpu_schedulers_online,dirty_cpu_schedulersAnd the most useful would be to check schedulers using microstate accounting?
msacc:start(1000), msacc:print(), msacc:stop().Might not be wise to use it in a production env…ityonemo
Can you set the number of dirty nif schedulers to number of cores - 1? Probably easiest way to test this is to do the following at startup:
NobbZ
Are you sure that your NIF is CPU bound and not IO bound?
solanav
I don’t know how useful this is, I’m just running iex -S mix on my NIF, it’s not running in production or anything, but I hope it helps.
Here I did :msacc.start(1000), ran the NIF, printed and then stopped.
solanav
So I tried doing that and it seems like the result is the same, probably because the number of dirty schedulers was 12 so I guess not much changed.
solanav
Well I am not. Not really. I know libextractor (the library I am wrapping with this NIF) does not wait for user input and does not connect to the internet. But I think it may be running some code stored on dynamic libraries.
But I have tried changing it to IO scheduler and even though the peak was of less duration, it still peaked at 80-100%. The graph was almost the same but no dotted line, just a normal one.
nonblockio
It seems that the performance surge only least for one or two seconds from the chart.
:msacc.start(1000)is kinda too short to catch the vibration since it only analyze scheduler in 1000ms. Could you increase the time of:msacc.start/1and try to fit the surge in that period?nonblockio
And could you provide the OTP version?
solanav
I’m using version 23.1 (tried both on WSL and on a ubuntu 20.04 virtual machine with the same results).
solanav
I can’t get it to work, the extractor won’t start running until :msacc.start() has finished. Is this code alright?