awksedgreep1
Syslog server performance issue with appending log
Apologies if this has been asked before, but I couldn’t find a reference to this issue on the forum.
I’m writing a syslog server for a DOCSIS stack I’m working on and would love to have an efficient way to append a log. I’m currently using File.write! with the following options, which seem ideal for what I’m trying to accomplish: [:append, {:delayed_write, 1000, 10_000}] This needs to scale to tens of thousands of devices(cable modems) logging to it so performance is paramount.
It appears that the append does what has been requested, but when the log files get large(>500M) the File.write! seems to slow down drastically. It seems to be reading the entire file to find the end after every delayed write tries to write the file.
Two questions:
Can I append to a file any other way?
Is there something I’m missing in my current implementation, an option perhaps that allows me to hold the end of the log position and append from there?
'njoy,
Mark
Marked As Solved
awksedgreep1
Was able to double performance again and get more consistent results with the following. Going to mark this as a solution since I couldn’t get a combination of options to work with :raw.
{:ok, file} = File.open(@csv_file_path, [:append, :utf8, :delayed_write])
What was taking about 15 minutes(10m writes) is down to 2:45.
'njoy,
Mark
Also Liked
axelson
If you haven’t already seen it you may also be interested in “raw” mode. It can greatly improve file performance although of course it comes with some tradeoffs.
awksedgreep1
Decided to read the source code and found this:
Warning: Every time this function is invoked, a file descriptor is opened
and a new process is spawned to write to the file. For this reason, if you are
doing multiple writes in a loop, opening the file via File.open/2 and using
the functions in IO to write to the file will yield much better performance
than calling this function multiple times.
That’s on File.write.
Going to refactor with file stream in state and writes via IO module and see if that improves the situation.
awksedgreep1
Performance is better after the refactor. I’ll try raw mode now and see if it improves further. I just timed the new IO.write and it took 4min 50sec to do 10m messages. Vast improvement, but I think it could be better. Thank you for the suggestion.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"