awksedgreep1
Syslog server performance issue with appending log
Apologies if this has been asked before, but I couldn’t find a reference to this issue on the forum.
I’m writing a syslog server for a DOCSIS stack I’m working on and would love to have an efficient way to append a log. I’m currently using File.write! with the following options, which seem ideal for what I’m trying to accomplish: [:append, {:delayed_write, 1000, 10_000}] This needs to scale to tens of thousands of devices(cable modems) logging to it so performance is paramount.
It appears that the append does what has been requested, but when the log files get large(>500M) the File.write! seems to slow down drastically. It seems to be reading the entire file to find the end after every delayed write tries to write the file.
Two questions:
Can I append to a file any other way?
Is there something I’m missing in my current implementation, an option perhaps that allows me to hold the end of the log position and append from there?
'njoy,
Mark
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 4 of 4 Posts
awksedgreep1
Decided to read the source code and found this:
Warning: Every time this function is invoked, a file descriptor is opened
and a new process is spawned to write to the file. For this reason, if you are
doing multiple writes in a loop, opening the file via
File.open/2and usingthe functions in
IOto write to the file will yield much better performancethan calling this function multiple times.
That’s on File.write.
Going to refactor with file stream in state and writes via IO module and see if that improves the situation.
axelson
If you haven’t already seen it you may also be interested in “raw” mode. It can greatly improve file performance although of course it comes with some tradeoffs.
awksedgreep1
Performance is better after the refactor. I’ll try raw mode now and see if it improves further. I just timed the new IO.write and it took 4min 50sec to do 10m messages. Vast improvement, but I think it could be better. Thank you for the suggestion.
awksedgreep1
Was able to double performance again and get more consistent results with the following. Going to mark this as a solution since I couldn’t get a combination of options to work with :raw.
{:ok, file} = File.open(@csv_file_path, [:append, :utf8, :delayed_write])
What was taking about 15 minutes(10m writes) is down to 2:45.
'njoy,
Mark