awksedgreep1
Apologies if this has been asked before, but I couldn’t find a reference to this issue on the forum.
I’m writing a syslog server for a DOCSIS stack I’m working on and would love to have an efficient way to append a log. I’m currently using File.write! with the following options, which seem ideal for what I’m trying to accomplish: [:append, {:delayed_write, 1000, 10_000}] This needs to scale to tens of thousands of devices(cable modems) logging to it so performance is paramount.
It appears that the append does what has been requested, but when the log files get large(>500M) the File.write! seems to slow down drastically. It seems to be reading the entire file to find the end after every delayed write tries to write the file.
Two questions:
Can I append to a file any other way?
Is there something I’m missing in my current implementation, an option perhaps that allows me to hold the end of the log position and append from there?
'njoy,
Mark
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #elixirconf-us
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync. Next: ElixirConf EU 2027
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
awksedgreep1
Decided to read the source code and found this:
Warning: Every time this function is invoked, a file descriptor is opened
and a new process is spawned to write to the file. For this reason, if you are
doing multiple writes in a loop, opening the file via
File.open/2and usingthe functions in
IOto write to the file will yield much better performancethan calling this function multiple times.
That’s on File.write.
Going to refactor with file stream in state and writes via IO module and see if that improves the situation.
axelson
If you haven’t already seen it you may also be interested in “raw” mode. It can greatly improve file performance although of course it comes with some tradeoffs.
awksedgreep1
Performance is better after the refactor. I’ll try raw mode now and see if it improves further. I just timed the new IO.write and it took 4min 50sec to do 10m messages. Vast improvement, but I think it could be better. Thank you for the suggestion.
awksedgreep1
Was able to double performance again and get more consistent results with the following. Going to mark this as a solution since I couldn’t get a combination of options to work with :raw.
{:ok, file} = File.open(@csv_file_path, [:append, :utf8, :delayed_write])
What was taking about 15 minutes(10m writes) is down to 2:45.
'njoy,
Mark