Syslog server performance issue with appending log

Apologies if this has been asked before, but I couldn’t find a reference to this issue on the forum.

I’m writing a syslog server for a DOCSIS stack I’m working on and would love to have an efficient way to append a log. I’m currently using File.write! with the following options, which seem ideal for what I’m trying to accomplish: [:append, {:delayed_write, 1000, 10_000}] This needs to scale to tens of thousands of devices(cable modems) logging to it so performance is paramount.

It appears that the append does what has been requested, but when the log files get large(>500M) the File.write! seems to slow down drastically. It seems to be reading the entire file to find the end after every delayed write tries to write the file.

Two questions:
Can I append to a file any other way?

Is there something I’m missing in my current implementation, an option perhaps that allows me to hold the end of the log position and append from there?

'njoy,
Mark

Decided to read the source code and found this:

Warning: Every time this function is invoked, a file descriptor is opened
and a new process is spawned to write to the file. For this reason, if you are
doing multiple writes in a loop, opening the file via File.open/2 and using
the functions in IO to write to the file will yield much better performance
than calling this function multiple times.

That’s on File.write.

Going to refactor with file stream in state and writes via IO module and see if that improves the situation.

1 Like

If you haven’t already seen it you may also be interested in “raw” mode. It can greatly improve file performance although of course it comes with some tradeoffs.

https://www.erlang.org/doc/apps/kernel/file.html#module-performance

3 Likes

Performance is better after the refactor. I’ll try raw mode now and see if it improves further. I just timed the new IO.write and it took 4min 50sec to do 10m messages. Vast improvement, but I think it could be better. Thank you for the suggestion.

1 Like

Was able to double performance again and get more consistent results with the following. Going to mark this as a solution since I couldn’t get a combination of options to work with :raw.

{:ok, file} = File.open(@csv_file_path, [:append, :utf8, :delayed_write])

What was taking about 15 minutes(10m writes) is down to 2:45.

'njoy,
Mark

3 Likes