Ultimately I’d look at logger_file_backend.ex only as a starting point. Your needs may require a different approach to logging that may be served better by your own custom backend.
Posts tagged with logging.
2 Likes
I am planning to write my own logger.
Then you might be interesting in reading through lpgauth/fast_disk_log.
Could you run observer on your system and show us the load charts? That would be a good starting point to see how your application is using the erlang schedulers which would indicate how much parallelism you actually have in your application. You start it with :observer.start().
4 Likes
I guess I found the reason.
It works fine with the file_server. When it tried to copy a file to another directory and both source and target directories are sitting in different mount, BEAM CPU usage goes way high.
When the copy is done over same mount disk, it was OK.
I need to find the reason for it.
ikim.
Now it’s entirely possible that I’ve gone off track somewhere, so take this with a grain of salt but …
While it seems that file_server is in fact honouring any additional modes it seems to me that specifying raw simply removes the overhead of the file processes but the file_server process is still doing the copying on it’s own time. This seems to be the standard behaviour when you pass filenames to file.copy/3 (i.e. both files still need to be opened).
If however at least one file is already open (i.e. a file descriptor is supplied) then the copy is never delegated to file_server.
So in order to distribute the CPU cost of copying multiple files concurrently it may actually be necessary to spawn a separate process for each file copy. That process would then open at least one file in raw mode (which doesn’t seem to call out of the file module either) while the other one should at least have raw mode specified in the {filename, [modes]} tuple. That way the copying work stays off of the file_server process.
The way Elixir File.copy/3 is implemented it should be possible to use something like:
File.copy(opened_raw_file, {"file.txt", [:raw]})
Of course the alternative is to just use :file.copy/3 directly.
2 Likes
You are absolutely right. As you mentioned, setting as :raw mode is not enough to be free from file_server.
I used code as below, but it was not distributed.
:file.copy({src_file, [:raw, :read_ahead, :binary]}, {tgt_file, [:delayed_write, :binary, :raw]})
Then I changed as you mentioned. Opening source file as :raw mode first.
{:ok, src_io} = :file.open(src_file, [:raw, :read_ahead, :binary])
And I used the source with :file:copy.
{:ok, bytes} = :file.copy( src_io, {tgt_file, [:delayed_write, :binary, :raw]})
Then the tasks are distributed well over multi-cores.
Thank you,
IKIM.
5 Likes
