Hi @tomazbracic ,
author of CubDB here while myself and others have used CubDB for many data logging tasks, your case may deserve special care. Most cases I am familiar with were either a slower continuous rate, or a higher one but in bursts, or a lower retention. In principle, I don’t see any hard limit, the write speed is not concerning, but due to the long retention and the continuous write, in practice, there are a couple of things to be mindful of.

The main such thing, as you correctly note, is compaction. Compaction progresses concurrently to read and writes, but can take a while for large files, and it will need to catch up with writes that happened while the compaction was ongoing. Since your data write is a continuous 50 writes per second, with no pause, there can be a lot to catch up with. Therefore, while it’s possible that everything would work just fine, compaction is one area where I would recommend doing some preliminary test. Since compaction only reclaims disk space, and does not affect read or write performance noticeably, if you have enough disk space you could disable auto compaction and perform it programmatically at specific chosen moments of downtime (if there is any).

Do you need to query old data live, or are you simply logging it? In the latter case, you could even disable auto compaction completely and start a new database in a new data directory each day, removing old databases and their data directories after the retention period.

In order to help you decide if CubDB is the right choice for you, I have a few other questions:

• How important is durability of data or atomicity of transactions in presence of sudden unexpected shutdowns? This is one of the situations that CubDB is designed to handle.
• Is it helpful to use native Elixir data structures in your data, or they could just be serialized strings? In the first case CubDB can help a lot, in the second you have more freedom on what to use to store data (SQLite for example).
• I assume the write pattern is that of a data logger (mostly only appends). What about your read pattern? Do you need to query your data live?
• Do you need the same granularity for all the retention period, or is it possible to summarize or coarse grain data older than, say, a few days?

If you really only append new logs, and basically never need to query such logs live, I think that CubDB might not be needed. The best features of CubDB are atomic transactions, extreme robustness in face of unexpected shutdowns and power loss, possibility to use native Elixir data structures, and ease of performing both point queries and selection. In your case, as far as I can see, none of that is really needed. You do need durability, but given the append-only write pattern, you can achieve it in other ways.

In light of what you wrote, probably the easiest option is to simply write plain flat log files, one per day, using a naming scheme such as year-month-day.log, and perform log rotation by removing the old ones after the retention period. If your data is such that there is a simple way to separate each entry (say, one per line), and you really never need to query that data, this would work and be simple to maintain and understand. If you need durability, you will have to make sure to fsync each write (slow, but safe), or if it’s acceptable to loose, say, the last few seconds of data, fsync every N seconds or every N writes (more performant). Using a human readable format for your data would have the added benefit of making your logs readable just by opening them in a text editor, or with command line tools like awk (although you would probably need to load the logs into some system for deeper analysis).

If you instead need to query this data live, you could follow a hybrid approach, where you log raw un-aggregated data in plain log files like explained above, but also store aggregated data into a database for live querying. In a few project, for example, my system was allowing live querying such as “what’s the average value of X between last Tuesday 3pm and Saturday 4pm”, which would be served by CubDB but could use data aggregated by minute, and also allowed operators to access the device and download un-aggregated raw log files for offline investigation.

If you do need to query the data live, at arbitrary granularity, but typically only for the current day, you could do more or less the same as the first suggestion, but using CubDB instead of plain log files. You could restart CubDB on a different data directory at the beginning of each day. You can at that point disable compaction completely, which also ensures that the DB file never changes name, and remove old data directories after the retention period, like you would do with plain files. This way, you gain the possibility to query your data live, but on the other hand your logs cannot be read in a plain text editor. Alternatively, you can use SQLite, which is a great option, but would also very likely need some tweaking of the configuration to support the write pattern and performance that you need.

I hope this helps, happy to answer further questions if needed

If you decide to use plain log files, note that the standard Elixir Logger with its default logger_std_h handler already supports log rotation, as well as delayed_write and filesync_repeat_interval options for tweaking durability and write performance (documented in the file module). Unfortunately, as far as I know, in Elixir and Erlang it is not generally possible to configure a special purpose logger distinct from the global logger.

It is possible to use filters to direct specific logs to a different handler based on level or metadata, but I find it obscures the intent a bit, when what you really want is to treat your data logs differently from all the rest.

One possibility then is to bypass logger and handlers completely and use directly the Erlang disk_log, which offers log rotation and other useful options. One limitation is that it rotates the files by size, not by date, which makes it more difficult to implement precisely the 30 days retention you want.