How to store lots of data in memory?

[tap349]

@cdegroot

Compile the set of rules. Clearly it’s code, not data. Problem solved.

Set of rules are loaded when applications starts - they are stored in YML because they are meant to be edited by ordinary users.

Correct me, please, if I’m wrong - by compilation you mean hard-coding this data somewhere in config file? In this case it wouldn’t work because rules should be stored in user-friendly format (YML is user-friendly enough IMO ).

@cdegroot

Another way - depending on the problem - would be to have a pool of workers that each have a copy of the ruleset as state and something like poolboy to route requests; that would also get rid of the copying.

So a limited number of processes (workers) have their own copies of data while user processes just pass request payload into workers to calculate the result? Well, that sounds reasonable - I’ll need to move all calculations from user processes into workers but that should not be a problem.

The only downside I see here is that I’ll be able to process max <pool_size> requests concurrently - but it must be okay for me too.