Being very painfully pedantic (very precise in the use of terminology)… there is a difference between “concurrency” and “parallelism”. “Concurrency” involves the means the establishment of more than one independent process composed together to form a solution. You can create concurrent solutions and execute them on a system with a only a single CPU.

Parallelism refers to the ability to execute two or more concurrent processes simultaneously. You must have more than one processing core to execute two processes in parallel.

Erlang is built for concurrency and will run concurrent solutions (even with a single CPU). Given multiple execution cores, it can also execute processes in parallel.

The number of processes you can make is not limited by the number of CPU cores. The number of processes that can actually run at exactly the same time, however, depends on the number of CPU cores. But it is not something you usually have to think about very deeply: Often, processes have to wait on IO (reading/writing files) or other calls to the operating system or external world. The scheduler knows when this happens, and uses that to switch to another process that currently is not waiting. So we usually run many more processes than CPUs, and end up with a system that is still much faster than having only at most the amount of processes as we have CPU cores.

In your case, many processes might work on the same file, which means they all have to wait on the same thing. Instead, it is more natural to have one process that extracts lines from this file, which passes it on to a pool of workers that work on each of the lines, which might pass it on to other workers if there is more work to be done, until at some point you reach a stage at which you want to combine the results, which probably means that you’re limited to a single process there again as well.

I highly suggest using a more higher-level library like Flow for this. It makes the hard decisions like ‘how many processes for each stage’ and ‘how to connect the stages’ for you (which you can fine-tune if you want, but the defaults are very sane).

I believe that is what @hauleth is saying in his posts- while this work can be done concurrently on a single core it will be slower than the single threaded approach because there is no chance of parallelism. Because of this you’d need to instead increase single threaded performance to complete the task sooner.