@OvermindDL1 If you follow this approach, you definitely need to use backpressure. However, I don’t think an extra process is needed here; we already have one (besides the current process), namely the process owning the file.

An alternative approach:

• Pick a chunk size that is larger than the largest piece you need to match on.
• Keep track of a ‘buffer’ bytestring.
• You can pattern match on this buffer’s front, and chop the parts of that are consumed.
• Whenever this buffer becomes shorter than the chunk size, you append the next file chunk to it, and in this way consume the file lazily.

@Qqwy, I like this approach. Once I’m smart enough to implement it this is what I’m going to do. Right now I’m just using File.read and I’m having to skip the files that are over 2gb as it won’t even attempt to open them.

I’d use Stream.transform/3/4 to consume the file and implement the reducer as some kind of automaton which consumes the file and emits a stream of tokens, which is then transformed into its final form.

Constructing the automaton can get pretty tedious when it out grows a certain size…