How may we generalize code across OS platforms, that use different `end-of-line`

Questions / Help
1

Hi,
I do my elixir dev on windows hence I always assume \r\n

For example:

item_codes = File.read!("path/to/item-codes.txt") |> String.split("\r\n")

This might not work on other OS that default to \n

Does (or can) Elixir offer any normalization to eliminate the need for a devs to worry about end of line formats on different OS?

2

You could use File.stream!/3 which by default reads the file one line at a time.
This might also be better than File.read! and String.split because it doesn’t require you to load the full file into memory right away.

Check out the docs for File.stream!/3 here:

https://hexdocs.pm/elixir/File.html#stream!/3

3

File stream does assume LF only.

Normalisation of line endings has already happened a long time ago. But windows folks do it their own way…

Most win devs I know do use LF only, because it’s how many tools want to see it. Go does even consider CRLF a parse error…

4

That not quite right.

File.stream!/3 relies on Erlang’s :io.read_line and does in fact normalize line endings. I’ve just tried it with a sample file:

File.read!("test.file")
# => "Line 1\r\nLine 2\r\nLine 3"
File.stream!("test.file") |> Enum.take(3)
# => ["Line 1\n", "Line 2\n", "Line 3"]

For comparison, the same code but with just \n line endings yields the same result with File.stream!:

File.read!("test.file")                  
# => "Line 1\nLine 2\nLine 3"
iex(11)> File.stream!("test.file") |> Enum.take(3)
# => ["Line 1\n", "Line 2\n", "Line 3"]
1 Like
5

Cool, maybe it was an older erts where I had the trouble becAuse of crlf…

6

Well I have experienced problems when using code that was written without consideration for \r\n on windows.

It used macros to expand a text file that was split on \n - took me a while too see why it would not work when compiled on my laptop.

Shouldn’t pure Elixir code just run on any OS without consideration for end-of-line differences?

One would think it should.

7

It sounds like the macro author was not considering eol differences, rather than Elixir itself?

8

One solution is to String.split on ~r/\R/.

That handles both \n and \r\n

2 Likes