File.Stream for a file on S3?

Questions / Help
,
1

Is it possible to stream a file from a URL ? ex: Amazon S3 ?

I tried to do:

stream= File.stream!(public_s3_file_path)

I got error, although the file is there and public:

** (File.Error) could not stream "filepath...": no such file or directory
    (elixir) lib/file/stream.ex:78: anonymous fn/2 in Enumerable.File.Stream.reduce/3
    (elixir) lib/stream.ex:1240: anonymous fn/5 in Stream.resource/3
    (elixir) lib/stream.ex:785: Stream.do_transform/8
    (elixir) lib/stream.ex:1403: Enumerable.Stream.do_each/4
    (elixir) lib/task/supervised.ex:216: Task.Supervised.stream_reduce/10
    (elixir) lib/stream.ex:570: Stream.run/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: &Stream.run/1
    Args: [#Function<0.55381211/2 in Task.Supervisor.build_stream/5>]

So, how would I stream it?

2

I found that when I tried to stream it, it was not fully uploaded… sorry, didn’t notice

3

I found that I was trying to stream after the file is fully uploaded and public, but I get error too:

Here is the minimum code to try:

stream= File.stream!("https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260")
Task.start_link(Stream, :run, [stream])

the error:

16:19:49.569 [error] Task #PID<0.168.0> started from #PID<0.165.0> terminating
** (File.Error) could not stream "https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260": no such file or directory
    (elixir) lib/file/stream.ex:78: anonymous fn/2 in Enumerable.File.Stream.reduce/3
    (elixir) lib/stream.ex:1240: anonymous fn/5 in Stream.resource/3
    (elixir) lib/stream.ex:570: Stream.run/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: &Stream.run/1
    Args: [%File.Stream{line_or_bytes: :line, modes: [:raw, :read_ahead, :binary], path: "https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260", raw: true}]

Any idea?

4

You can use httpoison for this. I think File is for the local filesystem.

5

The File.stream! function takes a path on the local filesystem and you are giving it a url.

To get the file from s3 you will need to use a http client like httpoison

6

So, I will download it locally using httpoison, then do streaming on it? or httpoison can stream from URL already?

7

Would File.stream! work on a file at a nother node?

8

Httpoison can do both :wink:
Check out the docs and the :stream_to option to learn more:
https://hexdocs.pm/httpoison/HTTPoison.html#content

9

I am fairly new to Elixir, how would I change the context of stream for my code that is built depending on File.stream??

stream= File.stream!(path)
        |> CSVParser.parse_stream
        |> Stream.chunk(chunck_n, step, [])
        |> Task.Supervisor.async_stream_nolink(CSVSupervisor, &1)

Task.start_link(Stream, :run, [stream])
10

hey with stream_to, is it possible to stream to a new url, or what are the options other than self()?

11

It takes a PID

1 Like