File.Stream for a file on S3?

sabri

Is it possible to stream a file from a URL ? ex: Amazon S3 ?

I tried to do:

stream= File.stream!(public_s3_file_path)

I got error, although the file is there and public:

** (File.Error) could not stream "filepath...": no such file or directory
    (elixir) lib/file/stream.ex:78: anonymous fn/2 in Enumerable.File.Stream.reduce/3
    (elixir) lib/stream.ex:1240: anonymous fn/5 in Stream.resource/3
    (elixir) lib/stream.ex:785: Stream.do_transform/8
    (elixir) lib/stream.ex:1403: Enumerable.Stream.do_each/4
    (elixir) lib/task/supervised.ex:216: Task.Supervised.stream_reduce/10
    (elixir) lib/stream.ex:570: Stream.run/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: &Stream.run/1
    Args: [#Function<0.55381211/2 in Task.Supervisor.build_stream/5>]

So, how would I stream it?

10 comments

#streams #how-to-question

0 4137 10

2019-03-08 14:04:43 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

sabri

I found that when I tried to stream it, it was not fully uploaded.. sorry, didn’t notice

Post #1

sabri

I found that I was trying to stream after the file is fully uploaded and public, but I get error too:

Here is the minimum code to try:

stream= File.stream!("https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260")
Task.start_link(Stream, :run, [stream])

the error:

16:19:49.569 [error] Task #PID<0.168.0> started from #PID<0.165.0> terminating
** (File.Error) could not stream "https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260": no such file or directory
    (elixir) lib/file/stream.ex:78: anonymous fn/2 in Enumerable.File.Stream.reduce/3
    (elixir) lib/stream.ex:1240: anonymous fn/5 in Stream.resource/3
    (elixir) lib/stream.ex:570: Stream.run/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: &Stream.run/1
    Args: [%File.Stream{line_or_bytes: :line, modes: [:raw, :read_ahead, :binary], path: "https://s3.eu-central-1.amazonaws.com/trackware.staging.schools/school-2/uploads/process%3A15073648797260", raw: true}]

Any idea?

Post #2

idi527

You can use httpoison for this. I think File is for the local filesystem.

Post #3

bkolobara

The File.stream! function takes a path on the local filesystem and you are giving it a url.

To get the file from s3 you will need to use a http client like httpoison

Post #4

sabri

So, I will download it locally using httpoison, then do streaming on it? or httpoison can stream from URL already?

Post #5

sabri

Would File.stream! work on a file at a nother node?

Post #6

wmnnd

Httpoison can do both
Check out the docs and the :stream_to option to learn more:

Post #7

sabri

I am fairly new to Elixir, how would I change the context of stream for my code that is built depending on File.stream??

stream= File.stream!(path)
        |> CSVParser.parse_stream
        |> Stream.chunk(chunck_n, step, [])
        |> Task.Supervisor.async_stream_nolink(CSVSupervisor, &1)

Task.start_link(Stream, :run, [stream])

Post #8