gpcweb

gpcweb

More than one supervisor?

I’am new on elixir and I’m building an application that downloads some pdf files from s3. It works just fine using a supervisor and poolboy to manage the process, but I would like to retrieve all the links from a database, I’m seeing ecto but when I used It tells me that I need a supervisor.

This is my current model module

defmodule Pdf.Consumption do
  use Ecto.Schema
  alias Pdf.Consumption
  alias Pdf.Repo

  schema "consumptions" do
    field :invoice_url
  end

  def all do
    Consumption
    |> Repo.all
  end
end 

This is my current application module

defmodule Pdf do
  use Application

  def start(_type, _args) do
    poolboy_config = [
      {:name, {:local, pool_name()}},
      {:worker_module, Pdf.Worker},
      {:size, 2},
      {:max_overflow, 0}
    ]

  children = [:poolboy.child_spec(pool_name(), poolboy_config, [])]

  options = [
   strategy: :one_for_one,
   name: Pdf.Supervisor
  ]

  Supervisor.start_link(children, options)
end

  defp pool_name() do
    :pdf_pool
  end

  defp pdf_pool(link) do
     :poolboy.transaction(
       pool_name(),
       fn(pid) -> :gen_server.call(pid, link) end, :infinity
     )
  end

  def parallel_pool(links) do
    Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
  end
end

And I would like to make this on parallel_pool

def parallel_pool do
  Pdf.Consumption.all
  |>Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
end

So my question is, how can achieve this? I need another module that use Application?

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

So there are really two questions here:

  1. How do you handle multiple supervisors
  2. What’s the best way to download a lot of files concurrently from S3.

I’m gonna answer #2. Poolboy is not the way to do this. The easiest way is to use the Elixir 1.4 Task.async_stream. function:

links
|> Task.async_stream(fn link ->
  process_link(link)
end, max_concurrency: 20)
|> Stream.run

This will download 20 links at a time. It will not return anything, I’m assuming that you’re writing to disk or something with process_link. If you need the results you would just |> Enum.to_list instead of Stream.run

gpcweb

gpcweb

Can you explain me why poolboy is not the way to do this?

NobbZ

NobbZ

Because it adds a dependency while the work can be done with stuff from the std-lib easily.

Where Next?

Popular in Questions Top

qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vegabook
I'm brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29305 241
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
vegabook
I'm brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement