gpcweb

gpcweb

More than one supervisor?

I’am new on elixir and I’m building an application that downloads some pdf files from s3. It works just fine using a supervisor and poolboy to manage the process, but I would like to retrieve all the links from a database, I’m seeing ecto but when I used It tells me that I need a supervisor.

This is my current model module

defmodule Pdf.Consumption do
  use Ecto.Schema
  alias Pdf.Consumption
  alias Pdf.Repo

  schema "consumptions" do
    field :invoice_url
  end

  def all do
    Consumption
    |> Repo.all
  end
end 

This is my current application module

defmodule Pdf do
  use Application

  def start(_type, _args) do
    poolboy_config = [
      {:name, {:local, pool_name()}},
      {:worker_module, Pdf.Worker},
      {:size, 2},
      {:max_overflow, 0}
    ]

  children = [:poolboy.child_spec(pool_name(), poolboy_config, [])]

  options = [
   strategy: :one_for_one,
   name: Pdf.Supervisor
  ]

  Supervisor.start_link(children, options)
end

  defp pool_name() do
    :pdf_pool
  end

  defp pdf_pool(link) do
     :poolboy.transaction(
       pool_name(),
       fn(pid) -> :gen_server.call(pid, link) end, :infinity
     )
  end

  def parallel_pool(links) do
    Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
  end
end

And I would like to make this on parallel_pool

def parallel_pool do
  Pdf.Consumption.all
  |>Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
end

So my question is, how can achieve this? I need another module that use Application?

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

So there are really two questions here:

  1. How do you handle multiple supervisors
  2. What’s the best way to download a lot of files concurrently from S3.

I’m gonna answer #2. Poolboy is not the way to do this. The easiest way is to use the Elixir 1.4 Task.async_stream. function:

links
|> Task.async_stream(fn link ->
  process_link(link)
end, max_concurrency: 20)
|> Stream.run

This will download 20 links at a time. It will not return anything, I’m assuming that you’re writing to disk or something with process_link. If you need the results you would just |> Enum.to_list instead of Stream.run

gpcweb

gpcweb

Can you explain me why poolboy is not the way to do this?

NobbZ

NobbZ

Because it adds a dependency while the work can be done with stuff from the std-lib easily.

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

We're in Beta

About us Mission Statement