gpcweb

gpcweb

More than one supervisor?

I’am new on elixir and I’m building an application that downloads some pdf files from s3. It works just fine using a supervisor and poolboy to manage the process, but I would like to retrieve all the links from a database, I’m seeing ecto but when I used It tells me that I need a supervisor.

This is my current model module

defmodule Pdf.Consumption do
  use Ecto.Schema
  alias Pdf.Consumption
  alias Pdf.Repo

  schema "consumptions" do
    field :invoice_url
  end

  def all do
    Consumption
    |> Repo.all
  end
end 

This is my current application module

defmodule Pdf do
  use Application

  def start(_type, _args) do
    poolboy_config = [
      {:name, {:local, pool_name()}},
      {:worker_module, Pdf.Worker},
      {:size, 2},
      {:max_overflow, 0}
    ]

  children = [:poolboy.child_spec(pool_name(), poolboy_config, [])]

  options = [
   strategy: :one_for_one,
   name: Pdf.Supervisor
  ]

  Supervisor.start_link(children, options)
end

  defp pool_name() do
    :pdf_pool
  end

  defp pdf_pool(link) do
     :poolboy.transaction(
       pool_name(),
       fn(pid) -> :gen_server.call(pid, link) end, :infinity
     )
  end

  def parallel_pool(links) do
    Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
  end
end

And I would like to make this on parallel_pool

def parallel_pool do
  Pdf.Consumption.all
  |>Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
end

So my question is, how can achieve this? I need another module that use Application?

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

So there are really two questions here:

  1. How do you handle multiple supervisors
  2. What’s the best way to download a lot of files concurrently from S3.

I’m gonna answer #2. Poolboy is not the way to do this. The easiest way is to use the Elixir 1.4 Task.async_stream. function:

links
|> Task.async_stream(fn link ->
  process_link(link)
end, max_concurrency: 20)
|> Stream.run

This will download 20 links at a time. It will not return anything, I’m assuming that you’re writing to disk or something with process_link. If you need the results you would just |> Enum.to_list instead of Stream.run

gpcweb

gpcweb

Can you explain me why poolboy is not the way to do this?

NobbZ

NobbZ

Because it adds a dependency while the work can be done with stuff from the std-lib easily.

Where Next?

Popular in Questions Top

New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130579 1222
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

We're in Beta

About us Mission Statement