gpcweb

gpcweb

I’am new on elixir and I’m building an application that downloads some pdf files from s3. It works just fine using a supervisor and poolboy to manage the process, but I would like to retrieve all the links from a database, I’m seeing ecto but when I used It tells me that I need a supervisor.

This is my current model module

defmodule Pdf.Consumption do
  use Ecto.Schema
  alias Pdf.Consumption
  alias Pdf.Repo

  schema "consumptions" do
    field :invoice_url
  end

  def all do
    Consumption
    |> Repo.all
  end
end 

This is my current application module

defmodule Pdf do
  use Application

  def start(_type, _args) do
    poolboy_config = [
      {:name, {:local, pool_name()}},
      {:worker_module, Pdf.Worker},
      {:size, 2},
      {:max_overflow, 0}
    ]

  children = [:poolboy.child_spec(pool_name(), poolboy_config, [])]

  options = [
   strategy: :one_for_one,
   name: Pdf.Supervisor
  ]

  Supervisor.start_link(children, options)
end

  defp pool_name() do
    :pdf_pool
  end

  defp pdf_pool(link) do
     :poolboy.transaction(
       pool_name(),
       fn(pid) -> :gen_server.call(pid, link) end, :infinity
     )
  end

  def parallel_pool(links) do
    Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
  end
end

And I would like to make this on parallel_pool

def parallel_pool do
  Pdf.Consumption.all
  |>Enum.each( links, fn(link) -> spawn( fn() -> pdf_pool(link) end ) end )
end

So my question is, how can achieve this? I need another module that use Application?

Showing Posts 1 to 3

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

So there are really two questions here:

  1. How do you handle multiple supervisors
  2. What’s the best way to download a lot of files concurrently from S3.

I’m gonna answer #2. Poolboy is not the way to do this. The easiest way is to use the Elixir 1.4 Task.async_stream. function:

links
|> Task.async_stream(fn link ->
  process_link(link)
end, max_concurrency: 20)
|> Stream.run

This will download 20 links at a time. It will not return anything, I’m assuming that you’re writing to disk or something with process_link. If you need the results you would just |> Enum.to_list instead of Stream.run

gpcweb

gpcweb OP

Can you explain me why poolboy is not the way to do this?

NobbZ

NobbZ

Because it adds a dependency while the work can be done with stuff from the std-lib easily.

— All posts loaded —

Where Next? Top

Trending in Questions Top

katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
achenet
Hello, I’m trying to build a basic Phoenix web-app, and I’d like to use Tailwind. However, when I launch mix phx.server, I get an error...
New
kpanic
Hi everyone, I am toying with the idea of building a “match maker” for giving personal help to people that wants to start coding. I sta...
New
asweet-confluent
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New
Cxx-mlr
I’m working on a small exercise involving update_in/3, and I came up with this solution: data = %{ name: "Periodic Table", category:...
New
ChrisAmelia
I’ve got trouble wrapping my head around the order in which functions are called in this snippet (from Phoenix’s authentication): toke...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews