Questions & Help>Questions

Fetch data from multiple pages

#api#json#json-api
cipher

cipher

Fetch data from multiple pages

how do I make my code get the number of pages available in advance, and then start several tasks to get several pages concurrently?

Example: example.com/api?id=1; example.com/api?id=2; …
and get the response from the server, until the page returns an empty array: “”

7 comments
#api#json#json-api
5 1434 7

Most Liked

kokolegorille

kokolegorille

There is this old post, I remember I liked it, but maybe now it’s a bit old…

Maybe You can get some inspiration. It uses poolboy and HTTPoison.

Because the problem is not really to be concurrent, but to manage this concurrency.

You don’t want to flood the host with 10’000 concurrent requests.

3
Post #8
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey cipher welcome! Any good programming task starts by breaking down a problem into simpler parts. The first part is: How to fetch a single page? What have you tried so far?

1
Post #2
kokolegorille

kokolegorille

It was not clear data comes from external API…

I have implemented GenStage pipeline with HTTPoison to get this concurrency.

But my latest web scrap was with Finch to download paintings/metadata concurently from WikiMedia.

1
Post #6

Where Next?

View thread on forum (has 7 responses!)
api
json
json-api
Home Questions & Help>Questions
#api#json#json-api
5 1439 7
Last post

Popular in Questions Top

qwerescape
Questions & Help>Questions

How to get the call stack/stack trace at any point in code?
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
53 16124 4
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43527 3
New
Patoshizzle
Questions & Help>Questions

(Postgrex.Error) FATAL 28P01 (invalid_password) password authentication failed for user “postgres”
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
#ecto#postgres#troubleshooting
10 29754 20
New
JeremM34
Questions & Help>Questions

How can I check Phoenix version?
Hello, how can I check the Phoenix version ? Thanks !
/phoenix
35 28221 8
New
nobody
Questions & Help>Questions

How to bind phoenix app to specific ip address?
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
/phoenix
17 19893 18
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11845 8
New
aalberti333
Questions & Help>Questions

Enum.map over list of key/value pairs with a map as the value
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
#enummap
7 19866 6
New
chensan
Questions & Help>Questions

(Postgrex.Error) ERROR 42804 (datatype_mismatch): column "" cannot be cast automatically to type integer
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
#ecto
29 13524 4
New
nsuchy
Questions & Help>Questions

IEX in Windows Powershell?
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
#iex#microsoft-windows#windows#powershell
15 30117 4
New
marick
Questions & Help>Questions

Many-to-many associations in phoenix and ecto
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
/phoenix
34 21273 20
New

Other popular topics Top

AstonJ
Chat & Discussions>Polls

Which Neovim configuration distro do you use/recommend? (Poll)
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
#polls#code-editors#neovim
79 13941 31
New
ovidiubadita
Questions & Help>Questions

Elixir for game development
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
#game
30 23559 25
New
stefanchrobot
Questions & Help>Questions

How to decode a JSON into a struct safely?
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
#structs#json
29 21152 26
New
vrod
Questions & Help>Questions

Starship (cross-shell prompt) error - (starship::utils): Executing command "elixir" timed out
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
#starship
8 17307 3
New
alice
Questions & Help>Questions

What are the main benefits of Elixir compared to Clojure?
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
#clojure
229 33752 47
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 23194 6
New
boundedvariable
Chat & Discussions>Discussions

Can we beat Kafka if we build it in Elixir?
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
#distributed-systems
96 20660 29
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 20317 15
New
marick
Questions & Help>Questions

Many-to-many associations in phoenix and ecto
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
/phoenix
34 21273 20
New
lanycrost
Questions & Help>Questions

How To Implement if...else if...else condition
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
#how-to-question
40 52286 6
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

WyewWorks

Enabling companies to succeed by building software people love.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement