Questions & Help>Questions

Fetch data from multiple pages

#api#json#json-api
cipher

cipher

Fetch data from multiple pages

how do I make my code get the number of pages available in advance, and then start several tasks to get several pages concurrently?

Example: example.com/api?id=1; example.com/api?id=2; …
and get the response from the server, until the page returns an empty array: “”

7 comments
#api#json#json-api
1490 7

Most Liked

kokolegorille

kokolegorille

There is this old post, I remember I liked it, but maybe now it’s a bit old…

Maybe You can get some inspiration. It uses poolboy and HTTPoison.

Because the problem is not really to be concurrent, but to manage this concurrency.

You don’t want to flood the host with 10’000 concurrent requests.

3
Post #7
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey cipher welcome! Any good programming task starts by breaking down a problem into simpler parts. The first part is: How to fetch a single page? What have you tried so far?

1
Post #1
kokolegorille

kokolegorille

It was not clear data comes from external API…

I have implemented GenStage pipeline with HTTPoison to get this concurrency.

But my latest web scrap was with Finch to download paintings/metadata concurently from WikiMedia.

1
Post #5

Where Next?

View thread on forum (has 7 replies!)
api
json
json-api
Home Questions & Help>Questions
#api#json#json-api
5 1490 7
Last post
REPLY VIEW THREAD WITH 7 POSTS

Popular in Questions Top

jononomo
Questions & Help>Questions

Where / How does the Mix environment variable get set?
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
#mix
31 23306 9
New
Trevoke
Questions & Help>Questions

(EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
I have an umbrella app. Some of the apps inside depend on other apps in the umbrella, unsurprisingly. I’m writing a test for one of the...
#otp#umbrella#testing#exunit#errors
26 22193 11
New
jononomo
Questions & Help>Questions

Websocket connection works on localhost, but get 403 error when deployed via docker
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
/phoenix#channels
8 27237 12
New
jay1
Questions & Help>Questions

Why isn’t mnesia the most preferred database for use in Elixir/Phoenix?
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
/phoenix#mnesia
124 21853 22
New
vrod
Questions & Help>Questions

Starship (cross-shell prompt) error - (starship::utils): Executing command "elixir" timed out
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
#starship
8 17621 3
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 20652 15
New
fayddelight
Questions & Help>Questions

Installing elixir via asdf shows zsh: command not found: iex
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
#erlang#asdf
44 17216 17
New

Other popular topics Top

vonH
Questions & Help>Questions

How do I kill a process ` #PID<0.186.0` in iex?
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
#iex#processes#how-to-question
37 21666 8
New
vonH
Questions & Help>Questions

Why would I choose Elixir as a general purpose programming language?
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
#functional-programming#use-cases
65 35269 13
New
JakeBecker
Chat & Discussions>Discussions

ElixirLS - the Elixir Language Server
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
#elixir-ls
1144 55125 245
New
greenz1
Questions & Help>Questions

Best way to send multiple files as HTTP response
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
/phoenix#api
3 19224 3
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
985 44778 311
New
dblack
Questions & Help>Questions

WebSocket is closed before the connection is established
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
/phoenix#troubleshooting#liveview
13 28242 22
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Honeybadger

Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking, logs, uptime, and Just Enough APM. Built for developers.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking, logs, uptime, and Just Enough APM. Built for developers.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement

Options

Show First Posts