ravernkoh
Simple crawler using GenStage (questions about the structure)
Hi! I intend to build a simple web crawler in Elixir, and since a web crawler is really just a pipeline of url-queue → fetcher → parser (with some filters and rate limiters in between) I have decided to use GenStage for this. I have already built the url queue. Now, I’m thinking of how to build the fetcher.
Obviously, the fetcher will need to perform HTTP requests in parallel, and since there is a limit to how many I should do at a time (bandwidth right…? correct me if I’m wrong about this), I should use some sort of pooling system. I have looked at ConsumerSupervisor in the GenStage docs, but it is only for the last part, the consumer portion.
How should I go about implementing this, where the fetcher stage is a producer_consumer (So i cant use the ConsumerSupervisor)?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts
mischov
For some ideas you might check out Crawlie, which I believe uses GenStage.
Qqwy
If your crawler wants to grow to become more sophisticated, I’d suggest looking into Event Sourcing: By persisting the results of each stage independently, you could at some point decide that the parser can be smarter (such as extract more info) without having to re-crawl all previously-crawled resources.
As for implementing the
. Great tip, @mischov!
fetcherstage itself, I think Crawlie’s UrlManager indeed gives some good suggestions