vkryukov
What's the best way to scrape a webpage in 2024?
Hello,
I working on a project that need to scrap web pages from time to time, and some of them require a “visit from a browser” to prevent scrapping, e.g., many website using Cloudflare protection:
iex(19)> Req.get("https://www.cell.com/chem/fulltext/S2451-9294(24)00494-7")
{:ok,
%Req.Response{
status: 403,
...
What’s the best way to solve this task with Elixir in 2024?
I can think of three possible ways:
- Use Wallaby (Commits · elixir-wallaby/wallaby · GitHub)
- Use Crawly (GitHub - elixir-crawly/crawly: Crawly, a high-level web crawling & scraping framework for Elixir. · GitHub), which seems to be a bit of an overkill for my use case
- Use GitHub - andrewvy/chrome-remote-interface: Elixir Client for the Chrome Debugger Protocol · GitHub (doesn’t seem to be supported anymore)
It looks like Wallaby is my best bet, but I’m curious if people have any comments or other suggestions here?
Most Liked
kevinschweikert
For simple stuff you could try GitHub - seanmor5/hop: A tiny web crawling framework for Elixir · GitHub
There is also a nice Dockyard article Web Crawling with Hop, Mighty, and Instructor
schneebyte
Might be enough to send proper HTTP headers and using the right User-Agent.
A lot of websites are nice enough these days to provide an internal API for their Javascript that already returns nicely structured JSON.
Search boxes and auto-complete are also great and often already return all the data i need.
Wallaby is probably easier to use but here is a fork with the newest cdp protocol versions, works fine.
D4no0
I did the exact same request with curl and it works, I think you are just missing some headers or you spammed too much and got temporarily banned.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"