JoeZMar
I am attempting to make a web scraper that started out as a tool for just one person. As word got around I am finding out that there is actually somewhat of a need for this service. I started out building this with the intention of learning how to integrate OTP into a phoenix app and so far so good.
The scraper itself requires a user to log in to the targeted SPA, and monitor items in an auction board. When an item drops that fits the users parameters, it alerts the user. Right now I am using Hound and PhantomJS headless so that JS is rendered, but it was also just a choice I made when I was thinking about using this for just one person.
What are some common pitfalls that I should make sure that I am watching out for when making requests from several different users? I am making sure that each user is using the same User Agent each request, but how would I go about setting a proxy for each connection? I ask this because currently I have one user that will log in and monitor the trading board pulling in each new “item” that gets put up, but I would like to eventually implement the “purchase” with the bot as well. For that I would need my application to login as the other user and make the purchase. I am currently having the user log in and my application stores their cookie for credentials for that part and not their actual credentials for the site.
What is the legality of using a bot to log in to another site as well? Is there a proper way to scrape a site that uses authorization? If anyone that has experience scraping sites at a large scale I would love to talk with them outside of the forums and possibly get some more advice.
Where Next?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
arnodirlam
As a first pointer, there was a really nice talk at ElixirConf 2019 by Adam Mokan on exactly that topic:
JoeZMar
I coincidentally sent him a PM yesterday because of some of his responses on the forum to scraping. I didn’t realize he had a talk. Thanks!