JoeZMar
Multiple Hound Sessions, but with different IP addresses?
I am creating a web scraper/bot. Because the site that I am scraping is very JS heavy I chose Hound with PhantomJS running.
Right now if I have 5 people that want to use the bot, they log in and provide the username and password for their account for the site they want to scrape. Each account spawns a new process, and uses Hound to bot the site.
My question is, is there any way to spawn each process with a different IP address? I think that you can configure the PhantomJS to run off a proxy, but is there a way to have each browser run off a proxy?
Most Liked
Shrike
From my experience with watir it should be possible, I’m running 84 different phantomJS processes on one computer without any cross contamination between them, and that’s ruby I wonder how easy it should be with Elixir.
Only reason it’s 84 is that on Amazon Linux the limit is 85 before I finished all the ports, and the memory was low any way so didn’t try to fix it or even check what was the problem.
Shrike
My problem was with phantomjs that after the 85 instance it couldn’t get a new port so it wouldn’t start, you should test it before moving on, my scalability was limited by that + memory phantomjs just love memory even for simple sites so 4gd server couldn’t hold more.
I used Sinatra to move the requests to multiple servers keeping the state in json and passing it back to our elb on fails, each server could deal with any request.
But to say the truth I got this idea from here and wouldn’t have used it if I didn’t know Elixir, but I had to get everything up and running in about a month if I had 3 months I would have done it using Elixir.
In watir proxy definitions are defined for each phantomjs by the use of the driver so I’m guessing you should find it in the same place in hound.
Also if you need loads of ips using proxies you should check luminati.io their data center offer is not bad.
Shrike
Never tried to run more than one session per instance, just thinking about all the concurrency problems that might happen would get my pants dirty, especially when running so many instances.
Come to think about it I already have 1k concurrent phantomjs sessions (12 computers) and I tested it and everything is working great even in prod (AWS t2.medium).
I would test chrome + Firefox headless before making a decision, too many unknown variables for me to give any meaningful advice.
Do some testing understand your constraints memory/available instances for each platform etc.
What I did is restart each phantomjs instance after I used it to make sure my memory is clean, you might not need to do it.
I would also recommend opening your browser in advance, when I tried opening each when needed I lost 10 seconds in overall time, it took the server 1sec to open each session and that wasn’t truly concurrent.
PS- not sure what would happen with session instead of instances as I never tried it always had to either deal with concurrency or one instance only and always with ruby, which is not the best case scenario, I wonder how hound would deal with it, and if each session uses a different port too.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









