JoeZMar

JoeZMar

Multiple Hound Sessions, but with different IP addresses?

I am creating a web scraper/bot. Because the site that I am scraping is very JS heavy I chose Hound with PhantomJS running.

Right now if I have 5 people that want to use the bot, they log in and provide the username and password for their account for the site they want to scrape. Each account spawns a new process, and uses Hound to bot the site.

My question is, is there any way to spawn each process with a different IP address? I think that you can configure the PhantomJS to run off a proxy, but is there a way to have each browser run off a proxy?

Most Liked

Shrike

Shrike

From my experience with watir it should be possible, I’m running 84 different phantomJS processes on one computer without any cross contamination between them, and that’s ruby I wonder how easy it should be with Elixir.
Only reason it’s 84 is that on Amazon Linux the limit is 85 before I finished all the ports, and the memory was low any way so didn’t try to fix it or even check what was the problem.

Shrike

Shrike

My problem was with phantomjs that after the 85 instance it couldn’t get a new port so it wouldn’t start, you should test it before moving on, my scalability was limited by that + memory phantomjs just love memory even for simple sites so 4gd server couldn’t hold more.
I used Sinatra to move the requests to multiple servers keeping the state in json and passing it back to our elb on fails, each server could deal with any request.
But to say the truth I got this idea from here and wouldn’t have used it if I didn’t know Elixir, but I had to get everything up and running in about a month if I had 3 months I would have done it using Elixir.
In watir proxy definitions are defined for each phantomjs by the use of the driver so I’m guessing you should find it in the same place in hound.
Also if you need loads of ips using proxies you should check luminati.io their data center offer is not bad.

Shrike

Shrike

Never tried to run more than one session per instance, just thinking about all the concurrency problems that might happen would get my pants dirty, especially when running so many instances.
Come to think about it I already have 1k concurrent phantomjs sessions (12 computers) and I tested it and everything is working great even in prod (AWS t2.medium).
I would test chrome + Firefox headless before making a decision, too many unknown variables for me to give any meaningful advice.
Do some testing understand your constraints memory/available instances for each platform etc.
What I did is restart each phantomjs instance after I used it to make sure my memory is clean, you might not need to do it.
I would also recommend opening your browser in advance, when I tried opening each when needed I lost 10 seconds in overall time, it took the server 1sec to open each session and that wasn’t truly concurrent.

PS- not sure what would happen with session instead of instances as I never tried it always had to either deal with concurrency or one instance only and always with ruby, which is not the best case scenario, I wonder how hound would deal with it, and if each session uses a different port too.

Where Next?

Popular in Questions Top

nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43757 214
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement