pillaiindu
Where do I put driver configuration for hound in a (non Phoenix) Elixir project for web scraping?
Hi Everyone!
I want to do some web scraping, for which I wanna use the hound package as it have everything required for web scraping.
I know how to use it in a Phoenix project for testing, which I learnt from Phoenix inside out, thanks to @shankardevy. In a Phoenix project we had to put config :hound, driver: "phantomjs" in config/config.exs, if we were using phantomjs.
But here the situation is a bit different.
- Its not a Phoenix project.
- I want to use it in a file inside
lib/nottest/, as this time I’m using it for web-scraping instead of testing.
Thank you!
Marked As Solved
DevotionGeo
I don’t know if it’s from the start or it changed in the recent versions (I predominantly created Phoenix apps, not bare Elixir apps), but a mix new something won’t add the config.exs by default.
Add a config.exs under a config\ folder with the following code. If you’re using something other than phantomjs edit the code accordingly.
use Mix.Config
config :hound, driver: "phantomjs"
Also let us know if it worked or not.
Also Liked
NobbZ
Of course :hound will fallback then to the defaults.
If you do not want to use the defaults, you need to configure them.
Though I’d check if there is some API for hound that allows for runtime configuration of single scrapers rather than a global boot time configuration…
Phoenix is just a library. It doesn’t make your projects different in any way, they are just plain OTP applications.
NobbZ
Seems like you can use Hound.Helpers.Session.start_session/1 to start a runtime configured session.
PS: I really don’t like hounds API, as it has a lot of hidden magic, I prefer wallabys explicit passing around of the responsible client/connection.
NobbZ
It should just work…
Make sure its not runtime: false and not only: :test in your deps.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"