Crawler Data

I usually use httpoison with floki. There has been some discussion about it

You can replace floki with

For any interaction with a database (postgres by default) you can use ecto.

So my approach would be (roughly) like this:

defmodule Crawler do
  def crawl!(url) do
    %HTTPoison.Response{body: body, status: 200} = HTTPoison.get!(url)
    html = Floki.parse(body)
    contents = Floki.find(html, "article") # or whatever you are interested in
    # see ecto docs to understand what Repo does
    Crawler.Repo.insert!(%Article{contents: contents})
    urls =
      html
      |> Floki.find("a")
      |> Enum.map(fn anchor -> Floki.attribute(anchor, "href") end)
    # spawn more tasks to crawl other pages, or keep crawling in the current process
  end
end

It’s not a “tool from god”, but it’s very easy to setup and it have very simple commands, so I think that’s best answer for all members (including beginners).

@suryaniwati: Let me summary everything:

Similar topic

We have already talked about tools for collect (scrape) data from web pages here:

Database

Easiest way to validate and save data into database is using Ecto library:

Rust dependencies

And for your last question:

It’s as simple as install rust stable (using asdf tool). The rest is as easy as add elixir dependencies and compile project.

Full setup

0. Ensure you have dependencies for compile sources (depends on your OS/distro)
1. Install asdf
2. Install Erlang plug-in for asdf
3. Install Erlang
4. Set Erlang version
5. Install Elixir plug-in for asdf
6. Install Elixir
7. Set Elixir version
8. Add Rust plug-in for asdf
9. Install Rust
10. Set Rust version
11. Add PostgreSQL plug-in for asdf
12. Install PostgreSQL database
13. Set PostgreSQL version

Example script

# Contents of file: "setup_environment.sh"

#!/bin/bash

# asdf
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.4.0
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc
source ~/.bashrc

# Erlang
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf install erlang 20.2.2
asdf global erlang 20.2.2

# Elixir
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf install elixir 1.6.1
asdf global elixir 1.6.1

# Rust
asdf plugin-add rust https://github.com/code-lever/asdf-rust.git
asdf install rust stable
asdf global rust stable

# PostgreSQL
asdf plugin-add postgres https://github.com/smashedtoatoms/asdf-postgres.git
asdf install postgres 10.1
asdf global postgres 10.1

Notes

1. Feel free to modify script and install different versions of packages or different databases.

Other helpful resources

Finally this article should be helpful for setup Ecto 2 based project: