Crawler Data

Questions / Help
,
7

@suryaniwati: Let me summary everything:

Similar topic

We have already talked about tools for collect (scrape) data from web pages here:

Database

Easiest way to validate and save data into database is using Ecto library:

Rust dependencies

And for your last question:

It’s as simple as install rust stable (using asdf tool). The rest is as easy as add elixir dependencies and compile project. :smile:

Full setup

  1. Ensure you have dependencies for compile sources (depends on your OS/distro)
  2. Install asdf
  3. Install Erlang plug-in for asdf
  4. Install Erlang
  5. Set Erlang version
  6. Install Elixir plug-in for asdf
  7. Install Elixir
  8. Set Elixir version
  9. Add Rust plug-in for asdf
  10. Install Rust
  11. Set Rust version
  12. Add PostgreSQL plug-in for asdf
  13. Install PostgreSQL database
  14. Set PostgreSQL version

Example script

# Contents of file: "setup_environment.sh"

#!/bin/bash

# asdf
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.4.0
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc
source ~/.bashrc

# Erlang
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf install erlang 20.2.2
asdf global erlang 20.2.2

# Elixir
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf install elixir 1.6.1
asdf global elixir 1.6.1

# Rust
asdf plugin-add rust https://github.com/code-lever/asdf-rust.git
asdf install rust stable
asdf global rust stable

# PostgreSQL
asdf plugin-add postgres https://github.com/smashedtoatoms/asdf-postgres.git
asdf install postgres 10.1
asdf global postgres 10.1

Notes

  1. Feel free to modify script and install different versions of packages or different databases.

Other helpful resources

Finally this article should be helpful for setup Ecto 2 based project:

2 Likes