Thanks for your work in updating your article! I’m starting to work on Crawly and will use your article as a reference.
Do you have any plans to make an article about downloading files? e.g., pdfs with Crawly. That would be really helpful for me and I suspect many others…
Sending you and your countrymen all the best! Thank you in advance!
Thanks for kind words! It would be interesting to give it a try. Do you have some website to check in mind?
Hi,
One limitation is see in Crawly is inability to submit urls after a spider started, we have a list of urls stored in a db and need to crawl it by some limit
Hi, I’ve been getting my head around Crawly for the past couple of days. Before that I was stubbing my toes in major skill-issues with Crawlee JS on the Apify platform.
My app has 20k urls that it’s tracking. Once a week I would like to ping each one to get the latest info. Am I able to pass more information with the start_urls or start_requests? I would like to pass along the db id of the related url so that it’s easier to handle it in the parse_item function.
I’m getting these debug messages. Is it a version mismatch?
Rebuilding...
[debug] Could not classify module Elixir.Hex.API as spider: %UndefinedFunctionError{module: Hex.API, function: :module_info, arity: 1, reason: nil, message: nil}
[debug] Could not classify module Elixir.Hex.API.Auth as spider: %UndefinedFunctionError{module: Hex.API.Auth, function: :module_info, arity: 1, reason: nil, message: nil}
[debug] Could not classify module Elixir.Hex.API.Key as spider: %UndefinedFunctionError{module: Hex.API.Key, function: :module_info, arity: 1, reason: nil, message: nil}
[debug] Could not classify module Elixir.Hex.API.Key.Organization as spider: %UndefinedFunctionError{module: Hex.API.Key.Organization, function: :module_info, arity: 1, reason: nil, message: nil}
[debug] Could not classify module Elixir.Hex.API.Package as spider: %UndefinedFunctionError{module: Hex.API.Package, function: :module_info, arity: 1, reason: nil, message: nil}
[debug] Could not classify module Elixir.Hex.API.Package.Owner as spider: %UndefinedFunctionError{module: Hex.API.Package.Owner, function: :module_info, arity: 1, reason: nil, message: nil}
Here are my mix.exs file
defp deps do
[
{:bcrypt_elixir, "~> 3.0"},
{:phoenix, "~> 1.7.14"},
{:phoenix_ecto, "~> 4.5"},
{:ecto_sql, "~> 3.10"},
{:postgrex, ">= 0.0.0"},
{:phoenix_html, "~> 4.1"},
{:phoenix_live_reload, "~> 1.2", only: :dev},
# TODO bump on release to {:phoenix_live_view, "~> 1.0.0"},
{:phoenix_live_view, "~> 1.0.0-rc.1", override: true},
# {:floki, ">= 0.30.0", only: :test},
{:floki, ">= 0.30.0"},
{:phoenix_live_dashboard, "~> 0.8.3"},
{:esbuild, "~> 0.8", runtime: Mix.env() == :dev},
{:tailwind, "~> 0.2", runtime: Mix.env() == :dev},
{:heroicons,
github: "tailwindlabs/heroicons",
tag: "v2.1.1",
sparse: "optimized",
app: false,
compile: false,
depth: 1},
{:swoosh, "~> 1.5"},
{:finch, "~> 0.13"},
{:telemetry_metrics, "~> 1.0"},
{:telemetry_poller, "~> 1.0"},
{:gettext, "~> 0.20"},
{:jason, "~> 1.2"},
{:dns_cluster, "~> 0.1.1"},
{:bandit, "~> 1.5"},
{:httpoison, "~> 2.2"},
{:dotenv, "~> 3.1"},
{:csv, "~> 3.2"},
{:crawly, "~> 0.17.2"}
]
end
No. Crawly scans the project files in order to find spiders inside. Those are debug messages, for files that are not spiders.
Thanks for taking the time, let me try to explain.
I’m trying to start the spider with a bunch of items from my database and – when parsing the responses – I want to write back to those records. It would be convenient to me to be able to pass the record itself as metadata so that inside parse_item/1 I can use that record’s information.
def init do
requests = Apps.list_apps()
|> Enum.map(fn app ->
Crawly.Request.new("https://#{app.bubble_id}.bubbleapps.io", [], [], meta: %{app: app})
end)
[start_requests: requests]
end
def parse_item(response) do
app = response.meta.app
# do something with response.body based on response.meta.app
end
Originally I thought I could (ab)use the middlewares of Crawly.Request.new/4, but that info would not be available in the response, correct?
As far as I see it’s not possible to pass meta information from init into parse_item.
As I see it, you have two approaches at this point:
parse_item it will be possible to update the record.response = Crawly.fetch("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", [headers: [id: "a897fe39b1053632"]])
response.request
%HTTPoison.Request{
method: :get,
url: "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
headers: [{"User-Agent", "Crawly Bot"}, {:id, "a897fe39b1053632"}],
body: "",
params: %{},
options: []
}
```