I’ve implemented my first web-scraper with Crawly following the tutorial published on the official docs.
I’m at the point where I start a single crawler module from iex and it seems it’s successfully crawling but it’s not writing to disk.
I’ve checked it’s crawling the website and extracting some items using IO.inspect.
I’ve limited the total number of items to make sure it completed gracefully a run.
The Crawly.Pipelines.WriteToFile module is on the pipeline and it’s configured to write to the user-writable folder /tmp/crawly, but at the end of the run it’s as empty as ever.
I’ve enabled logging pointing in the same folder but that’s not written either.
I’ve never debugged an Elixir/Erlang app and I still find the idea a bit daunting.
Anyone has a clue why it’s not writing to disk?
