I needed a crawler for something else anyway so I put a quick project together to try all this out. Used the elixir ecosystem to handle each aspect of search. It’s interesting the design concerns that come up as you can see from my growing wish-list on the readme. That search article you linked @achempion has a lot of great food for thoughts I may try to address.
Anyway running a search site has been pretty fun so I posted a demo site with only a couple sites partially indexed (elixirforum and infogalactic.. the index will grow a little every day). And I actually need Riak for something too so I may try swap that in. The readme lists the tech choices I made thus far.
Thank you for posting the source code.