KristerV

How I currently use Hexdocs

I use hexdocs all day every day, but finding the right module and function takes too long even with my bookmark system.

Too much scrolling, reading, clicking around for the right function. I already know what function I need!

Solution

So I’m making a tool that lets you search many Elixir packages from a single website. Yeah, it’s a website and not a package (couldn’t find a better section).

I’ll show you what I mean. This is very much a WIP, but i have the data in and search does function.

Here’s a few implementation details.

Data source

There are many ways to get data, but I wanted this to work with any package (not that all are supported) and without much going on on disk (don’t want to generate docs myself).

I have a spider that

Gets the sidebar-items.json file from hexdocs.pm/{package}
Parses JSON, convert it into rows
Inserts into DB

Crawly was way too much overhead, would use HTTPoison and Floki next time. May refactor at one point.

Fuzzy search

I’m just using ILIKE in Postgres for now.

def list_fuzzy(str) do
    search_str =
      str
      |> String.split(" ")
      |> Enum.intersperse("%")
      |> Enum.join()
      |> then(&"%#{&1}%")

    from(p in Page,
      where: fragment("? ilike ?", p.search_body, ^search_str)
    )
    |> Repo.all()
  end

Ideally the index would be client-side for instant results, but I’m not sure how to do fuzzy search in JS yet and I probably wont think about it unless the service gets slow.

Version 1.0 plans

The end result would ideally have

iframe with hexdocs straight in it for instant results (not sure how possible)
auto-bookmark popular pages
keyboard navigation

But we’ll see. It’s just a fun side-project.

Feedback welcome

I am building it for myself, but obviously would love more users and ideas on what would be awesome.
Also looking for cool domain names. otherwise it’s going to be hexdocs-search.krister.ee - not so pretty. I’d go with hexdocs.search, but that’s not easy to aquire. TLD is supposed to exist, but not on my provider, Namecheap.

57 comments

#search #hexdocs #library-rfc

18 6123 57

2025-01-19 09:21:23 UTC

Showing Posts 57 to 48

Show Best Posts
Show All (oldest first)
Show All (newest first)

KristerV

welp, looks like this project is now deprecated in favor of an official solution New HexDocs home page · Issue #49 · hexpm/hexdocs · GitHub

was happy to fill the gap in the meantime. i’ll move the v2 to the main url and leave it up until the official solution is available.

Post #57

KristerV

i went reading and indeed with fly deploy the fly scale memory command result is reset to whatever the fly.toml defines. so changing the conf and service will stay up. also increased DB CPU, because the app became so slow.

Post #56

tcoopman

Now it works indeed. I’ll give it a try later. Thanks

Post #55

KristerV

okay, very strange. i scaled the app to 512mb, but right now it seems to be 256mb again.. how can this be.. i must be confusing something. anyway scaled it to 512mb again and it’s online.

edit: since i left in the automation that machines go offline if not used - perhaps it resets the scaling then.. hmm

Post #54

tcoopman

It doesn’t seem to load for me right now. I’m on mobile, but it just hangs when loading.

Post #53

KristerV

Updates!

Wrote it from scratch (with Ash)

The v2 is available at http://hexdocs2.krister.ee/ - please test it out. I’ll move it to the main (sub)domain after a while.

I bricked my original server. The Erlang version didn’t install anymore with asdf, the ubuntu on GitLab CICD was discontinued, once I started updating stuff things got worse and now the server is in the limbo state where if I restart it it most likely just wont come up at all.

I didn’t like some architecture stuff anyway, so decided to rewrite everything with Ash and Oban. I must say Ash is a game changed. Steep learning curve, but saves an immense amount of time when building stuff.

In any case crawling a new package now takes a few seconds like it’s supposed to.

I also spent a stupid amount of time trying to deploy an umbrella app with this app inside, but ultimately failed. Figured out a lot of problems, but ultimately didn’t make it. I have a bunch of apps where i’m the only user, thought this would save on server costs.

Mobile support

So automatic sidebar hiding and then the layout is such that the landing page makes sense. I might reverse the automatic hiding on desktop, wanted to try it out.

Full-text search!

I decided to go with the cheat code option where pressing enter will go to the packages own search function. It can be a bit slow and the results are just as messy as the official page. But it does work and “live hooks” does return the JS interoperability page. Needs more testing.

Post #52

KristerV

that is a good question indeed. there’s the ILIKE method that’s fast to implement, but then there’s the actual full text plugin that is okay to implement, not sure how performant either of those are. we all know that a separate server with elasticsearch or something is the real solution, but i really don’t want to go there, because of time.

however i recently added some basic analytics to the page and seems like 10 people a day are actually using it. so i’m kind of motivated to figure this out now. not sure when though, got another one of those never ending fires that i’m dealing with for a client atm

edit: i just realized there’s a cheat code. the official docs do full-text when you press enter. what if on enter it takes the first keyword and if it’s a package, then it gives you the official website full-text search? could even mimic the UI where the automatic results appear in a drop-down, although i feel like that’s going to get confusing with a lot of results.

Post #51

D4no0

OK, now this changes things. I thought that you process all text.

I wonder how well would a out of the box postgres text search would fare with full documentation, I’ve used it once on a project but never with so much text.

Post #50

tcoopman

I just tried this out and one thing I’ve noticed is that searching for hooks instead of hook doesn’t return that result.

I quickly typed hooks and never saw the backend hooks. So I’m wondering if some sort of distance function might be useful. (the different amount of results from hook vs hooks is quite big)
Probably not?

What’s probably worse is that phx-click doesn’t return any results.

Post #49

KristerV

haha, those are some great reasons to use hexdocs on mobile you’ve convinced me. i’ll get it done.

yeah, i’ve had that specific problem myself. i don’t do any content parsing atm. not sure what the best approach is for that considering getting into ML and search servers would take way too much time.

Post #48

Load more posts (47 remaining)