Hexdocs search engine for us devs

Your Libraries & OS Mentoring Seeking Feedback
,
1

How I currently use Hexdocs

I use hexdocs all day every day, but finding the right module and function takes too long even with my bookmark system.

image

Too much scrolling, reading, clicking around for the right function. I already know what function I need!

Solution

So I’m making a tool that lets you search many Elixir packages from a single website. Yeah, it’s a website and not a package (couldn’t find a better section).

I’ll show you what I mean. This is very much a WIP, but i have the data in and search does function.

image
image1537×468 68.2 KB

Here’s a few implementation details.

Data source

There are many ways to get data, but I wanted this to work with any package (not that all are supported) and without much going on on disk (don’t want to generate docs myself).

I have a spider that

  1. Gets the sidebar-items.json file from hexdocs.pm/{package}
  2. Parses JSON, convert it into rows
  3. Inserts into DB

Crawly was way too much overhead, would use HTTPoison and Floki next time. May refactor at one point.

Fuzzy search

I’m just using ILIKE in Postgres for now.

def list_fuzzy(str) do
    search_str =
      str
      |> String.split(" ")
      |> Enum.intersperse("%")
      |> Enum.join()
      |> then(&"%#{&1}%")

    from(p in Page,
      where: fragment("? ilike ?", p.search_body, ^search_str)
    )
    |> Repo.all()
  end

Ideally the index would be client-side for instant results, but I’m not sure how to do fuzzy search in JS yet and I probably wont think about it unless the service gets slow.

Version 1.0 plans

The end result would ideally have

  • iframe with hexdocs straight in it for instant results (not sure how possible)
  • auto-bookmark popular pages
  • keyboard navigation

But we’ll see. It’s just a fun side-project.

Feedback welcome

I am building it for myself, but obviously would love more users and ideas on what would be awesome.
Also looking for cool domain names. otherwise it’s going to be hexdocs-search.krister.ee - not so pretty. I’d go with hexdocs.search, but that’s not easy to aquire. TLD is supposed to exist, but not on my provider, Namecheap.

16 Likes
2

I don’t want to take away your fun on a side project, but if it were me, I’d use mix hex.docs fetch to cache all docs for my project’s deps locally, then run rg (ripgrep) in that directory for function names.

14 Likes
3

Yeah, that’s exactly what I do. I crafted a few very simple ripgrep regex-ish expressions and alias-ed them in my ~/.zshrc and then just do e.g. exdoc Enum.group_by in the terminal.

…Actually it might be time to extend that script to use open (Mac) / xdg-open (Linux) so as to just open the docs in a browser. :thinking:

6 Likes
4

Yes, and you should blog it and/or gist it for sharing!

4 Likes
5

Yeah, probably before hitting 50 years old would be nice :003: (currently 42).

6 Likes
6

TIL mix hex.docs fetch! It’s exactly what I was looking for!

But… it would be great to have something like mix hex.docs offline (without more args) that would open up aggregated docs for the dependencies of the current project. I was actually thinking about opening an issue on ExDoc to suggest something like that. Thoughts?

10 Likes
7

Like this:

 mix help hex.docs

                                  mix hex.docs                                  

Fetches or opens documentation of a package.

If no version is specified, defaults to version used in the current mix
project. If called outside of a mix project or the dependency is not used in
the current mix project, defaults to the latest version.

## Fetch documentation for all dependencies in the current mix project

    $ mix hex.docs fetch

## Fetch documentation for offline use

Fetches documentation for the specified package that you can later open with
mix hex.docs offline.

    $ mix hex.docs fetch PACKAGE [VERSION]

## Open a browser window with offline documentation

    $ mix hex.docs offline PACKAGE [VERSION]

## Open a browser window with online documentation

    $ mix hex.docs online PACKAGE [VERSION]

## Command line options

  • --module Some.Module - Open a specified module documentation page
    inside desired package
  • --organization ORGANIZATION - Set this for private packages belonging
    to an organization
  • --latest - Looks for the latest release of a package
  • --format epub - When opening documentation offline, use this flag to
    open the epub formatted version

Location: /home/developer/.mix/archives/hex-1.0.1/hex-1.0.1/ebin
3 Likes
8

Yes, but one that allows easy jumping between offline docs.

1 Like
9

I believe dash can do hex docs.

4 Likes
10

I don’t mind. I thought about this approach, but it seems to me that the complexity outweighs the benefits here. btw i want the search to reflect the menu items and not just the functions, since guides and sections can be very useful. so it makes more sense to just parse the JSON that the sidebar is compiled of. No disk management, no grepping, no mistakes. Or is my logic flawed?

I’d love to see that too.

For Linux and Windows there’s also Zeal, but I couldn’t get used to the interface. Also there’s only one broken repo that transforms hexdocs into docsets, which Zeal needs, so only Elixir itself is supported.

2 Likes
11

I was toying yesterday with porting the code to a ex_doc formatter (similar to the epub one), so they could be generated directly with ex_doc and possibly hosted by hexdocs.

4 Likes
12

The service is now live. Lots of work to be done still, but basically usable already now.

Website: https://hexdocs.krister.ee/
GitLab: Krister Viirsaar / hexdocs-search · GitLab

Notes:

  • the search results dropdown is meant to stay open. sometimes you want to compare different functions.
  • The packages included for now are (open for suggestions)
[
    "elixir",
    "plug",
    "ecto",
    "ecto_sql",
    "phoenix",
    "phoenix_html",
    "phoenix_live_view",
    "phoenix_pubsub",
    "ex_unit",
    "timex",
    "quantum"
  ]

oh, that didn’t even occur to me. good idea! please keep us updated.

edit: oh… i had falsely assumed if it works for /elixir it’ll work for others and in prod it’s clear it doesn’t.
edit2: assumption was correct. code wasn’t. fixed.

7 Likes
13

absinthe and oban should be there by default as well IMO.

2 Likes
14

no problem. added.

image

up to date list available at https://gitlab.com/KristerV/hexdocs-search/-/blob/master/config/prod.exs#L100

3 Likes
15

Update.

  • fix: lots of functions were missing due to bug
  • design: much narrower results box with module based sections
  • UX: input text doesn’t disappear on click

image

3 Likes
16

New structure for results

I was stuck for long on how to get readable results. I tried a lot of ways to visualize the results and in the end I guess the conclusion is hexdocs itself has an amazing approach. So…

image
image479×873 41.5 KB

Hide results

Amongst other UX fixes you can now temporarily hide the results. (click for video)

https://imgur.com/a/KVVy88V

Onward

Now need to figure out how to order better.

2 Likes
17

DevDocs API Documentation ?

2 Likes
18

Does this support hex packages? Afaik it only has “Elixir” docs, but not docs for packages.

1 Like
19

i am just a devdocs.io user and was triggered by the UI similarities. But it seems possible to add other hex packages. See devdocs/adding-docs.md at main · freeCodeCamp/devdocs · GitHub and devdocs/elixir.rb at 44d42bbd5c13f728d081a2cd1c3b0052b223ab3f · freeCodeCamp/devdocs · GitHub

1 Like
20

Looks like adding packages is quite trivial. Except that the versioning needs to be reworked in that case.

Also Elixir itself is old version.

Hmm. Perhaps I wouldn’t have built this if I had known about devdocs. At the same time I still prefer native hexdocs, so not sad about it either. Now at least I can keep developing it according to my needs. Perhaps something original will come out of it. Of course maybe it’s a waste of time :slight_smile: Time will tell.

Thanks for dropping the link. Perhaps I’ll come back to it one day.

1 Like