KristerV

KristerV

Hexdocs search engine for devs

How I currently use Hexdocs

I use hexdocs all day every day, but finding the right module and function takes too long even with my bookmark system.

Too much scrolling, reading, clicking around for the right function. I already know what function I need!

Solution

So I’m making a tool that lets you search many Elixir packages from a single website. Yeah, it’s a website and not a package (couldn’t find a better section).

I’ll show you what I mean. This is very much a WIP, but i have the data in and search does function.

Here’s a few implementation details.

Data source

There are many ways to get data, but I wanted this to work with any package (not that all are supported) and without much going on on disk (don’t want to generate docs myself).

I have a spider that

  1. Gets the sidebar-items.json file from hexdocs.pm/{package}
  2. Parses JSON, convert it into rows
  3. Inserts into DB

Crawly was way too much overhead, would use HTTPoison and Floki next time. May refactor at one point.

Fuzzy search

I’m just using ILIKE in Postgres for now.

def list_fuzzy(str) do
    search_str =
      str
      |> String.split(" ")
      |> Enum.intersperse("%")
      |> Enum.join()
      |> then(&"%#{&1}%")

    from(p in Page,
      where: fragment("? ilike ?", p.search_body, ^search_str)
    )
    |> Repo.all()
  end

Ideally the index would be client-side for instant results, but I’m not sure how to do fuzzy search in JS yet and I probably wont think about it unless the service gets slow.

Version 1.0 plans

The end result would ideally have

  • iframe with hexdocs straight in it for instant results (not sure how possible)
  • auto-bookmark popular pages
  • keyboard navigation

But we’ll see. It’s just a fun side-project.

Feedback welcome

I am building it for myself, but obviously would love more users and ideas on what would be awesome.
Also looking for cool domain names. otherwise it’s going to be hexdocs-search.krister.ee - not so pretty. I’d go with hexdocs.search, but that’s not easy to aquire. TLD is supposed to exist, but not on my provider, Namecheap.

First 10 of 57 Posts Switch mode

gregvaughn

gregvaughn

I don’t want to take away your fun on a side project, but if it were me, I’d use mix hex.docs fetch to cache all docs for my project’s deps locally, then run rg (ripgrep) in that directory for function names.

17
Post #1
dimitarvp

dimitarvp

Yeah, that’s exactly what I do. I crafted a few very simple ripgrep regex-ish expressions and alias-ed them in my ~/.zshrc and then just do e.g. exdoc Enum.group_by in the terminal.

…Actually it might be time to extend that script to use open (Mac) / xdg-open (Linux) so as to just open the docs in a browser. :thinking:

gregvaughn

gregvaughn

Yes, and you should blog it and/or gist it for sharing!

dimitarvp

dimitarvp

Yeah, probably before hitting 50 years old would be nice :003: (currently 42).

stefanchrobot

stefanchrobot

TIL mix hex.docs fetch! It’s exactly what I was looking for!

But… it would be great to have something like mix hex.docs offline (without more args) that would open up aggregated docs for the dependencies of the current project. I was actually thinking about opening an issue on ExDoc to suggest something like that. Thoughts?

11
Post #5
Exadra37

Exadra37

Like this:

 mix help hex.docs

                                  mix hex.docs                                  

Fetches or opens documentation of a package.

If no version is specified, defaults to version used in the current mix
project. If called outside of a mix project or the dependency is not used in
the current mix project, defaults to the latest version.

## Fetch documentation for all dependencies in the current mix project

    $ mix hex.docs fetch

## Fetch documentation for offline use

Fetches documentation for the specified package that you can later open with
mix hex.docs offline.

    $ mix hex.docs fetch PACKAGE [VERSION]

## Open a browser window with offline documentation

    $ mix hex.docs offline PACKAGE [VERSION]

## Open a browser window with online documentation

    $ mix hex.docs online PACKAGE [VERSION]

## Command line options

  • --module Some.Module - Open a specified module documentation page
    inside desired package
  • --organization ORGANIZATION - Set this for private packages belonging
    to an organization
  • --latest - Looks for the latest release of a package
  • --format epub - When opening documentation offline, use this flag to
    open the epub formatted version

Location: /home/developer/.mix/archives/hex-1.0.1/hex-1.0.1/ebin

stefanchrobot

stefanchrobot

Yes, but one that allows easy jumping between offline docs.

cmo

cmo

I believe dash can do hex docs.

KristerV

KristerV OP

I don’t mind. I thought about this approach, but it seems to me that the complexity outweighs the benefits here. btw i want the search to reflect the menu items and not just the functions, since guides and sections can be very useful. so it makes more sense to just parse the JSON that the sidebar is compiled of. No disk management, no grepping, no mistakes. Or is my logic flawed?

I’d love to see that too.

For Linux and Windows there’s also Zeal, but I couldn’t get used to the interface. Also there’s only one broken repo that transforms hexdocs into docsets, which Zeal needs, so only Elixir itself is supported.

LostKobrakai

LostKobrakai

I was toying yesterday with porting the code to a ex_doc formatter (similar to the epub one), so they could be generated directly with ex_doc and possibly hosted by hexdocs.

Where Next?

Trending in RFCs Top

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New

We're in Beta

About us Mission Statement