Hex.pm mirror for isolated environments

ejpcmac

Starting November, I will spend one year in Kerguelen Islands as the System Administrator of the scientific base. There, internet access is very low—understand: 100 kb/s for 40 to 100 persons, therefore reserved for professional activities. Provided this constraint, I am currently working on getting my whole development environment offline-ready. I’ve started reasearch about building a local binary cache for Nix and crates.io (the Rust equivalent for hex.pm). Now, I am here for the Elixir part.

I want to be able to start new Elixir projects there. To achieve this, I would like to setup an offline read-only hex.pm mirror. Here are my questions:

How can I get a full copy of the hex.pm repository?
What do I need to use it with Mix, so I can do mix deps.get and my dependencies are fetched from the local cache?
Would it be possible to setup a local copy for the web interface too, so I could search and browse the packages? Otherwise, how to look for packages for a new use case?
Same for Hexdocs, it would be neat to have it offline too.

These options would be interesting for other kind of isolated environments, like restricted networks in some companies for instance.

4 comments

#hex #hexdocs #offline

2 3020 4

2019-10-29 15:39:33 UTC

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

OvermindDL1

By chance MiniRepo was just released, a mini version of hex.

As for a duplicate of hex.pm, that sounds large, probably have to ask one of the platformatic people here, @josevalim is one, unsure who else?

You can generate docs from packages in hex too for note.

Post #1

ejpcmac

Oh neat! I’d seen MiniRepo, but not read about its mirroring capabilities. So if I setup a mirror like in the development example but without the only parameter, I should be able to mirror the full hex.pm and add HEX_MIRROR_URL in my environment so that I use it by default, right? I’ll try this next week for sure

P.S.: Sorry for the delayed reply, I’m moving soon + I need to prepare my canteens which are shipped next week, so until then I have an unprecedented high latency.

Post #2

wojtekmach

Hex Core Team

Yes, you can configure your mirror to fetch all packages. MiniRepo is pretty minimal, it doesn’t mirror docs or allow you to do search and similiar that you mentioned in the initial post. For docs, I’d write a small script that uses GitHub - hexpm/hex_core: Reference implementation of Hex specifications. · GitHub to grab package versions and download and unpack relevant docs.

Post #3

ejpcmac

For posterity: one neat thing about how the repo is stored is it can be served statically. This way, I can use mini_repo on any server to keep a mirror in sync, replicating it to my laptop through ZFS’s send/recv commands—which will also allow me to send incremental updates through the postal services.

On my laptop side, the repo is served by a simple Apache httpd, which I also use to serve other mirrors. By setting HEX_MIRROR_URL in my global environment, this whole process becomes really transparent and I can work fully offline without any warning. That’s perfect

Post #4

— All posts loaded —