Offline Mirror for mix deps?

I know if the deps are already downloaded in deps/ and then you run mix deps.get while offline, mix will attempt to contact hex.pm and then fall back to what is downloaded locally. So if it’s possible to cache deps/. in ci, that might help.

I’m curious if there’s a way to have mix skip attempting to contact hex.pm if it already has everything downloaded that it needs. That would save me a couple seconds each CI run.

Deps have the option to have a path, like:

deps() do
   [
       {:local_dependency, path: "path/to/local_dependency"}
   ]
end

Link to the docs, although I am not sure it accepts an actual package. The repo would have to be cloned?

One thing I was considering setting up was a MiniRepo Docker image pre-loaded with the hex deps the project needed. The build happens in a docker-compose so it makes sense in my case but maybe not all cases.

I use that sometimes for development work, but I don’t want the manual process of keeping something like that up to date. The nice thing about how yarn does it, is it saves the tarballs locally and the checksum for them in the lock file. That way you’re getting the exact same thing npm would be giving you, and you only have to track a smaller list of files. They could even be stored in git LFS.

I don’t like to store the reps folder in the repo because it’s so many files that can change a lot during updates. I do like having the assurance of the exact files I expect being stored in a way I can control though. It’s a tough compromise to make, but I feel like yarn addressed it in a pretty elegant way.

Reading through the yarn article it seems like Hex does all of this except it does it automatically without needing to configure a mirror.

All Hex packages you download are cached in ~/.hex which means that if your network is down and you need to fetch the package again the cache will be used as a fallback. You can force the cache to be used by setting the environment flag HEX_OFFLINE=1. Additionally, all your dependencies are locked in the mix.lock file to ensure you always get the same versions that were initially fetched and locked.

If you need something else then please elaborate on what you want to do and how it is different from what Hex provides today.

If I try to mix deps.get on a project, will it try to use this cache? Or will it always prefer to download “fresh” libs (even if same version)?

There are multiple layers of caching for the different stages of fetching dependencies. We will always try to use the cache if it’s available but that does not necessarily mean we make no HTTP requests if everything is cached.

First of mix deps.get will not do anything if you already have fetched dependencies in the projects deps/ directory and they match the lock. If dependencies are missing in deps/ or they do not match the lockfile they will be fetched.

Before fetching dependencies we may need to do dependency resolution in case dependencies are not locked or they are being updated with mix deps.update. To perform dependency resolution we need to fetch package indexes from the registry. Package indexes are cached but we will always make conditional HTTP requests [1] for them to ensure you have the latest version of the index, if you have the latest version they are not downloaded again.

When dependencies are resolved to a specific version their package tarballs can be fetched, the tarballs are also cached so if you fetched the tarball previously on the same machine and the checksum matches then no request will be made.

[1] HTTP conditional requests - HTTP | MDN

Thanks @ericmj , it seems like hex does handle this well already. However, it would be difficult to commit the cached tarballs since they reside in the user’s home directory. Is there any way to specify the location of the cache, so we can make sure it’s available in a CI/CD docker process?

We don’t have a way to configure only the cache directory location.

Can you explain why you want this feature, what existing problems does it solve? Personally I don’t think you should commit files to version control to help the reliability of CI/CD. I think any concerns about that should be handled by the CI/CD process, by making it more reliable or adding caching there.

• To handle when hex is down in CI => set up private repo. ( Self hosted hex repo - #8 by zazaian ) Unfortunately artifactory/nexus do not have them ( Mix deps.get - with Artifactory? )
• To speed up CI by caching => you may cache deps folder.

More notes

• You may just tarball deps folder for example and use HEX_OFFLINE env var. However, I’ve seen its content changes (e.g. creating new file under /deps) during compile time..
• You can easily set up hex repo with mini_repo - but you have to list packages to sync, or sync all.