Parallel compiler cannot find files sometimes

Questions / Help
, , ,
1

I’m getting an intermittent compilation error when building a project in Docker:

== Compilation error in file lib/sentry/logger.ex ==
** (MatchError) no match of right hand side value: {:error, :enoent}
    (elixir) lib/kernel/parallel_compiler.ex:198: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6
could not compile dependency :sentry, "mix compile" failed. You can recompile this dependency with "mix deps.compile sentry", update it with "mix deps.update sentry" or clean it with "mix deps.clean sentry"

Kernel.ParallelCompiler.spawn_workers/6 calls :elixir_compiler.file/2, which calls file:read_file/1, which returns {error, enoent} if it can’t find a file.

:elixir_compiler.file/2 tries to pattern-match…

{ok, Bin} = file:read_file(File)

…fails, and raises the MatchError we see above.

Here are a few thoughts:

  • I could just do what the instructions tell me to do :slight_smile: …and recompile the dependency with mix deps.compile sentry, but this only happens intermittently, so I don’t think the problem is with how I’m compiling. This is part of an automated build, too, and recompiling one dependency occasionally isn’t ideal.
  • I’m using the most recent version of Sentry, so mix deps.update sentry won’t help.
  • I don’t think mix deps.clean sentry is the answer because I’m starting off with a clean slate; I’m building the project while building a Docker image. Both deps/ and _build/ are in my .dockerignore file.
  • This only seems to happen with the Sentry library.
  • The issue is intermittent. On most attempts to build the project and the Docker image, everything compiles correctly.
  • Even though the error was in lib/sentry/logger.ex in the code snippet above, this error sometimes springs up in other files in the Sentry library.
  • My Docker image is running Erlang/OTP 20, and Elixir 1.6.1 compiled with OTP 20, on Debian GNU/Linux 8.

Do you have any suggestions for how to debug this? I’m a bit flummoxed. There’s an existing issue on the Sentry project that describes the problem, but I thought it might actually be a bug in the Elixir compiler, so I posted here, just in case :smile:

2

One idea is to use strace at the moment of compilation and trace all of the system calls happening on lib/sentry/logger.ex. Something like:

$ mix deps.get
$ enable strace on deps/sentry
$ mix deps.compile
$ disable strace
$ mix compile

For all purposes, it seems the file exists in disk. So we need to figure out if for some reasons docker is saying it does not. In case docker does say it exists and Erlang still says it doesn’t, then there is something else at play.

1 Like
3

Are you seeing the problem on a Mac?

Using my mix project and docker files (which are unrelated to Sentry) on both an Mac and on our Linux-based build server, I’ve seen similar compile problems “frequently” when running in the dev environment on my Mac laptop, but never when running the container on the build server. The Mac often finishes without error - but will fail once out of 4 or 5 times.

I suspect that it may be related to the synchronization and timing between the Macintosh file system, and the container’s virtual file system. You may choose to read https://docs.docker.com/docker-for-mac/osxfs/#performance-issues-solutions-and-roadmap

But please understand that this is just a suspicion on my part - I have no verifiable evidence to prove or disprove.

2 Likes
4

I’m only seeing problems on the build server, unfortunately. I made a loop to clean, get, and compile deps during the build process, and I got four failures out of 100 attempts, so the problem isn’t easy to reproduce. Your suspicion of the synchronization and timing between file systems sounds like a reasonable one. :thumbsup: I’ll look into that.

5

strace is a good idea! It looks like Docker won’t let me use strace as part of the build process, due to permissions issues. I had to docker exec -it into the running container to use it. I ran strace mix deps.compile. strace gave around 1000 lines of output, but the output stopped when the files started compiling. I may see if something like tracefile does the trick.

As I was playing around with this, I got some more failures. The weird thing is that one compilation failure was in a file that only depended on :os.system_time, DateTime, Map, String, Mix.Dep, and Enum. All of those should have been available.