You probably have some transitive compile time dependencies, where module A needs module B to compile, but to compile module B you need to compile C.
The command mix xref graph --label compile --sink lib/my_module.ex is a good starting point, the files listed depend on lib/my_module.ex to compile, so any change to it will recompile all of them.
These are some pages to understand what is going on and how to fix compile time dependencies:
