Did some more reading. ExDoc is using Code.fetch_docs
under the hood it seems inside defp docs_chunk. Cool I give it a try:
Code.ensure_loaded?(MathOps)
# true
Code.fetch_docs(MathOps)
# {:error, :module_not_found}
Bummer. Well after more reading I find this: Module documentation not immediatelly available after ensuring then module is compiled - #3 by josevalim. So I go down another rabbit hole trying to ensure all modules are done compiling etc get raw beam file etc and it feels overly complicated now. Hopefully someone can provide insight.