Compiling igniter ** (Mix) Can't continue due to errors on dependencies

My best guess so far is that this is a bug in rewrite, as the error is coming from there

In fact i can see the Rewrite error message here :

i haven’t been able to see any async or concurrent calls to these, but my thought would be that the issue could be that somewhere?

Ultimately the only time that error is ever produced is when the phashes don’t match, so something must ultimately be the cause of that. The phash is of the file path and the contents, of the current file and of the file on disk.

Rgr that continuing

1 Like

I found an oddity:
iex(10)> debug_igniter.()

=== File Check ===
File exists?: true
File stats: %File.Stat{
size: 1281,
type: :regular,
access: :read_write,
atime: {{2025, 1, 21}, {0, 3, 54}},
mtime: {{2025, 1, 21}, {0, 3, 54}},
ctime: {{2025, 1, 21}, {0, 3, 54}},
mode: 33188,
links: 1,
major_device: 53,
minor_device: 0,
inode: 420502,
uid: 1000,
gid: 1000
}

=== File Read Test ===
File is readable
File size: 1281 bytes

=== Source Check ===
Error getting source: %Rewrite.Error{reason: :nosource, path: “/home/hunsazk/Documents/Github/hillbilly_millionaire/lib/hillbilly_millionaire/application.ex”, missing_paths: nil, duplicated_paths: nil, message: nil}
Current rewrite state: #Rewrite<1 source(s)>

=== Alternative Path Checks ===
Trying relative path: lib/hillbilly_millionaire/application.ex
No source found with relative path

=== Path Resolution ===
Current directory: /home/hunsazk/Documents/Github/hillbilly_millionaire
Absolute path: /home/hunsazk/Documents/Github/hillbilly_millionaire/lib/hillbilly_millionaire/application.ex
Relative path resolution: /home/hunsazk/Documents/Github/hillbilly_millionaire/lib/hillbilly_millionaire/application.ex
:ok
iex(11)>

drwxr-xr-x. 1 hunsazk hunsazk 38 Jan 20 18:03 accounts
-rw-r–r–. 1 hunsazk hunsazk 220 Jan 20 18:03 accounts.ex
-rw-r–r–. 1 hunsazk hunsazk 1.3K Jan 20 18:03 application.ex
-rw-r–r–. 1 hunsazk hunsazk 98 Jan 20 17:56 mailer.ex
-rw-r–r–. 1 hunsazk hunsazk 442 Jan 20 18:03 repo.ex
-rw-r–r–. 1 hunsazk hunsazk 297 Jan 20 18:03 secrets.ex

@zachdaniel , I have no idea why but I think it thinks the file does not exist.

Is that source check checking the igniter itself for that source? Or is it checking the actual file system?

Can you print out the sources that it thinks it has?

This is my naive attempt (you can mock, laugh, or help guide me, but i’m good with any of them :

debug_rewrite = fn ->
  # Create igniter instance with the install task
  igniter = Igniter.new()
  |> Igniter.compose_task(Mix.Task.get("igniter.install"), [
    "ash_phoenix",
    "ash_json_api", 
    "ash_postgres",
    "ash_authentication",
    "ash_authentication_phoenix",
    "ash_csv",
    "ash_state_machine",
    "ash_paper_trail",
    "cloak",
    "ash_cloak"
  ])

  # Get current directory
  current_dir = File.cwd!()
  IO.puts("\n=== Current Directory ===")
  IO.puts(current_dir)

  # Inspect the initial igniter state
  IO.puts("\n=== Initial Igniter State ===")
  IO.inspect(igniter, label: "Initial igniter", pretty: true)

  # Check Rewrite state before any operations
  IO.puts("\n=== Initial Rewrite State ===")
  IO.inspect(igniter.rewrite, label: "Initial rewrite state", pretty: true)

  # Try to get source for application.ex
  app_path = Path.join([current_dir, "lib", "hillbilly_millionaire", "application.ex"])
  IO.puts("\n=== Attempting to get source for application.ex ===")
  try do
    source = Rewrite.source!(igniter.rewrite, app_path)
    IO.inspect(source, label: "Source for application.ex", pretty: true)
    
    # Get file contents
    file = File.read!(app_path)
    igniter_original_contents = Rewrite.Source.get(source, :content, 1)
    
    IO.puts("\n=== Path Comparison ===")
    IO.inspect(Rewrite.Source.get(source, :path, 1), label: "Source path v1")
    IO.inspect(Rewrite.Source.get(source, :path), label: "Source path current")
    
    IO.puts("\n=== Hash Comparison ===")
    IO.inspect(source.hash, label: "Source hash")
    IO.inspect(:erlang.phash2({app_path, file}), label: "Current file hash")
    IO.inspect(:erlang.phash2({app_path, igniter_original_contents}), label: "Original content hash")
    
    IO.puts("\n=== Content Comparison ===")
    IO.puts("Original content:")
    IO.puts(igniter_original_contents)
    IO.puts("\nCurrent file content:")
    IO.puts(file)
  rescue
    e -> 
      IO.puts("Error getting source: #{inspect(e)}")
      IO.inspect(igniter.rewrite.sources, label: "All sources in rewrite state", pretty: true)
  end

  # Return the final igniter state
  IO.puts("\n=== Final Igniter State ===")
  IO.inspect(igniter, limit: :infinity, pretty: true)
  
  # Return the igniter for further inspection
  igniter
end

and get the following:

iex(2)> debug_rewrite.()

=== Current Directory ===
/home/hunsazk/Documents/Github/hillbilly_millionaire

=== Initial Igniter State ===
Initial igniter: #Igniter<rewrite: #Rewrite<1 source(s)>>

=== Initial Rewrite State ===
Initial rewrite state: #Rewrite<1 source(s)>

=== Attempting to get source for application.ex ===
Error getting source: %Rewrite.Error{reason: :nosource, path: “/home/hunsazk/Documents/Github/hillbilly_millionaire/lib/hillbilly_millionaire/application.ex”, missing_paths: nil, duplicated_paths: nil, message: nil}
All sources in rewrite state: %{“.igniter.exs” => #Rewrite.Source<.igniter.exs>}

=== Final Igniter State ===
#Igniter<rewrite: #Rewrite<1 source(s)>>
#Igniter<rewrite: #Rewrite<1 source(s)>>
iex(3)> result = debug_rewrite.()

=== Current Directory ===
/home/hunsazk/Documents/Github/hillbilly_millionaire

=== Initial Igniter State ===
Initial igniter: #Igniter<rewrite: #Rewrite<1 source(s)>>

=== Initial Rewrite State ===
Initial rewrite state: #Rewrite<1 source(s)>

=== Attempting to get source for application.ex ===
Error getting source: %Rewrite.Error{reason: :nosource, path: “/home/hunsazk/Documents/Github/hillbilly_millionaire/lib/hillbilly_millionaire/application.ex”, missing_paths: nil, duplicated_paths: nil, message: nil}
All sources in rewrite state: %{“.igniter.exs” => #Rewrite.Source<.igniter.exs>}

=== Final Igniter State ===
#Igniter<rewrite: #Rewrite<1 source(s)>>
#Igniter<rewrite: #Rewrite<1 source(s)>>
iex(4)> IO.inspect(result.rewrite.sources, label: “All sources”, limit: :infinity, pretty: true)
All sources: %{“.igniter.exs” => #Rewrite.Source<.igniter.exs>}
%{“.igniter.exs” => #Rewrite.Source<.igniter.exs>}

contents of .igniter.exs are :

This is a configuration file for igniter.

For option documentation, see Igniter.Project.IgniterConfig — igniter v0.5.12

To keep it up to date, use mix igniter.setup

[
module_location: :outside_matching_folder,
extensions: [{Igniter.Extensions.Phoenix, }],
source_folders: [“lib”, “test/support”],
dont_move_files: [~r"lib/mix"]
]

No mocking here, you’re being ridiculously helpful :bowing_man:

If you don’t do any of that, does your call to compose_task fail with the error about a changed file? If so that’s good news :smile: I’m not sure if compositing the igniter.install task is actually doing anything :sweat: . You may need to call into Igniter.Util.Install with a list of tasks. Kinds of throws a wrench into things, but the igniter install task is not, itself, an igniter task that composes :smile:

My suspicion is that the only way this will work is when run as the mix task. So that would involve creating a new project, and using {:igniter, path: "../path/to/igniter"}, then having your debug code placed into your local copy of igniter, and then running mix igniter.install ...

Its late here, so I’m going to hit the sack, but the next step may be for someone who is encountering this bug to video call me and let me drive on their computer to fix this :laughing:

Im working on that and will paste results here but been weird issues where it isnt hitting code blocks id expect

1 Like

ok @zachdaniel I’ve debugged everything using the local path reference as you suggested and I think this may be a race condition on file access? I still don’t know how or can’t explain it, but when I added a debug function to each of these everything just worked as you can see at the pastebin issue at the end. After examining the code (pretentious of me and i’m sure you know better) it seems like potentially one of two things 1. A race condition where the file is being accessed before being closed, adding the private function slowed it down enough not to be an issue? I don’t think that is the case because of immutability and not seeing concurrent tasks anyways, 2. the other option is you can see I had to clean the deps cache a couple of times because I tagged a new version locally with the private function and wanted to make sure the local version was being used, so it could be something to do with the cache not getting cleaned all the way or something? but here are the outputs from the debugging and sorry I couldn’t quite pin it down all the way:

what if I were to tell you that it was a space in the name with a certain length? When I do

sh <(curl 'https://ash-hq.org/new/hairbnb?install=phoenix') \
    && cd hairbnb \
      && mix igniter.install \
    ash_phoenix ash_json_api ash_postgres ash_authentication \
    ash_authentication_phoenix ash_admin ash_state_machine \
    ash_paper_trail cloak ash_cloak \
    --auth-strategy password \
      --yes

The above works, but the below will fail.

sh <(curl 'https://ash-hq.org/new/some_long_name_with_spaces?install=phoenix') \
    && cd some_long_name_with_spaces \
      && mix igniter.install \
    ash_phoenix ash_json_api ash_postgres ash_authentication \
    ash_authentication_phoenix ash_admin ash_state_machine ash_paper_trail cloak \
    ash_cloak \
    --auth-strategy password \
      --yes

I get the error that

Issues:

  • ** (Rewrite.SourceError) could not write to file “lib/some_long_name_with_spaces/application.ex”: file changed since reading

i’ll pull that down, connect, and get you a copy of that debug.

here is the pastebin for the one that gives an issue

here was the debug_state function I called:

defp debug_state(label, igniter, extra \\ nil) do
    IO.puts("\n=== DEBUG: #{label} ===")

    sources = igniter.rewrite.sources
    IO.puts("Sources count: #{map_size(sources)}")

    if extra do
      IO.puts("\nExtra Info:")
      IO.inspect(extra, label: "Extra data", pretty: true)
    end

    Enum.each(sources, fn {path, source} ->
      IO.puts("\nSource: #{path}")
      IO.inspect(source.hash, label: "Hash")

      try do
        v1 = Rewrite.Source.get(source, :content, 1)
        current = Rewrite.Source.get(source, :content)
        IO.puts("Has v1?: #{v1 != nil}")
        IO.puts("Has current?: #{current != nil}")

        if v1 && current do
          IO.puts("Changed?: #{v1 != current}")
        end
      rescue
        _ -> IO.puts("Error accessing content")
      end
    end)

    IO.puts("=== End #{label} ===\n")
    # Important: return the igniter
    igniter
  end

YESSSS you did it! Thank you :heart: The “other OSes” problem was a total red herring. I can reproduce with a long name :fire: Thank you so much.

1 Like

When you find out why im dying to know lol

1 Like

will definitely report back

Wowee, that was a rabbit hole :laughing: Fixed versions are released. Here is what was happening:

Duplicate files were being added into the rewrite struct, due to how in some places (when adding a glob), we were adding files by absolute path, instead of relative to CWD path.

The kicker is how we were alerted to this problem :sweat: The reason that it only happened when the app name was longer is because rewrite uses Task.async_stream to write files in parallel. Inside of that stream each thing reads its original contents and then writes it if it hasn’t changed. The slight additional extra time that it took to either read the file or hash the file name (not sure which) for the longer file names, changed the performance of the writing enough that one of the tasks was slow enough to see the new file contents from one of the other tasks completing and writing its file contents. Now there are no duplicates because everything is using relative paths.

This could also have been the source of other subtle bugs :sob:

Thank you for your help in figuring this one out!

4 Likes