How to set Bakeware env vars on GitHub Actions for a Windows release?

[Fl4m3Ph03n1x]

Background

I am trying to get a Github Action working with Windows and Bakeware because I am trying to create a release using it.

However, I am having issues with the environment variables.

Code

In Bakeware’s setup page it is mentioned that we have to set the MAKE and CC environment variables:

• GitHub - bake-bake-bake/bakeware: Compile Elixir applications into single, easily distributed executable binaries

In my Github Action, that is exactly what I do (I think):

name: build

env:
  MIX_ENV: test
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  build:
    name: Build on Windows
    runs-on: windows-2019
    env: 
        CC: gcc
        MAKE: make

    steps:
    - uses: actions/checkout@v2
    - uses: erlef/setup-beam@v1
      with:
        elixir-version: '1.13.x' # Define the elixir version [required]
        otp-version: '24.2.x' # Define the OTP version [required]
    
    - name: Install choco
      shell: powershell
      run: |
        Set-ExecutionPolicy -ExecutionPolicy Bypass
        Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

    - name: Install bakeware dependencies
      shell: powershell
      run: choco install -y zstandard make mingw

    - name: Install Dependencies
      shell: powershell
      run: mix deps.get
 
    - name: Run credo code analyser
      shell: powershell
      run: mix credo --strict

I am even using powershell to do it (even though I am not really sure if this is needed).

Problem

However my GitHub Actions code comes back with this error:

==> bakeware

mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj"

mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"

mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"

mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory

make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1

could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"

** (Mix) Could not compile with "make" (exit status: 2).

It says it cannot compile with make.

Question

I have tried copy/pasting the section:

env: 
  CC: gcc
  MAKE: make

To every section in that file I could think of, but I always end up with the same issue.

What am I doing wrong?

[cmo]

I wonder if it’s because mkdir is not creating the parents of that last path. You could try a different shell…?

Edit: I forgot I added tests to the bakeware CI. Try copy what’s in here:

bake-bake-bake/bakeware/blob/main/.github/workflows/main.yml

[Fl4m3Ph03n1x]

I tested this script and I got the same error:

 ==> bakeware
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress"
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
mingw32-make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "make" (exit status: 2).

This is my tweaked file:

name: CI

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

on: [pull_request, push]

jobs:
  linux:
    name: Test on Ubuntu (Elixir ${{ matrix.elixir_version }}, OTP ${{ matrix.otp_version }})
    runs-on: ubuntu-latest

    strategy:
      matrix:
        elixir_version: ["1.13.1", "1.13.2"]
        otp_version: ["24.2"]
        exclude:
          - elixir_version: "1.13.x"
            otp_version: "24.2.x"

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Elixir
        uses: erlef/setup-beam@v1
        with:
          elixir-version: ${{ matrix.elixir_version }}
          otp-version: ${{ matrix.otp_version }}
      - name: Restore deps and _build
        uses: actions/cache@v2
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-mix-${{ matrix.elixir_version }}-${{ matrix.otp_version }}-${{ hashFiles(format('{0}{1}', github.workspace, '/mix.lock')) }}
      - name: Restore plts
        uses: actions/cache@v2
        with:
          path: priv/plts
          key: ${{ runner.os }}-dialyzer-${{ matrix.elixir_version }}-${{ matrix.otp_version }}-${{ hashFiles(format('{0}{1}', github.workspace, '/mix.lock')) }}
      - run: mix deps.get
      - run: MIX_ENV=test mix compile --warnings-as-errors
      # - run: mix test
      - name: Extra checks
        run: |
          mix dialyzer --halt-exit-status

  windows:
    name: Test on Windows
    runs-on: windows-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Cache scoop downloads
        uses: actions/cache@v2
        with:
          path: C:\Users\runneradmin\scoop\cache
          key: ${{ runner.os }}-scoop-${{ github.sha }}
          restore-keys: ${{ runner.os }}-scoop-
      - name: Setup scoop
        uses: MinoruSekine/setup-scoop@v1
      - name: Install erlang, elixir and make
        run: |
          scoop install erlang 
          scoop install elixir 
          echo $env:USERPROFILE\scoop\apps\erlang\current\bin | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
          echo $env:USERPROFILE\scoop\apps\elixir\current\bin | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get

      - name: Build (Command Prompt)
        run: mix compile --warnings-as-errors
        shell: cmd
        env:
          MAKE: make
          CC: gcc
      - name: Test (Command Prompt)
        run: mix test
        shell: cmd
        env:
          MAKE: make
          CC: gcc

      - name: Clean up test files
        run: Remove-Item 'tmp' -Recurse -Force
      - run: mix clean --deps
        env:
          MAKE: make
          CC: gcc

      - name: Compile (PowerShell)
        run: mix compile --warnings-as-errors
        env:
          MAKE: make
          CC: gcc
      - name: Test (PowerShell)
        run: mix test
        env:
          MAKE: make
          CC: gcc

Bakeware environment vars keep being the problem for the windows setup.

[cmo]

Why do you think it’s the env vars? The error states it is mkdir that is failing. You’ll see that the Makefile tries to handle mkdir differently for Windows.

Could you try to isolate whether it is the CMD run or PowerShell run that is failing, or both?

[Fl4m3Ph03n1x]

I have tried to install choco with powershell and then run the commands using cmd. The result is still the same.
This is likely the most vanilla action I can think of, I dont get why something as simple as mkdir would not work.

This Action:

name: build

env:
  MIX_ENV: test
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  build:
    name: Build on Windows
    runs-on: windows-2019

    steps:
    - uses: actions/checkout@v2
    - uses: erlef/setup-beam@v1
      with:
        elixir-version: '1.13.x' # Define the elixir version [required]
        otp-version: '24.2.x' # Define the OTP version [required]
    
    - name: Install choco
      shell: powershell
      run: |
        Set-ExecutionPolicy -ExecutionPolicy Bypass
        Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

    - name: Install bakeware dependencies
      shell: powershell
      run: choco install -y zstandard make

    - name: Install Dependencies
      shell: cmd
      run: mix deps.get
      env: 
        CC: C:\ProgramData\Chocolatey\bin\gcc.exe
        MAKE: C:\ProgramData\Chocolatey\bin\make.exe

    - name: Compile everything
      shell: cmd
      run: |
        mix deps.compile --all
        mix compile
      env: 
        CC: C:\ProgramData\Chocolatey\bin\gcc.exe
        MAKE: C:\ProgramData\Chocolatey\bin\make.exe
    
    - name: Run credo code analyser
      shell: cmd
      run: mix credo --strict
      env: 
        CC: C:\ProgramData\Chocolatey\bin\gcc.exe
        MAKE: C:\ProgramData\Chocolatey\bin\make.exe

Results in this error:

==> bakeware
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1
Error: Process completed with exit code 1.

[Fl4m3Ph03n1x]

By changing the GitHub Action a little I was able to inspect some behaviour:

    - name: Compile everything
      shell: powershell
      run: |
        mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj" 
        mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/launcher"
        mkdir "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
        ls .\_build\test\lib\bakeware\obj\zstd\lib\decompress
        Test-Path -Path "d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress"
        mix deps.compile --all
        mix compile
      env: 
        CC: C:\ProgramData\Chocolatey\bin\gcc.exe
        MAKE: C:\ProgramData\Chocolatey\bin\make.exe
  

In this snippet I manually create the needed folders and then I assert on them. The output is as follows:

    Directory: D:\a\market_manager\market_manager\_build\test\lib\bakeware


Mode                LastWriteTime         Length Name                                                                  
----                -------------         ------ ----                                                                  
d-----         2/2/2022  11:06 AM                obj                                                                   
d-----         2/2/2022  11:06 AM                launcher                                                              


    Directory: D:\a\market_manager\market_manager\_build\test\lib\bakeware\obj\zstd\lib


Mode                LastWriteTime         Length Name                                                                  
----                -------------         ------ ----                                                                  
d-----         2/2/2022  11:06 AM                decompress                                                            
True

We can see that:

• All directories are created
• Test-Path returns True, which means that 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress' does exist.

This only makes things more confusing to me as the error has still not changed:

mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/test/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "C:\ProgramData\Chocolatey\bin\make.exe" (exit status: 2).

I really can’t understand the error no such file of directory when every command I used shows beyond reasonable doubt that the folder does exist.

Any ideas or suggestions for investigation would be welcome :S

[Fl4m3Ph03n1x]

To make it clear, using the exact same github action from bakeware (I removed the macos part which I dont need):

name: CI

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

on: [pull_request, push]

jobs:
  linux:
    name: Test on Ubuntu (Elixir ${{ matrix.elixir_version }}, OTP ${{ matrix.otp_version }})
    runs-on: ubuntu-latest

    strategy:
      matrix:
        elixir_version: ['1.10.4', '1.11.4', '1.12.3']
        otp_version: ['23.3.4.7', '24.1']
        exclude:
          - elixir_version: '1.10.4'
            otp_version: '24.1'

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Elixir
        uses: erlef/setup-beam@v1
        with:
          elixir-version: ${{ matrix.elixir_version }}
          otp-version: ${{ matrix.otp_version }}
      - name: Restore deps and _build
        uses: actions/cache@v2
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-mix-${{ matrix.elixir_version }}-${{ matrix.otp_version }}-${{ hashFiles(format('{0}{1}', github.workspace, '/mix.lock')) }}
      - name: Restore plts
        uses: actions/cache@v2
        with:
          path: priv/plts
          key: ${{ runner.os }}-dialyzer-${{ matrix.elixir_version }}-${{ matrix.otp_version }}-${{ hashFiles(format('{0}{1}', github.workspace, '/mix.lock')) }}
      - run: mix deps.get
      - run: MIX_ENV=test mix compile --warnings-as-errors
      - run: mix test
      - name: Extra checks
        if: ${{ contains(matrix.elixir_version, '1.12') }}
        run: |
          mix format --check-formatted
          mix dialyzer --halt-exit-status


  windows:
    name: Test on Windows
    runs-on: windows-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Cache scoop downloads
        uses: actions/cache@v2
        with:
          path: C:\Users\runneradmin\scoop\cache
          key: ${{ runner.os }}-scoop-${{ github.sha }}
          restore-keys: ${{ runner.os }}-scoop-
      - name: Setup scoop
        uses: MinoruSekine/setup-scoop@v1
      - name: Install erlang, elixir and make
        run: |
          scoop install erlang 
          scoop install elixir 
          echo $env:USERPROFILE\scoop\apps\erlang\current\bin | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
          echo $env:USERPROFILE\scoop\apps\elixir\current\bin | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get

      - name: Build (Command Prompt)
        run: mix compile --warnings-as-errors
        shell: cmd
        env:
          MAKE: make
          CC: gcc
      - name: Test (Command Prompt)
        run: mix test
        shell: cmd
        env:
          MAKE: make
          CC: gcc

      - name: Clean up test files
        run: Remove-Item 'tmp' -Recurse -Force
      - run: mix clean --deps
        env:
          MAKE: make
          CC: gcc

      - name: Compile (PowerShell)
        run: mix compile --warnings-as-errors
        env:
          MAKE: make
          CC: gcc
      - name: Test (PowerShell)
        run: mix test
        env:
          MAKE: make
          CC: gcc

      - name: Setup MSYS2
        uses: msys2/setup-msys2@v2
        with:
          path-type: inherit
      - name: Compile and test (MSYS2)
        env:
          MAKE: make
          CC: gcc
        shell: msys2 {0}
        run: |
          rm -rf ./tmp
          mix clean --deps
          mix deps.get
          mix compile --warnings-as-errors
          mix test

Still fails with the same error:

==> bakeware
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj"
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/launcher"
mkdir "d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress"
mkdir: cannot create directory 'd:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress': No such file or directory
mingw32-make: *** [Makefile:70: d:/a/market_manager/market_manager/_build/dev/lib/bakeware/obj/zstd/lib/decompress] Error 1
could not compile dependency :bakeware, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile bakeware", update it with "mix deps.update bakeware" or clean it with "mix deps.clean bakeware"
** (Mix) Could not compile with "make" (exit status: 2).

I wonder if I need some special configuration for my Github Workers that maybe I am not aware of?

[Fl4m3Ph03n1x]

I created a MWE called bake_test:

• first commit · Fl4m3Ph03n1x/bake_test@1e90bb7 · GitHub

In this minimal umbrella app, the github action still fails for windows and Ubuntu.
I am using the exact same file for the github workflow used by bakeware.

I now believe one of 2 things:

• my mix.exs file has issues I am not aware of
• bakeware has an issue

I am trying to figure out what is going on, but I am having trouble.

[Fl4m3Ph03n1x]

I think I finally figured it out. To my understanding, the issue was in the Makefile.
I have submitted a PR that fixes the issue:

• Fix mkdir #128

Basically, mkdir was not behaving as expected between cmd and powershell.
I am now awaiting for approval of the authors.
Furthermore, I also updated my MWE with the fork in my account so anyone can see it working.