SemanticReleaseHex - Fully automated version management and release processes

sheerlox · November 20, 2023, 1:10am

Hello, Elixir community!

I’m excited to introduce a new tool that aims to streamline version management and release processes for Elixir projects: semantic-release-hex.

Disclaimers

Before we dive in, I want to clarify that semantic-release-hex is a JavaScript library designed to enhance version management and release processes in Elixir projects. While it’s not an Elixir library per se, its integration with Elixir projects can bring the power and simplicity of semantic-release to your workflow.

I also want to mention that although it’s only been two weeks, I’ve recently been invited to the semantic-release organization. The reason I’m developing this library is unrelated to my membership in the organization and is solely motivated by my complete trust and utter admiration for that project, which I’ve been using for more than 6 years in all of my projects, both personal and professional.*

Why?

In the course of developing a SaaS project with Elixir & Phoenix, I encountered the challenge of automating version management while adhering to best practices. Initially being a JavaScript developer, I turned to the reliable semantic-release tool but found it lacking in Elixir integration. This led me to create semantic-release-hex.

semantic-release features

Determining the next version number
Generating release notes & updating the changelog file
Creating a git tag
Publishing GitHub/GitLab releases
Commenting on released Pull Requests/Issues

semantic-release-hex features

Updating the version in mix.exs

semantic-release-hex roadmap

Updating the version in README
Updating the version in Mix.install/2 scripts
Distributed as a hex package (similar to tailwind)
Publishing to Hex

Benefits

Fully automated release
Enforce Semantic Versioning specification
New features and fixes are immediately available to users
Notify maintainers and users of new releases
Use formalized commit message convention to document changes in the codebase
Integrate with your continuous integration workflow
Avoid potential errors associated with manual releases
Simple and reusable configuration via shareable configurations

Trying it out

I’ve set up a demo repository to showcase what that would look like in an Elixir project: semantic-release-hex-demo. Feel free to explore its content, commit history, or even fork it to test things out, as it’s what it’s meant for.

You will also find detailed instructions on how to set it up in your own Elixir project on the main repository.

I’m eager to hear your thoughts, answer any questions, and discuss how semantic-release-hex can enhance your Elixir projects!

Also, if you’d like to share your views on features currently being specified/implemented, don’t hesitate to visit the Discussions/Issues for feature requests.

Thank you for reading

Acknowledgements

Special thanks to @Eiji and @D4no0 for their invaluable feedback!

Eiji · November 20, 2023, 4:53am

Isn’t it easier to create a mix task which firstly guess based on configurable prefix (let’s say: "fix: ") and then asks for a type of version change giving the guessed one as default (to just press enter)?

config ::semantic_release, :prefix,
  major: ["…"],
  minor: ["…"],
  major: ["fix: ", "other: "],

Some projects would need to add node as extra environment dependency which is not always a good idea. If we are going to simplify one thing we shouldn’t complicate other one. For me it’s simple on my laptop, because I use asdf, but not everyone is in such case. Also obviously the documentation is not on hex with all of it’s side-effects which may be confusing especially for new developers.

$ mix semantic.release --log-level debug
[debug] Found "…" prefix in commit message which is configured for … version change
[debug] Found (…)
(…)
[info] The determined version change is …
Which version change should be applied?
1) Major
2) Minor
3) Bug fix (default)
Choose type of version change [1/2/3] _
[debug] The chosen type is …
[info] The new version is …
Are you ready to push changes [Y/n] _
[debug] Pushes commit(s)
Do you want to create a new git tag "v1.x.y"? [Y/n] _
[debug] Created git tag "v1.x.y"
Do you want to publish a release to hex? [Y/n] _
[debug] The release has been published
[debug] The documentation has been published
The version has been successfully changed from … to …

$ mix semantic.release --yes-all
Which version change should be applied?
1) Major
2) Minor (default)
3) Bug fix
Choose type of version change [1/2/3] _
The version has been successfully changed from … to …

$ mix semantic.release --yes-all --default
The version has been successfully changed from … to …

Does it have any strict format or it’s configurable somehow?

Does it creates a git tags automatically as well (git checkout v1.x.y or so)?

What is the comment content? Is it configurable? How about wiki?

For me the best documentation is:

wiki as user perspective
issue as developer perspective
pull request as implementation details

How about Installation in README.md file? It should be easy to find in markdown as long as you know the project name and previous version.

Those links are relative, so browser searches it on this forum …

sheerlox · November 20, 2023, 1:38pm

Isn’t it easier to create a mix task which firstly guess based on configurable prefix (let’s say: "fix: ") and then asks for a type of version change giving the guessed one as default (to just press enter)?

I thought about making an actual Elixir library to solve the issue at hand.

However, semantic-release is the industry standard battle-tested tool (was created almost 10 years ago and is the 128th most depended upon package on NPM, ahead of jquery, underscore, babel and lots of other staples), packs in a lot of useful features (e.g. support for pre-release / maintenance branches) and is highly configurable to suit almost any workflow.

For these reasons, while it would be extremely instructive to try and create an Elixir equivalent, I don’t think reinventing the wheel is a good use of time, especially when there is virtually no downside to using (and extending) the already available tooling.

Some projects would need to add node as extra environment dependency which is not always a good idea.

Technically, because semantic-release is made with a CI-first approach in mind, it doesn’t have to ever be run locally. Because of this, the Node.js environment dependency is only in the release workflow file. Node.js is required locally only when initializing the tool, to install the packages and run the dry-run to check everything is working as expected.

Also obviously the documentation is not on hex with all of it’s side-effects which may be confusing especially for new developers.

I’m not sure what you mean by “side-effects”, but I don’t think not having the documentation on hex is an issue, since this isn’t an Elixir library per see, and the only place it will be referenced is in the release configuration. Similar to CI tools, where you wouldn’t expect GitHub Actions documentation to be on Hex, the relevant documentation is readily available where it’s most relevant.

Does it have any strict format or it’s configurable somehow?

This is fully configurable by choosing an existing conventional changelog preset, or creating your own by forking one of the existing presets and making adjustments as I did (to be exact it is a maintained fork) with @insurgent/conventional-changelog-preset (which includes all commit types and adds emojis before categories). You can take a look at the relevant source code in lib/commit-transform.js. But note that with all the available presets, you probably don’t have to create a new one.

Does it creates a git tags automatically as well (git checkout v1.x.y or so)?

Yes, I forgot to mention that in the post, but semantic-release tags the version to the release commit before creating a GitHub/GitLab release, as it actually relies heavily on tags (e.g. to determine the current version).

What is the comment content? Is it configurable? How about wiki ?

You can take a look at an example comment here: semantic-release/semantic-release#3062 (comment).

It is also fully configurable, please refer to the semantic-release/github options documentation if you’d like to know more.

Regarding the wiki, I’m not sure what you mean by that. Could you please expand on the use case(s) you have in mind?

How about Installation in README.md file? It should be easy to find in markdown as long as you know the project name and previous version.

That’s actually a very good idea, thank you so much! I’ll be adding it to the feature requests and will start working on it as soon as it is specified correctly (btw if you’d like to share your views on features currently being specified/implemented, don’t hesitate to visit the Discussions/Issues for feature requests).

Those links are relative, so browser searches it on this forum …

Sorry about that, thanks for pointing this out I’ll fix them immediately.

Thank you so much for responding to my post to share your views and concerns, and I hope I’ve addressed them as best as possible

D4no0 · November 20, 2023, 2:12pm

I’m also not a big fan of installing node and random JS packages either on dev or CI/CD, I am tired of redoing the CI script every few months because of some forced “deprecations” introduced in the ways to install node or other js packages.

If the implementation of the package is what matters, I think making a CLI and packaging it together with a elixir library, just like tailwind does is a much more sane approach and would make everyone happy by not caring about any implementation details.

sheerlox · November 20, 2023, 3:16pm

I’m also not a big fan of installing node and random JS packages either on dev or CI/CD, I am tired of redoing the CI script every few months because of some forced “deprecations” introduced in the ways to install node or other js packages.

Unrelated: superfluous defense of Node.js and NPM

As a JavaScript developer for more than 10 years, I’m not aware of any such “forced deprecations” requiring substantial changes to CI workflows.

For example, the official GitHub Action setup-node did not make changes to its interface between v1 and v4 (current version) other than renaming the version option to node-version in v3.0.0 (and adding a lot of non-breaking features, e.g. support for .tool-versions).

Similarly, the interface of a basic npm install command did not change between the (pre-v1) 2010 specification and today’s latest (v10) documentation. Of course, the way NPM handles the installation and optimization of packages changed a lot under the hood (thank god, because the early days were … something ), but breaking changes never were an issue, at least for me. Searching through @npm/cli releases supports those claims.

It’s also worth mentioning that Node.js releases LTS versions, which “typically guarantees that critical bugs will be fixed for a total of 30 months”.

If the implementation of the package is what matters, I think making a CLI and packaging it together with a elixir library, just like tailwind does is a much more sane approach and would make everyone happy by not caring about any implementation details.

That being said, I can understand that one might not want to install a Node.js runtime and packages in an Elixir project, even if only on the CI.

I really like your idea of packaging the library in a hex package!

Unfortunately, there’s a major difference between tailwind and semantic-release, being that the former is a frontend library (hence it runs in the browser). If we were to try and do the same thing for semantic-release, I’m afraid we’d have to bundle a Node.js runtime inside the hex package, which I don’t think would be either a practical or good idea and might even not be possible at all.

EDIT: I realized you were mentioning that Elixir’s tailwind uses the Tailwind standalone CLI under the hood, and so does not require having Node.js installed. I wasn’t aware of that, and it definitely looks very interesting, I’ll be looking further into it!

Anyway, I’ll be looking into this idea further, even if that means installing the Node.js runtime on the CI separately. That would allow to simplify the installation process and also provide a mix task for running the release. Thanks for the great suggestion!

Eiji · November 20, 2023, 3:46pm

ex_doc as well as everything in Elixir environment is extremely powerful. As long as yo have it as a dependency you can easily link to every it’s module, function and so on. No matter how good or not is it, it would never support every possible source (not only npm). Therefore the developer who want to describe everything in details in a guide would need to use absolute urls. In my personal opinion Elixir’s documentation tool output looks brillant and I’m really used to it. Suddenly changing a look like feeling of documentation by clicking on “some url” is not … handy.

Since the documentation could not be found on hex or hexdocs page and Google uses a terrible algorithms it may be hard to find such tool. How much people are searching for a package here on forum? Keep in mind there is “hex” in the project name it’s of course correct on one side, but confusing on the other one as what developer expects is not what is provided.

I remember it was something about semantic versioning … What was it’s name? Weird … there is nothing about it on hex … Did I forgot some keyword?

That would be my first thoughts …

Having a tool outside of the ecosystem is of course your decision, but it’s rather a matter of time for a side-effects to come. It’s like a butterfly effect. If some (even small) % of developers would have trouble search for it then in long term it would not give anything good.

Of course I may be wrong here, because I’m writing it from my perspective. For example in many cases I already had a desired page on some nth results page. As above I’m not searching for a specific package/tool on forum, but rather on hex/hexdocs.

I’m really not sure what to recommend here. Maybe try to go a similar route in the Phoenix assets. Instead of using npm they are downloading some tools (esbuild and tailwind) and run them as watchers. Of course watchers here does not makes sense, but I don’t know … downloading a standalone node binary (if possible) to some directory and call your code (like install_and_run in watchers config). Therefore most of your code would stay on npm and you could keep documentation including guides for Elixir stuff on hexdocs making it’s easier to find. Does it makes sense for your case?

Edit: Oh, actually @D4no0 proposed it before me.

Not really … it’s like a … web service. I mean you do not care in which language Slack API was created. The API itself is not a part of environment, but it’s a 3rd-paty service to integrate. Also if such a web service is related to language environment then we have for example a Slack API client hex dependency which instead giving tons of links to Slack API documents everything on hex. As long as it works “nobody cares” how it works i.e. it could be a REST API or GraphQL one - the hex package gives it’s own Elixir’s API.

I had not idea how your code is working, so I was wondering if there was some AI-based text generators, so actually never mind

sheerlox · November 20, 2023, 4:28pm

That all makes total sense, I now understand your point about having the documentation on hexdocs, which would be even further solidified by the creation of an Elixir package abstracting the installation and release mechanisms associated with Node.js. I really like this solution and have already started looking into it.

I’m not aware of any mention of a standalone release of semantic-release-cli, but that will be a great addition to the tool that I’m sure the core team will welcome with enthusiasm! I’ll run the idea by them while working on a prototype.

Regarding the AI-based documentation generator, semantic-release indeed does not ship with that feature, but maybe someone will create a plugin for that in the future

Thank you for being so patient with me, your input is really valuable and highly appreciated!

thiagomajesk · November 20, 2023, 4:41pm

I have used another lib called git_opts before, which is written in Elixir. Out of curiosity: did you find any specific functionality lacking in the ecosystem that prompted you to create another library?

sheerlox · November 20, 2023, 6:05pm

git_ops seems to be lacking the following:

Committing & pushing changes to the repository
Creating & pushing tags to the repository
Publishing GitHub/GitLab releases
Publishing to Hex
Commenting on released Pull Requests/Issues
CI Integration (ability to use GitHub/GitLab tokens for auth)
Support for a wide variety of workflows
Modularity (existing plugins)
Extensibility (ability to easily create a new plugin)
Customization (extensive configuration options)
Comprehensive documentation

Ultimately, its missing git & CI integrations make it unable to be fully automated easily.

I had noticed git_opts while researching the topic, but thought it was unmaintained due to misreading “Jun 21” for `“June 2021”. I now realize the package receives a bit of attention.

Though git_ops looks like a great project and would have the benefit of allowing any Elixir developer to contribute without knowing JavaScript, I think it is unfortunate that someone would invest their time in reconstructing a tool from scratch when there’s practically no drawback to utilizing and enhancing the existing tooling (as I stated before).

semantic-release is a mature and stable tool, with features that would take months (or even years) to develop in git_ops with a dedicated and active team. This is what led me to believe creating a bridge between Elixir and semantic-release to bring its capabilities to the ecosystem was the best solution.

Disclaimer: I haven’t mentioned that before as it did not seem relevant since it’s only been two weeks, but I’ve recently been invited to the semantic-release organization. The reason I’m developing this library is unrelated to my membership in the organization and is solely motivated by my complete trust and utter admiration for that tool, which I’ve been using for more than 6 years in all of my projects, both personal and professional.

sheerlox · November 20, 2023, 7:42pm

In case that’s of interest to anyone, here’s the GitHub Discussion I just created on semantic-release which will host the exchanges with the core team regarding this project:

I’ve mentioned the need for creating an official standalone executable, required to bundle everything as a hex package, and the possible approach to fulfill that requirement. I’m waiting for their preliminary feedback before beginning any work.

georgeguimaraes · November 21, 2023, 12:42pm

Thanks for creating semantic-release-hex. I think it’s a great addition to the semantic-release ecosystem.

I think that the Elixir community is not so keen on using tools like semantic-release to continuously release libs and generate CHANGELOGs. I think the majority of us are publishing our libs manually to hex.

I’ve been using semantic-release in my plugins for Logseq (the open-source outliner note-taking app) since that’s the recommended way of releasing Logseq plugins. Although I admit it was a bit confusing to set up, I’m really enjoying committing stuff using feat: new feature here and that immediately triggering a new version and release.

I thought about creating something like semantic-release for Elixir, but as you said, there’s lots of functionality. I particularly enjoy all the PR comments when you release a new version.

I’m not maintaining any Elixir lib right now, but if I do, I’ll sure use semantic-release-hex. Thanks and congrats!

sheerlox · November 21, 2023, 1:29pm

Thank you so much for your kind words, George!

I also noticed the Elixir libraries are almost all released manually, but (maybe incorrectly) attributed that to the missing tooling for reliable release automation.

I’m a tremendous advocate for automating this workflow, as I truly believe it is an excellent enabler of good practices (always having the main branch deployable, providing bug fixes to maintenance versions, consistent versioning, maintaining a changelog, etc…).

My objective is to enable project maintainers to automate their release workflows if they ever want to, and I’d be delighted if my efforts bring value to even just one project!

I’m thrilled to know you’d be a semantic-release-hex user if you’d have to maintain an Elixir lib, thanks again for your message!

sheerlox · June 15, 2024, 8:35pm

Hey there, it’s been a while I come with great news!

I made good progress on this project a while ago now but didn’t take the time to update this thread. That gave me the time to extensively test the library (for 6 months now), mainly on Talent Ideal, my first company, which uses Elixir/Phoenix.

To address @D4no0’s concerns about having to manage Node.js installation, and @Eiji’s concerns about not having the documentation hosted on HexDocs, I have developed and released the semantic_release package on Hex.pm, based on the awesome tailwindcss package made by Chris McCord!

Here’s what it does:

downloads and extracts the Node.js binary for the right platform to the _build directory (using the nodelix package I’ve developed for the occasion, cc @ryanwinchester )
installs the Semantic Release packages from npm in that same directory
provides a way to manage versions and add plugins without managing a package.json file
expose running Semantic Release through a Mix task (mix semantic_release)

All in all, this greatly reduces the overhead for using Semantic Release in an Elixir project:

add the Hex package to mix.exs
configure the dependencies with config.exs
configure the release workflow with .releaserc (will explore a way to move that to config.exs as well in the future)
add the Mix task to a CI release workflow (which ideally depends on a test workflow)

I will update my demo project in the next coming days, as well as the main post of this thread, but if anyone wants to get a sneak peek before then, here are a few links to get you on the right path (numbered based on the steps above):

There’s still a lot of room for improvement, but it’s already working great and I hope this work can be useful to the community in the long run

Eiji · June 15, 2024, 11:59pm

github.com

talent-ideal/semantic_release/blob/dev/lib/semantic_release.ex

defmodule SemanticRelease do
  @moduledoc false

  @spec install_path() :: String.t()
  def install_path() do
    Path.join([Mix.Project.build_path(), "semantic_release"])
  end

  @spec bin_path() :: String.t()
  def bin_path() do
    Path.join([install_path(), "node_modules", ".bin", "semantic-release"])
  end
end

How about making install_path configurable? I would think twice before I would touch _build from my code as there is no guarantee how Elixir would behave in next releases … That’s said maybe I’m just overthinking it.

Anyway, currently Phoenix-based backends using Node dependencies were using assets/node_modules, so there is no need to hide this directory in _build. It should not be a problem to create/edit package.json by adding dependencies, so the developer would simply call npm install as they do in their projects.

What do you think about it?

sheerlox · June 16, 2024, 10:41am

Hey Eiji, thanks for the feedback!

The _build directory seemed the most obvious place to put binaries given the citation below, and it’s also where tailwind stores its executable.

Excerpt from Phoenix’s “Directory structure” guide, which supports this decision imo:

_build - a directory created by the mix command line tool that ships as part of Elixir that holds all compilation artifacts. […] This directory must not be checked into version control and it can be removed at any time. Removing it will force Mix to rebuild your application from scratch.

That being said it could be made configurable, which would for example enable sharing the same Node.js binaries between projects.

The decision for Phoenix to store node_modules in the assets directory makes sense, as it actually contains assets used during the front-end bundling process.

Regarding semantic_release, I believe the decision to hide its node_modules behind _build makes sense, because it should only be used by itself.

Moreover, if multiple projects using nodelix install specific package versions in a shared node_modules in the root directory, that would lead to potential conflicts and unexpected behavior (without relying on a shared root package.json, which I discuss below).

Currently, the idea behind semantic_release is to abstract as much as possible the fact it’s using Node.js under the hood (thanks to nodelix). Having to maintain a package.json or run npm install in an Elixir project might be seen as an unwelcome overhead for many project maintainers.

The goal is that by simply installing semantic_release and running the Mix task without any configuration, you get the default Semantic Release experience. Then if you want a different release workflow than the default (which most likely will be the case), you can configure that through config.exs (to specify plugins to be installed) and .releaserc (to configure the workflow).

Additionally, I believe letting the end user manage the versions himself in a root package.json would limit the library’s ability to handle compatibility with the various Semantic Release major versions.

Eiji · June 16, 2024, 12:08pm

Oh, right. Yeah, that part completely makes sense. I completely forgot about binaries. Look that hex packages and their documentation are saved globally in $HOME/.hex directory, so it could be a bit confusing for me.

Then I would say follow phoenix way i.e. generate package.json as described already, but also install binaries into _build directory.

Hmm … Now I see the point. Still prefer assets way, but maybe with some changes … assets/semantic_release?

Originally I was thinking (assuming same node_modules) that developers are familiar with basic npm stuff, because as said above it’s not really something new to add Node dependencies into Phoenix projects. That’s said it may not be best for new projects … If we would follow similar things then I would look for phoenix generators which instructs developers to add routing. Here instead of routes there could be adding or modifying existing alias i.e. assets.setup. It’'s just a copy-paste and assets.setup is part of most Phoenix projects. I guess every Elixir developer which does not started learning recently would get it easily.

Is that so? Not remember now how tailwind is doing that, but by default it fetches latest stable build, right? Modifying json files today is a piece of cake, so in worst case there should not be a problem with updating package.json file or maybe I have missed something again.

Also you worry about best case (actively maintained package). What about worst case (no support)? How would developers deal with possible updates? Look what have changed over years … Previously every Phoenix project was based on Webpack and now we have a completely new way i.e. esbuild.

Summary:

_build sounds really well for binaries especially if phoenix does the same thing (previously I was focused more on assets stuff)
Does something like assets/semantic_release, semantic_release or similar makes any sense?
package.json (regardless if creating in extra directory or shared with root project) is more intuitive for developers. Especially those who are new in Elixir. It’s simpler to understand well documented mix aliases than a source code of mix task in one of possibly many project dependencies.

Well … all above is not some kind of advice, but just some loose thoughts, so no need to consider everything. After what you wrote I’m no longer convinced with both node_modules in _build, in assets (shared with root project) as same as separate directory. My intuition tells me it’s something similar, but I have no idea what it might be exactly.

Right before posting I got a weird idea … apps/my_app_semantic_release/assets/node_modules or something like that, but it looks like I’m overthinking it a bit.