Nemerosa

Auto versioning on promotion

Damien Coraboeuf -

Whenever you have components which depend on others, there are many ways to upgrade them. This could be as simple as holding a version string in a file. This would typically the case in GitOps driven deployments.

So how do you upgrade these versions?

Easy right? You open a pull request upgrading the version of the module into the parent, merge it and you're done.

Some solutions, some issues

Still pretty manual though, can we do better? Sure, you can now get the latest version module and create, and merge, the PR automatically.

But this raises several questions:

  • how do we detect when there is a latest version?
  • what is even a latest version?
  • how do we know this upgrade won't break the parent?

This last question is fairly easy to solve: we can still rely on the PR build to check for non regressions. Of course, if we don't want to waste too much time, we'll a do bare minimum testing and do the bulk of the testing in the parent pipelines, in a true CI spirit.

If the module itself is built the same way, with a short pipeline producing an artifact and some longer pipelines giving additional confidence, the latest version may not be the best version:

The latest version is not necessarily the best

And now, the first question remains: how do we know when it's time to launch an upgrade of the now fully qualified module into a parent?

Here comes Ontrack

We now call Ontrack to the rescue and its powerful concept of promotion.

As a reminder, if you're new to this blog, Ontrack is there to collect data about the events in your CI/CD pipelines, and in turn use them to drive automation even further.

Ontrack starts by collecting data from the source code to its way to any kind of artifact manager:

Ontrack will allow you to have a view on the lifecycle of all your pipelines, in one page:

In short, every build passes through validations which represent the different stages of your build, performance, test, quality, etc. pipelines. You then associate a set of validations to a promotion. For example, we may decide that when a build has passed through all the automated functional tests, it becomes promoted to bronze. This is in reality a vocabulary trick which allows to convene a sense of the overall quality of your deliveries across your whole organization. The names of these promotions do not really matter as long as the whole organization agrees on them.

Pipeline after pipeline, validation after validation, you start adding more and more promotions to your builds (which look more and more like actual release candidates).

Now, given this notion of promotion, we start to have the proper tools to drive the auto versioning we are trying to achieve.

Auto versioning requirements

Let's start by given some scope to our auto versioning:

  • Only dependencies between application modules or GitOps deployment manifests
  • “Normal” libraries are out of scope and simple dependency ranges and locks can be used

The last point is important. While the Ontrack "auto versioning on promotion", as exposed below, can be used in many circumstances, I would not use it in contexts where simple dependency locks mechanisms (like Gradle, NPM, etc.) can be used instead. The "auto versioning on promotion" will only shine whenever you have high level dependencies (like between a modular monolith and its dependencies, like in a GitOps environment, etc.).

Finally, I'd have requirements about what auto versioning is supposed to achieve:

  • when a dependency has a new version, we want its version in the parent to be automatically changed. That's basically the definition of the auto versioning.
  • loose coupling – we don’t want to drive the update of the parent from the module dependencies. Because that'd be a natural way to do this, wouldn't it? From the module pipeline(s), we drive the update into the parent. While in many cases that could be enough, there are also cases where a module may be used in several contexts, leading to entangled pipelines. Inversion of coupling solves this issue.
  • only dependencies of a given quality level are eligible for being used. That's where the notion of promotion intervenes. In real-life long and delayed quality pipelines, the latest version is not necessarily the best version.
  • upgrades must be controlled before being accepted. We want to rely on the PR pipelines to check for breaking changes. This is only a last minute sanity check: to really check for conformity, you'd better rely on contract testing on both module & parent sides.

Auto versioning on promotion - an overview

How does it work in Ontrack?

(0) The parent is ultimately responsible for defining its list of dependencies and its pipeline (as code) will register them in Ontrack. When I say "list of dependencies", it looks something like: I need to use the latest bronze promotion of the 4.* versions of this dependency.

(1) at one point, the module dependency is promoted into Ontrack. This triggers (2) a chain of events which in the end creates (3) a pull request in the parent's repository.

(4) the pull request is automatically controlled (business as usual) by its pipeline and upon success (5), this pull request is merged automatically, leading to an effective upgrade of the module into the parent.

Having done this, we have met our requirements:

  • Automation of upgrades
  • Only the parent knows about its modules (loose coupling)
  • Control of the quality of the module through promotion levels
  • Control of the quality of the upgrade through pull requests & associated builds

Some tweaks

This is the general picture and there are many options available to tweak this process for your needs:

  • pull requests may run in client mode (where Ontrack takes full control of the approval & merge process) or SCM mode (where Ontrack just approves the PR and tells the SCM to set the PR in "auto merge" mode - a kind of "fire and forget" approach).
  • in some cases, changing a version into a file of the parent is not enough and some kind of post processing is needed (like resolving dependency locks for example). Ontrack can delegate this post processing to CI engines like Jenkins or GitHub Actions, but this is still still configured as code at the parent level.
  • pull requests auto merge can be disabled so that manual approvals remain necessary (if you need formal approvals before going to production for example).

Supported files

We said that this process would update some files in the parent's repository with the version of the dependency. But which kind of file? Ontrack "auto versioning on promotion" works with the following formats (and please create an issue if you need more):

  • any text files using regular expressions
  • properties files
  • NPM package.json files
  • Maven pom.xml files
  • YAML files (using Spring EL expressions)

Finally, a bit of code

That's all very nice, but how do you configure all of this, really?

First of all, there is an official documentation.

But in the end, from a developer point of view, it's all very simple. Your Jenkins pipeline or GitHub Actions workflow will ultimately use an "auto versioning descritor" like an auto-versioning.yml file stored in the parent's repository.

It looks like:

dependencies:
  - project: "module"
    branch: "release/1\..*"
    promotion: "GOLD"
    path: "package.json"
    property: "@my/module"
    propertyType: "npm"

In this example, we declare that any version of the module dependency on the release/1* branches reaching the GOLD promotion will be upgraded in the parent's package.json file in its @my/module dependency.

And, from a developer point of view, that's really it. This file will be ingested by Ontrack which will set the whole auto versioning mechanism in place.

Seeing is believing

Whenever auto versioning is put in place, some links between the different Ontrack builds are automatically created internally and can be visualized at dependency and parent level.

This, in turn, allows Ontrack to build downstream & upstream dependency graphs, which show, in real time, how your dependencies are evolving.

Here is a concrete example showing the deployment of Ontrack in its Saas offering:

We see here some different components of the deployments, including one called ontrack-pro-portal where its auto versioning is marked as being errored.

BTW I saw that while writing this article, so I'll need to stop and check - see below.

Every auto versioning event is recorded in Ontrack in an audit log which you can access from the UI (and the API, of course):

Now, I can start to understand and fix the issue.

Conclusion

The whole idea of "auto versioning on promotion" is to automate the painful processes of upgrading your dependencies, without sacrificying the quality:

This can be done by declaring dependencies requirements at parent level, by making sure the dependencies are promoted to the expected levels of quality and to leave Ontrack do the rest.

For more information, you can consult the Ontrack documentation or consult me for questions or subscribing to Ontrack Saas in Slack.

https://nemerosa.github.io/ontrack/