Freitag, 27. September 2013

Lightweight release builds with Apache Maven

This article is a follow-up of http://diplingfh.blogspot.ch/2013/02/how-has-our-release-process-matured.html as I have enhancements to the version.override feature I presented in this article the first time.

For quick readers here a short howto:
  • Download maven-core-extensions-0.3.2.jar and store it under $M2_HOME/lib/ext
  • Configure your root pom.xml according to [1]
  • execute mvn clean install -Dversion.override
See https://github.com/rotscher/emerging for more information.

Some more explanation


It's not the main purpose to replace the maven-release-plugin. But still, for doing continuous delivery during development phase it's not convenient:

We need a build with a unique version
And I don't mean unique SNAPSHOTs as this generated version might be different for each module inside a multimodule project. And there is no 100% safety that I really reference the latest SNAPSHOT! And this timestamp-buildId construct is ok for machines but not for humans (my tester answers my "which-version-is-deployed-question" with 2.2.3-201309261313-1).

When the build- and deployment process is started a unique version is generated and used throughout the toolchain (E.g. as build parameter in Jenkins, or input parameter to a deployment tool, or mentioned in a status report).

We don't want the pom version updated and committed daily
The developers don't like that all pom files are updated and they have to build from scratch to get new SNAPSHOT artifacts into the local maven repository.

we don't need a tag daily
The unique version mentioned above (and the revision from version control, too) is stored where ever it makes sense:
  • display-name of web.xml (this is visible in Tomcat manager)
  • somewhere in the (web) application and then visible for all users
  • in a database table
  • in all MANIFEST.MF
With all that information it's easy to track down problems during the development (when done in short cycles ;-). And it's even possible to reproduce a build if necessary at all.

The build must fail fast
How many times did your release process failed because you still had configuration errors? And how good do you feel about your release when you finally fixed all that errors? I mean, you just released something different than what you tested.
This is mainly the problems of SNAPSHOT versions. With the version.override feature you detect such configuration errors very fast!

The release plugin is not handy
In the last phase of a development iteration it should be possible to deliver as fast as possible. The release plugin builds three times and commits twice: this takes time and the chance of failures is high.
How many times did you fixed the scm connection configuration in your root pom?
And ever tried to execute the release plugin in an automated way? E.g. automatically triggered at 9pm?

Final thoughts
To me, SNAPSHOT versions are ok on local developer machines but it's subobtimal when SNAPSHOTs are in a remote repository, or even on a Jenkins with shared local repository (but having private repositories is not always possible due to infrastructure limitations).
This approach here is as easy as executing normal SNAPSHOT builds (also on local developer machines!), is not changing anything in the version control (and no risk of failures with the version control), but is behaving like release:perform. And it still leaves the choice to use all options in the development process:

mvn clean install (after each commit during the day)
mvn clean install -Dversion.override (once per day)
mvn release:prepare release:perform (when day x arrives  ;-)


[1] version.override custom install plugin configuration
<profiles>
    <profile>
        <id>version.override</id>
        <activation>
            <property>
                <name>version.override</name>
            </property>
        </activation>
        <build>
            <pluginManagement>
                <plugins>
                    <!-- deactivate the default install plugin -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-install-plugin</artifactId>
                        <executions>
                            <execution>
                                <id>default-install</id>
                                <phase>none</phase>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </pluginManagement>
            <plugins>
                <!-- configure the custom install plugin for lightweight release builds -->
                <plugin>
                    <groupId>ch.rotscher.maven.plugins</groupId>
                    <artifactId>install-custom-version-plugin</artifactId>
                    <version>0.3.1</version>
                    <executions>
                        <execution>
                            <id>custom-install</id>
                            <phase>install</phase>
                            <goals>
                                <goal>install</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

Mittwoch, 20. Februar 2013

Maven - From POM to HOM...

...and other ideas for a future maven releases


Bored by all the Ant-vs.-Maven-vs.-Gradle articles (sorry for all build tools not mentioned here), I'd like to give some ideas how Maven could be improved.

Sometimes, I swear about Maven when I don't get to where I actually wanted to go to. Or, I don't like the idea to run mvn clean install and it's deploying something to a database which means it's not at all doing for what Maven is actually designed for. But as I know what Maven can and what it can't and knowing that no build tool is perfect I don't want to change it.

So, here is a list of ideas which can be summarized as follows: ideas in paragraph 1 and 2 optimize the current implementation, paragraph 3 are ideas extending Maven for supporting continuous delivery.
Before I really start:
  • I assume that projects follow the existing maven conventions, e.g. having a hiearchical folder structure.
  • There exist no proof of concept if any of this ideas can be implemented.

1 - More convention over configuration...

... which would reduce the lines of configuration

 

no explicit parent declaration

the parent is one directory above, why do I have to declare that on five lines?

no artifactId declaration

Convention for the artifactId could be the folder name, recursivly concatenated with the parent folders (e.g. parent-child, parentA-parentB-child).

no version declaration for internal dependencies

dependencies which are modules of the reactor are internal dependencies, and normally have the same version. In other words: the ${project.verison} should be declared once in the root pom of the project.

reduce the amount of pom.xml

not all modules must have a pom.xml, especially intermediate folders being itself parent of submodules dont' always need a pom.xml. The file should be optional, taking groupId and version from the parent, artifactId is based on the foldername.

2 - Make the POM more human readable...

...the current one is ok for machines

 

Why not have attributes in the POM? Instead of configuring
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.1.1</version>
        <optional>true</optional>
    </dependency>


the dependency could be configured with attributes*
    <dependency groupId="commons-logging" artifactId="commons-logging" 
                version="1.1.1" optional="true" />

*in a normal editor this would be a one liner

I would leave the current (stable) POM, but humans should configure the modules in a hom.xml, human (friendly) object model, which is tranformed to a POM at runtime. This could be easily done in a generic manner whith XSLT: all attributes are transformed to child elements.

3 - New features...

...we're not just building software, it also gets deployed!

 

version.override

It should be possible to override the version when a build is started
(e.g. mvn deploy -Dversion.override=1.2.3)
This would help creating "releases" in a lightweight manner without having any snapshots and is helpful when doing continuous delivery where a unique version is generated outside of maven and is reused for later deployments to different stages.

more lifecycles

apart from the existing build lifecycle which creates jars, ears and so on there should be more follow-up lifecycles
  • dist lifecycle, for creating a distribution
  • stage lifecycle, for deployment (not 'mvn deploy') to an environment (app server, database etc)
mvn clean install dist stage would (locally) build all modules, create a dist and deploy the dist to a (local) app server

Of course, the dist lifecycle can be integrated in the existing build lifecycle (normally with the assembly plugin) but it should already be better integrated at all. Like with the ear type: Normally, the convention is enough, no or only little additional configuration is necessary. Compared to the assembly descriptor files. Why not have a packaging type like zip/tar (and rpm, deb, exe etc.)?

That's it...

...for now


Next thing for me would be to play around with the one (hom.xml) or other idea (mvn stage) to see what is feasible.

 

 

Montag, 26. November 2012

How has our "release" process matured with Maven...

Follow-up: http://diplingfh.blogspot.ch/2013/09/lightweight-release-builds-with-apache.html

...without-using-the-maven-release-plugin?!! Below is a simplified account of the advancements in our software release process. This was in an organizationally and technically complex project. Whether all of the efforts were justified or not is only hypothetical - it was just the way it happened - and now everything is much better.

In the Beginning

The project is organized in a standard Maven structure with a CI server responsible for doing builds. In the beginning, the corresponding SNAPSHOT artifacts from the Maven repository were downloaded and manually deployed on the test systems. If any problems were found, the developers would commit the fixes, the modules built again and so on. At that time, a release was a defined list of current SNAPSHOT artifacts. At some point, the testers would ask that the current "release" be deployed to another test environment. Which version was that exactly? We're still on SNAPSHOTs and in the meantime we may have built new SNAPSHOTs! We also tried to use the Maven release plugin, but with 8 hour builds, it just wasn't practical. 

The First Improvement

So a first optimization was that any deployed release was manually copied to the file-system in a directory with a unique release number. This allowed us to re-deploy the "release" artifacts consistently. 

The Second Improvement

Then came some automation. The modules were downloaded using the Maven dependency plugin and stored away the file-system. And while we're at it: why not take the same artifacts, re-pack them (with the assembly plugin) to the appropriate deployment packages, and deploy them to the "release" repository? Said and done. To implement this, the (non-SNAPSHOT) version number must be passed from the outside with a Java system property (-D), which meant that the ${project.version} was no longer a constant and that the "release" POM looked like the following:

<groupId>groupId</groupId>
<artifactId>release-artifact</artifactId>
<version>${distribution-id}</version>
<packaging>pom</packaging>
<build>
  <plugins>
    <plugin>
      <artifactId>maven-dependency-plugin</artifactId>
      <!-- execution configuration goes here -->
      <!-- use copy goal to download all artifacts 
           being part of your release -->
    </plugin>
    <plugin>
      <groupId>ch.rotscher.maven</groupId>
      <artifactId>dependeny-attach-plugin</artifactId>
      <version>0.1.1</version>
      <!-- execution configuration goes here -->
      <!-- use copy goal to download and attach all artifacts  
           being part of your release -->
    </plugin>
    <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <!-- execution configuration goes here -->
      <!-- assembly all your artifacts to 
           a distribution (zip, ear, tar etc.) -->
    </plugin>
  </plugins>
</build>
 
With mvn clean deploy -Ddistribution-id=1.2.3-RC-5 a release is created! The dependency-attach-plugin is a small extension to have artifacts attached to the project and deployed to the "release" repository -- those that belong to the individual release but are either not packaged with the assembly plugin or those that we wanted "re-versioned" from SNAPSHOTs to releases. See [1] to get similiar information about you can set up your builds and why the maven-release-plugin is not practical.

Separation of the Build and the Release

Using the above mechanism, the software build and release creation could be separated. The software build consisted of a Maven build, which built all the artifacts with SNAPSHOT versions. This was done whenever a developer commits a change to the SCM. When the software build was successful and a release was required, the release build (using POMs like the one above) would download SNAPSHOT artifacts, and upload "release" artifacts with the version given as the "distribution-id" system property. Inside of our release artifacts there would still be SNAPSHOT artifacts (e.g., SNAPSHOT JARs inside of WARs). Is that bad? Maven purists would say "yes". However, we found that this process functioned successfully – we never had the wrong SNAPSHOT artifact in one of our deployments. Nevertheless, SNAPSHOTs still make people uncomfortable… 

Another Improvement – Good Bye Snapshots

How do we get completely away from SNAPSHOTs? No, we would not try the release plugin again. Wouldn't it be nice if the POMs could normally be configured with a constant version (e.g., a SNAPSHOT version), and this constant could be overridden at build-time by another value (i.e., a release version) without the POM file being changed? How could this be done? The solution is to adapt the version when Maven is loading the POMs to build the project -- provided that an appropriate system property is set (we have chosen the property name "version.override"). If "version.override" is not set, just leave the constant in the POM. The implementation requires a small Maven extension which must be installed in the extension directory ($M2_HOME/lib/ext) on the CI build server. With this extension installed, we can do mvn clean deploy -Dversion.override=1.2.3-RC-5 and all built artifacts are deployed to the repository with the release version "1.2.3-RC-5". This means that every CI server build can be a real release with easily locatable artifacts in the repository, and that every developer's local build can continue to just use SNAPSHOT versions (and no Maven extension is required on the developer workstations). However, there is one drawback to using this Maven extension. The POMs that are deployed will still contain the original constant version, and not that from the "version.override" property. This can lead to problems if other projects reference the POM and it has transitive dependencies. We are looking into ways of rectifying this issue. As of this writing, the Maven extension has been successfully used for a few weeks in our Continuous Delivery environment.

Links & References

[1] http://www.axelfontaine.com/2011/01/maven-releases-on-steroids-adios.html
[2] Source code and documentation: https://github.com/rotscher/emerging
[3] Binaries - Maven Core extension: http://search.maven.org/#browse%7C1365347524
                  - dependency-attach-plugin: http://search.maven.org/#browse%7C818638860

Montag, 12. November 2012

maven: dependency-attach-plugin

Never ever had the need to get a dependency and just redeploy without doing anything to it?  E.g. Download the dependency in version 0.0.5-SNAPSHOT and redeploy as 1.1-RC-5? Maybe this seems strange but we did it exactly this way to create a release based on current SNAPSHOTs from our artifact repository.

See [1] to get an impression. Executing mvn clean deploy would download the list of artifact items and deploy it as

my.releases:distribution:module-a:1.8-RC-5:jar


Of course, you’re right saying that all this makes no sense. Get a hint about how you can pimp your pom to make the version “flexible” at this blog.

[1] example pom

<project>
  <groupId>my.releases</groupId>
  <artifactId>distribution</artifactId>
  <version>1.8-RC-5</version>
  <packaging>pom</packaging>
  <build>
    <plugins>
      <plugin>
        <groupId>ch.rotscher.maven.plugins</groupId>
        <artifactId>dependency-attach-plugin</artifactId>
        <version>0.1.1</version>
        <executions>
          <execution>
            <goals>
              <goal>copy</goal>
            </goals>
            <phase>generate-resources</phase>
            <configuration>
              <useOriginalArtifactIdInClassifier>true</useOriginalArtifactIdInClassifier>
              <artifactItems>
                <artifactItem>
                  <groupId>my.modules</groupId>
                  <artifactId>module-a</artifactId>
                  <version>2.0.1-SNAPSHOT</version>
                </artifactItem>
              </artifactItems>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

[2] Download

From maven central (http://search.maven.org)
<dependency>
  <groupId>ch.rotscher.maven.plugins</groupId>
  <artifactId>dependency-attach-plugin</artifactId>
  <version>0.1.1</version>
</dependency>