Recently I decided it was time to add a couple of new TeamCity agents to the pool. More concurrent builds to speed up our continuous integration flows can only be a good thing for development, right?
Problem was the existing build agent was a mess of bashed together bits and pieces added in for new projects, to test out new ideas (that were promptly dropped), and never cleaned up as things were retired. As such, I made the brilliant decision to build these new build agents from scratch.
Two eons later I have finally setup all the SDKs, third-party tools, frameworks, features, add-ons, and configurations required to get all of our code building. I think there are 5 different versions of the .NET Framework being run simultaneously in our platform and there are some third-party tools where the words “backwards compatibility” are more of a nice day dream than a rule to live by. I keep promising myself I’ll clean it up but who has ever actually managed that?
Enough is enough. Having a zip folder containing 30 different installers, apps, etc. with one giant readme is all very well but I’d really rather never do it again. Enter docker, or more specifically “building in docker”.
The idea is simple: get a docker image that has the tooling required to build that project and that project only, copy the code over, run “docker build”, and pull out the artifacts once it’s done. Go one step further and turn those artifacts into a new docker image and then we can deploy that with, once again, only the tools we need. No more setting up everything under the sun just to speed up a few build chains.
So what are the issues here? Well, lets take an example. We want to build a new website, along side which, because we are “professionals”, there will be a suite a tests to be run.
Firstly, we’re going to switch from .NET Framework to .NET Core as it’ll actually work inside of a Linux container. Microsoft is making some great strides with Windows containers but it’s not quite ready yet (my opinion).
From this point actually building our website, including running those tests, within a container is easy enough. We get ourselves a TeamCity build agent on a Linux box with docker installed and build an image from a Dockerfile looking something like:
FROM microsoft/dotnet:2.0-sdk WORKDIR /build COPY ./ ./ RUN dotnet build -c Release \ && dotnet test -c Release --no-build ./mywebsite-tests && dotnet publish -c Release -o /release ./mywebsite
Just run “docker build . –name mywebsite-build” on the docker file in our TeamCity build configuration and Bob’s your uncle. Problem is: how do we extract the artifacts back to the host for later use? In this case we want to analyze the test results and make a docker image containing the website’s published artifacts (dlls, configs, etc.). Just to complete our example, here’s a Dockerfile for building our releasable:
FROM microsoft/dotnet:2.0 WORKDIR /website COPY ./release ./ ENTRYPOINT ["dotnet", "mywebsite.dll"]
So how do we deal with the artifacts?
How do we get the website built in the builder image to the releasable image? We can do this through the use of the docker command line with “docker create” and “docker cp” and “docker rm” but it’s not exactly what you’d call a neat solution, especially as we’ll need to support it for years to come (legacy never dies).
And how do we get the test results? Currently they just get written to the console. Sure the test step will return non-zero if any of the tests fail which will stop the build, but how are we going to look into the failures? Do you want to read through the command line logs because I don’t. I prefer to use TeamCity’s built in tests tab to see what’s been going on (not to mention all the other good stuff TeamCity does with test results).
So what can we do?
TeamCity 2017.2 (still EAP when I wrote this blog) has a new feature already supported in Jenkins: docker wrappers. With 2017.2 we can “wrap” our build process in a docker container of our choosing. Currently it supports wrapping up Maven, Gradle, and Command Line steps. As we’re not building a Java application we’ll be using the command line. Now there is an important difference between what’s described above and how TeamCity does the wrapping: TeamCity runs the step inside a running container, not while building the image.
TeamCity can mount the entire checkout drive into the docker container and then run the commands as if we were in a normal console. All the resulting artifacts are on the host (thanks to the mount) so we can do with them what we will.
To read the test results we can output them to a trx formatted file and use the “XML report processing” build feature in our build configuration to process the test results just like we would if we had used an NUnit step. So our command line build step, wrapped in the microsoft/dotnet:2.0-sdk image, would look like:
dotnet build -c Release dotnet test -c Release --no-build --logger trx ./mywebsite-tests dotnet publish -c Release -o ./release ./mywebsite
TeamCity hides all of the container magic so we don’t have to worry about. Nice of it but it it does make duplicating the functionality on our laptops a bit of a pain. Locally we can use the builder Dockerfile above with a docker build, but TeamCity isn’t doing that. It’s doing it’s own thing. What if we want to add a new step (maybe another test suite) to the build process? Now we have to change both the local build Dockerfile and the TeamCity build configuration. Ideally we want TeamCity to follow exactly the same steps as we do on our laptops (read Continuous Delivery by Jex Humble and Dave Farley for more info on this topic).
We can write all the scripts to copy TeamCity’s functionality down to our laptop (which though possible has problems when we bring docker-compose into the mix as discussed later on) or we can change how TeamCity does things.
Multi-stage docker builds
Multi-stage builds are a way of creating a build chain where steps are done inside individual image builds and artifacts can be passed on to the next step with the final step being the releasable image.
With this we can just write a single Dockerfile that will run our build process inside the build image, copy the resulting artifacts to the releasable, and then build the releasable image. Doing all that requires a single “docker build” command:
FROM microsoft/dotnet:2.0-sdk AS builder WORKDIR /build COPY ./ ./ RUN dotnet build -c Release \ && dotnet test -c Release --no-build ./mywebsite-tests && dotnet publish -c Release -o /release ./mywebsite FROM microsoft/dotnet:2.0 WORKDIR /website COPY --from=builder /release . ENTRYPOINT ["dotnet", "mywebsite.dll"]
Wonderful yes? Now our TeamCity step doesn’t need to use the docker wrapper, it just does a “docker build” same as we do. Note that a Docker Build step has also been added as part of 2017.2 so we can use that instead of a Command Line step.
We can now go one step further and spool up docker-compose on our laptops. Using this we can easily build and run our website inside of docker locally with things like port mapping all dealt for us using just a bit of YAML:
--- version: '3.2' services: mywebsite-service: build: . ports: - "5000:50" container_name: "mywebsite"
Then just run “docker-compose up -d –no-cache” and our build process will fire off. Provided the build works and the tests succeed we will end up with our website running in a docker container on our laptop. Cool right?
But what about those test result?
So though the multi-stage builds are really helpful for keeping TeamCity’s build configuration the same as our local process, as well as making building and running the website on our laptop as simple as running a single command, TeamCity no longer has any access to the contents of the image.
Multi-stage builds are designed to clean up for themselves and for good reason. We don’t want those intermediate images hogging disk space. Problem is, without that first image existing how are we going to access the test results? Right now we are back to them just writing to the console. As, unlike TeamCity’s docker wrapper, we are running the build inside “docker build” instead of a running container we can’t just mount a directory and write the results to it.
Option 1: We could just pass the test results to the releasable image as trx files and extract them from there using “docker cp” but that means we’re putting things that aren’t part of our website into our releasable. Not terrible considering the size of most test results files but definitely a “CI smell”. We should only package what is needed to run the website into that releasable.
Option 2: We could write the results to trx files as before and then upload them via SFTP/Samba/REST-API/etc. This is messy in so many ways. Regardless of what route we take, those trx files have to end up in the running TeamCity build agent’s checkout directory for this build for the “XML reporting feature” to pick them up. After we’ve bullied the build into doing that we will have to work around this bit of functionality on our laptops. This ends up turning our nice 3 line build script into something a lot more complex.
Option 3: Figure out how to make TeamCity understand the test results as they are written to the console. This would remove the need for the trx files and the XML reporting feature entirely as well as keeping our build script nice and simple. But how to do it?
TeamCity service messages
Within TeamCity is something called “service messages“. In short, service messages are messages printed to the console in format understood by TeamCity. Depending on the type of service message, TeamCity will interpret it in different ways. For example we can publish build artifacts (provided the build agent can access them from a file path so not a lot of good for us here).
The important one is the reporting of test results. If we can get the “dotnet test” command outputting the results as service messages then TeamCity will be able to understand the tests and handle them as normal. As we saw above we can add a “–logger trx” argument to the “dotnet test” command to make it output the test results into a trx file. All we need then is a logger that will output service messages instead.
Unfortunately this is not a part of .NET Core that is overly well documented but luckily for us someone has already written some third-party loggers:
Now all we need to do is write our own that will output the test results as TeamCity service messages. Here’s one I made earlier.
What this does is listen to test events with message handlers. Each time a message comes in it is interpreted and the relevant service messages are printed out (start suite, start test, test results, etc.). TeamCity picks up these messages and handles them as proper tests.
In order to use the logger we can do one of two things:
- Add the logger as a reference to the test library
- Add the logger’s dll to the .NET extensions directory:
Windows: C:\Program Files\dotnet\sdk\2.0.0\Extensions Linux: /usr/share/dotnet/sdk/2.0.0/Extensions/
After installation the logger can be used like a normal dotnet test logger:
dotnet test --logger teamcity
Easiest way is just to create a new root image that adds the logger dll to the extensions directory of the SDK image, such as this one.
Now our final Dockerfile looks like:
FROM bangonet/dotnet-teamcity-logger:2.0-sdk AS builder WORKDIR /build COPY ./ ./ RUN dotnet build -c Release \ && dotnet test -c Release --no-build --logger teamcity ./mywebsite-tests && dotnet publish -c Release -o /release ./mywebsite FROM microsoft/dotnet:2.0 WORKDIR /website COPY --from=builder /release . ENTRYPOINT ["dotnet", "mywebsite.dll"]
So there we are. We and TeamCity can use the same Dockerfile to build and run the application, with the added bonuses that we can use docker-compose and TeamCity can read the test results.
Hope it’ll be of help to some others but if not, well at least I’m happy(ish).