Upgrade Debian package from systemd service

Goal is to distribute a Java application as a Debian package and allow users to upgrade the package to newer version from within the application itself. Application is running as a systemd service, meaning itself and processes which it spawns are isolated in a control groups "container".

This article does not describe a real world implementation. Instead, it can be used as a template for someone trying to achieve similar results.

Prerequisites

Debian package installs a Java application to a target system which is the executed as a systemd service.
Java application runs with privileges of an "ordinary" user.
Package will be distributed via APT and is upgraded by apt upgrade command.
The application will be distributed as a single package and runs as a single service. So, "manager" packages or services are not allowed.
Build will be executed on Ubuntu 18.04 with build-essential, debhelper, devscripts and openjdk-11-jdk packages installed.
Target system for the application is Ubuntu 18.04.

Implementation

Part of this article is the Debian Package Demo project which can be used for testing of following techniques.

Application and service

In the demo project all Java application's sources and configurations are located in src/ directory under the project's base directory. In this demo scenario the Java application is represented by a simple DebianPackageDemo.java which invokes upgrade mechanism every 30 seconds.

Next to the application file lies systemd service configuration. There is not too much into it except for, the application will be installed in the /usr/lib/debian-package-demo/ directory and this directory will also be application's working directory and home directory of debian-package-demo user.

systemd configuration will be later installed to the /lib/systemd/system/ directory.

Extract from the systemd debian-package-demo.service configuration.

...
[Service]
User=debian-package-demo
# Application is installed in the following directory.
WorkingDirectory=/usr/lib/debian-package-demo/

ExecStart=/usr/bin/java DebianPackageDemo

# Successful exit code of JVM is not 0, but 143
SuccessExitStatus=143
...

Debian package

To create a Debian package, the project must contain debian/ directory with apropriate files in it.

Because debuild, tool for building Debian packages from source files, generates its output to a parent directory of a directory in which debian/ is located, in the demo application there is a package/debian/ structure so all files are generated into project's base dir instead of directory one level above it. This also means that all Debian package commands should be executed in package/ directory.

control file contains basic information about the package (name, description) as well as list of dependencies needed to build the package and to install it on a target machine.

In case of this demo, only debhelper and JDK are needed for building the application and JRE for running it. Following example shows bare minimum of source package control fields needed for creating a valid control file.
```
Source: debian-package-demo
    Maintainer: Some Name <some.name@email.com>
    Build-Depends: debhelper (>= 9), openjdk-11-jdk (>= 11)
    Standards-Version: 3.9.8

    Package: debian-package-demo
    Architecture: any
    Depends: openjdk-11-jre (>= 11)
    Description: Debian Package Demo application.
     More description of Debian Package Demo application.
```
changelog file contains information about versions and releases of the application. This file has defined structure and even though it is possible to modify it manually it is highly encouraged to use dch command for its manipulation.

By navigating to package/ directory, you can use following commands for releasing new version.
```
# Add new record to changelog. -U use standard version number, ie: 1.0.1, -i insert new version.
    dch -Ui "New version comment"
    # Mark latest version as released. Use empty string "" to NOT add new comment.
    dhc -r ""
```

rules is a makefile used for application building. Those who are not familiar with makefiles, should take in consider that makefile is similar to shell scripts, but there are some differences described in GNU make documentation. For example only tabs and no spaces are allowed in front of commands, variables can be defined in a particular pars of the script, and they are referred to by using normal brackets $(VARIABLE), etc.

In the demo application the makefile build target is used for compiling the application and install target for copying it's files to a directory from which it will be picked up by debuild and packed into a generated Debian package.

The usual way to create rules file is not to write it from scratch, but to use dh command sequencer for every target and then use target hooks for targets you want to override. See the "any" target pattern %:.

#!/usr/bin/make -f

    SOURCE = $(CURDIR)/../src
    TARGET = $(CURDIR)/debian/debian-package-demo

    %:
            dh $@

    override_dh_auto_build:
            javac $(SOURCE)/DebianPackageDemo.java

    override_dh_auto_install:
            # Created directory structure represents directories where files will be installed on a target machine.
            mkdir -p $(TARGET)/usr/lib/debian-package-demo
            mkdir -p $(TARGET)/lib/systemd/system
            mkdir -p $(TARGET)/etc/sudoers.d/
            cp $(SOURCE)/DebianPackageDemo.class $(TARGET)/usr/lib/debian-package-demo
            cp $(SOURCE)/debian-package-demo.service $(TARGET)/lib/systemd/system
            # Reason for following files will be explained later.
            cp $(SOURCE)/debian-package-demo-upgrade.sh $(TARGET)/usr/lib/debian-package-demo
            cp $(SOURCE)/debian-package-demo-sudoers $(TARGET)/etc/sudoers.d

copyright file contains licence.
compat (compatibility) file contains a version of debhelper needed to make the package.

Invoking the upgrade from within the application

Upgrade process can be described by following steps:

The Java application creates apt upgrade process. The apt upgrade command is wrapped in the debian-package-demo-upgrade.sh shell script.
At the start of upgrade, APT will stop the application itself. This is handled by preinst script.
Then APT will copy files from deb package to their target locations.
APT will start service in postinst script.

First of all, because APT has to be called with elevated permissions, to be able to install a package, sudoers file has to be created. Due to this file, debian-package-demo user running the application will be able to execute apt upgrade command without password. If better security is required the sudo command can read passwords from standard input if -S switch is provided.

# Upgrade the application without password.
debian-package-demo ALL=NOPASSWD: /usr/bin/apt -y upgrade debian-package-demo

Next, the Java application cannot simply create child process which will stop it. Even if created process is detached. Because systemd service is running in a container, every child process created in that container will be destroyed as soon as the service itself is stopped.

To bypass this behaviour, one-off user task is scheduled by systemd-run from the Java application with activation delay 1 second. This task will be later picked up by systemd and executed in a new, separated container.

In the Java application there is code which invokes upgrade:

String scheduleUpgradeCommand = "systemd-run --user --on-active=1 /usr/lib/debian-package-demo/debian-package-demo-upgrade.sh";
ProcessBuilder processBuilder = new ProcessBuilder("/bin/sh", "-c", scheduleUpgradeCommand);
// Environment variable XDG_RUNTIME_DIR has to be set for
// systemd-run to run correctly. Otherwise "Failed to create
// bus connection" error will occur.
processBuilder.environment().putIfAbsent("XDG_RUNTIME_DIR", "/run/user/" + new UnixSystem().getUid());
// Redirect process I/O to the application's I/O so it's output can be
// observed by journalctl.
processBuilder.inheritIO();
processBuilder.start();

Shell script debian-package-demo-upgrade.sh contains simple command:

#!/bin/sh -e

sudo apt -y upgrade debian-package-demo

Finally, the Debian package should contain preinst, prerm and postinst scripts which will stop and start systemd service during installation and eventually set-up the environment for the application.

Post installation script postinst creates user for the application and enables systemd session lingering for the user.

Scheduled upgrade task is created as a systemd user timer which needs systemd user session to be present. The session is usually created during user's login and destroyed on its logout. In case of demo application debian-package-demo user will never login, so the session has to be created on system startup. This can be achieved by so called session lingering which will create the session on system boot.

#!/bin/sh -e

userName=debian-package-demo
serviceName=debian-package-demo

if ! id ${userName} > /dev/null 2>&1; then
        adduser --quiet --system ${userName} --home /usr/lib/${serviceName}
        # Because the Java application will schedule one-off systemd user
        # services systemd session has to exist for user even if the user
        # is not logged-in. To ensure this session lingering will be enabled
        # for user.
        loginctl enable-linger ${userName}
fi

systemctl daemon-reload
if ! systemctl is-active --quiet ${serviceName}; then
        echo "Starting ${serviceName} service..."
        systemctl enable ${serviceName}
        systemctl start ${serviceName}
fi

Scripts preinst and prerm are similar to postinst.

Build and distribution of the package

To build and distribute first version of the package and then to release a new version and upgrade it we need to:

Build Debian package by calling debuild.
Move generated files to repository directory and generate repository index by dpkg-scanpackages.
Add the repository to target systems APT sources list.
Install the package and start the application / service.
Release a new version.
Let the application to upgrade itself.

The demo application has following directory structure.

. <-- Debian Package Demo project's base dir
├── package
│   └── debian
│       ├── changelog
│       ├── compat
│       ├── control
│       ├── copyright
│       ├── postinst
│       ├── preinst
│       ├── prerm
│       └── rules
├── repository
└── src
    ├── DebianPackageDemo.java
    ├── debian-package-demo.service
    ├── debian-package-demo-sudoers
    └── debian-package-demo-upgrade.sh

In package/ directory call the debuild -us -uc command which will compile Java application and build Debian package files into its parent directory. Switches -us -uc will cause unsigned package to be built which is acceptable for testing purposes, but it is highly recommended to sign packages in production environment.

Next step is to move generated debian-packages-demo* files to repository/ directory and then to generate repository index. This is done by the dpkg-scanpackages -m . /dev/null > Packages command.

Let's assume the application will be installed by APT on a machine where it is build. Then the repository directory can be added to APT sources list as a simple file repository.

# Because packages are not signed, the repository has to be marked as trusted.
deb [trusted=yes] file:///debian-packages-demo-base-dir/repository ./

Now the demo project can be installed.

apt update
apt install debian-package-demo

To release a new version of the application version of the Debian package has to be increased, new package has to be build and moved to repository.

Use the dch command to increase version as described before.
Build the package by debuild and move generated files to the repository/ directory.
Regenerate repository index by dpkg-scanpackages.
Call apt update to update sources list and wait for the application pick up latest version by invoking the apt upgrade command.

Troubleshooting

If your "package experiments" led you to the point where a package cannot be removed or reinstalled because APT complains about "something is horribly broken", then the package can be "forcefully" removed from the system by directly calling of the underlying dpkg package manager. Note: APT is "just" a wrapper which downloads relevant packages from a repository to a target computer. Then it calls dpkg which installs, upgrades or removes it.
```
dpkg --force-remove-reinstreq -r debian-package-demo
```
Standard I/O from the processes created by Java application are redirected to the Java application's I/O itself. Then, the output can be observed in the systemd log of the application. If the application is not able to "upgrade itself" it is a good idea to look into it.
```
journalctl -u debian-package-demo
```
If the application reports a relatively common error "Failed to create bus connection: No such file or directory" make sure lingered systemd session for debian-package-demo user exists. In this case the /run/user/$UUID directory must exist, where UUID is a number ID of the user. Also, an environment variable $XDG_RUNTIME_DIR must point to that directory, and the variable must be available for the environment executing systemd-run.