# Upgrading from `v0.15.x` to `v0.16.x`

Stalwart `v0.16.x` introduces **significant breaking changes** that make its configuration and management layer **completely incompatible** with every previous release. The database layout used to store user data (emails, calendars, contacts, files, blobs, search indexes) is **not** affected by this change, so message bodies, mailboxes, calendar events, and shared files remain on disk unchanged. What does change is **how the server is configured and managed**, and because those records live inside the same database, a multi-step migration is required.

Before continuing, please read this document in full. Skipping steps will leave the server in an unrecoverable state and will require restoring from a backup.

If any step below raises questions, a dedicated discussion thread for the `v0.16` upgrade is open at https://support.stalw.art. The earlier design discussion that led to these changes is also public at https://github.com/stalwartlabs/stalwart/discussions/2892 and describes the user-reported problems that motivated each breaking change.

## A note on downtime

Email is a critical service, and we understand that a forced maintenance window is disruptive:  in some environments it is simply not an option. The breaking changes in `v0.16` are not cosmetic. Stalwart has been under continuous development for close to five years; in that time the feature set and the user base have both grown well beyond what the original configuration and management layer was designed for. The gap between what users need and what the old architecture can cleanly support has widened to the point where a redesign was unavoidable:  and the redesign itself unlocks a long list of frequently-requested features that were simply not implementable under the previous model. The storage layer is untouched by all of this: emails, calendars, contacts, files, and every other piece of user data stay exactly where they are. The migration is about configuration, not about data.

Operators who cannot accept downtime should **wait**. In the next two to three weeks we plan to release two tools that work together:

- A **zero-downtime migration utility** that moves data (accounts, mailboxes, calendars, contacts, files) from an existing (`v0.15.x` or below) deployment to a freshly-installed `v0.16.x` deployment one account at a time, while both servers are running.
- A **proxy** that sits in front of both deployments and routes each incoming connection to the server that currently owns that account. As accounts are migrated one by one, the proxy transparently shifts their traffic from the old deployment to the new one, so end users never notice a cutover.

Together, these let operators migrate a live production deployment on an account-by-account basis with no scheduled maintenance window. When those tools are available, the instructions in this document will be superseded for most deployments. Everyone else can follow the manual steps below during a scheduled maintenance window.

## What has changed

### No more TOML configuration files

The previous server used one or more TOML files, with some settings living on local disk and others living in the database. In `v0.16` there is a single small `config.json` on disk that describes **only** the datastore (the database Stalwart uses to keep everything else). Every other configuration and management setting:  domains, accounts, mail routing, DKIM signatures, storage backends, rate limits, spam rules, and so on:  is now stored inside that datastore as a **JMAP object**. JMAP ("JSON Meta Application Protocol") is the JSON-based API Stalwart uses to expose its data; treating configuration as JMAP objects means the same API that serves email metadata also serves server configuration.

This change is driven by two real problems with the old model. First, in a **clustered deployment** every node had to carry its own copy of the configuration file and stay in lockstep with every other node. Divergence was easy to introduce and hard to debug, and it made distributed deployments unnecessarily fragile. Centralising everything in the database means configuration is consistent across the cluster by definition. Second, the **split between "settings in the file" and "settings in the database"** was a persistent source of user confusion: the same conceptual setting had to be documented in two places depending on where it happened to live, and administrators routinely edited the wrong one. A single unified model removes that entire category of mistake, and it gives management tooling (the WebUI and the CLI) a complete view of the system.

**For Ansible, NixOS, Terraform, and other declarative tooling:** the small `config.json` is still a plain file and can be managed with existing tooling exactly as before. Everything that used to live in TOML is now managed through [`stalwart-cli apply`](https://stalw.art/docs/management/cli/apply), which accepts a declarative plan file and idempotently reconciles the live server state to match it, creating what is missing, updating what has changed, and removing what the plan no longer declares. This is the same pattern used by CockroachDB (cluster settings via SQL/CLI), Consul (KV store), Elasticsearch (`PUT /_cluster/settings`), and HashiCorp Vault (CLI/API for policies and secrets); infrastructure-as-code tooling targets the API rather than a file. The workflow becomes: commit the declarative plan to version control, deploy `config.json` through existing tooling, and invoke `stalwart-cli apply` as an idempotent step in a playbook or activation script.

### REST API replaced by JMAP

The `/api/...` endpoints from previous releases no longer exist. All management operations happen through **JMAP objects** reachable at `/jmap`. JMAP (RFC 8620) is a well-specified, transport-efficient protocol with first-class support for batch operations, push notifications, and fine-grained change tracking. Stalwart already speaks JMAP for email:  extending it to administration gives operators and integrators a single consistent protocol for interacting with the entire server. In practice this means dozens of configuration changes can be applied in a single round-trip (the `apply` command uses this), any JMAP client library works against the management surface, and the same authentication flow covers both mail access and administration. Existing scripts and integrations that called the old REST endpoints must be updated; the new CLI is the straightforward replacement for most of them.

### Account names must be email addresses

Every user and group principal now has a **local part** (the name) and an associated **domain**. In previous releases an account could be a bare string such as `alice`; in `v0.16` it must be `alice@example.com`. The migration script handles this automatically:  accounts without a domain are assigned the default domain of the deployment (chosen by scanning existing principals for the most common domain), so no users are lost during conversion.

To avoid locking existing users out of their mail clients on the first login after the upgrade, `v0.16` **automatically appends the default domain** when a client authenticates with a bare username. Administrators running an **external directory** (LDAP, SQL, etc.), however, do need to update their directory filters to query by full email address rather than by bare account name; the old filters will no longer match.

**CalDAV, CardDAV, and WebDAV clients need one manual adjustment.** These protocols use the account name as part of the URL path (for example `/dav/cal/alice`), and because the account name is now a full email address, that path changes. The `@` character is reserved in URLs and must be encoded as `%40`, so the equivalent path in `v0.16` becomes `/dav/cal/alice%40example.com`. Authentication itself still works (the server accepts the bare username and appends the default domain, as described above), but calendar, contact, and file sync will stop working until each client is reconfigured to point at the new path. It is a good idea to notify users before the upgrade so that they can update their calendar and contacts accounts in Apple Calendar, Thunderbird, DAVx⁵, and similar clients.

Two reasons drove this requirement. The first is **support for multiple external directories simultaneously**: when account names are bare strings there is no reliable way to tell which directory owns a given username, whereas email addresses are naturally namespaced by domain and make that mapping unambiguous. The second, and more consequential, reason is the **PACC specification** ([draft-ietf-mailmaint-pacc](https://datatracker.ietf.org/doc/draft-ietf-mailmaint-pacc/)):  the IETF's replacement for the fragmented collection of autoconfig / autodiscover / SRV-record mechanisms that mail clients use today to discover server settings. PACC expects login names shaped like email addresses; when they are not, the server has to reveal whether a given account exists just to disambiguate the login, which is exactly the privacy leak the spec is designed to prevent. Aligning account names with email addresses is what lets Stalwart implement PACC correctly.

PACC also brings OAuth into the autodiscovery flow, and because the draft originates from Apple, a correct PACC implementation is the path to supporting Apple Mail clients with OIDC and MFA:  a long-standing user request that only becomes possible once this groundwork is in place.

## What has been added

- **A brand-new WebUI**, rewritten from scratch on top of the new JMAP-based management API.
- **A brand-new CLI** (`stalwart-cli`) that also uses the JMAP API and can be used for day-to-day administration, scripted deployments, and infrastructure-as-code workflows. Full documentation is available at https://stalw.art/docs/management/cli.
- **[Over one hundred feature requests and bug fixes](https://github.com/stalwartlabs/stalwart/blob/main/CHANGELOG.md#0160---2026-xx-xx)** across every subsystem.

## Evaluate `v0.16` before migrating

Because so much has changed, `v0.16` will feel like a different product at first contact. Concepts have been renamed, some have been removed, and several new ones have been introduced.

It is **strongly recommended** that operators first install a fresh `v0.16` instance in a Docker container or a throwaway virtual machine, log into the new WebUI, and spend time becoming familiar with how configuration works in the new release. This avoids the situation where a critical production upgrade is the first time an operator sees the new interface.

A second, equally important benefit: any settings created in the test deployment (directory integrations, SMTP listeners, spam rules, rate limits, TLS providers, etc.) can be exported using the [`snapshot`](https://stalw.art/docs/management/cli/overview/snapshot) command. The resulting JSON file is an `apply` plan that can be fed directly into the production instance after the migration completes. Time spent on a test deployment is not thrown away.

## How the migration works

The migration is a **multi-step, offline** process. At a high level:

1. If the server is still on a version older than `v0.15.x`, it must first be upgraded to `v0.15.x`. The `v0.16` migration tooling does not support anything older. Operators who cannot upgrade to `v0.15.x` now should wait for the zero-downtime proxy described above, which will perform a direct migration from older releases.
2. A Python helper script is run against the live `v0.15.x` server. It downloads the current settings and principals, converts them to the new format, and produces two files: `config.json` (the new on-disk datastore configuration) and `export.json` (a snapshot of everything else, in a format that the new CLI can replay).
3. The `v0.15.x` server is stopped and its database is backed up.
4. The `v0.16` binary (or Docker image) is started in **recovery mode**. On first start it detects the old data, wipes the pieces that are no longer compatible, migrates the spam classifier model, and comes up listening on a single HTTP port (`8080`) exposing the management API.
5. `stalwart-cli apply` replays `export.json` (and, optionally, any snapshots from the test deployment) against the recovery-mode server.
6. Recovery mode is disabled, the service manager (systemd / init.d / Docker) is reconfigured to use the new `config.json`, and the server is restarted normally.
7. Post-migration tasks are triggered from the WebUI to recalculate disk quotas.

The following sections describe each step in detail.

> **Note for clustered deployments.** Before starting the migration, **every node in the cluster must be stopped**. If even one node is left running on `v0.15.x` while another is being upgraded, it will write records in the old format and cause data corruption that can only be repaired by manually deleting the offending keys. This requirement is repeated in the binary and Docker sections below, but it applies globally.

## Step 1:  Convert existing settings into a configuration snapshot

This step is **independent of how Stalwart is deployed** and **does not require stopping the server**. The migration script talks to the running `v0.15.x` server over its management API and produces two JSON files on the machine where it is run. Running this step early is encouraged: it gives the operator a chance to review the generated files before touching the server, and to rerun the conversion with different options if needed.

### Download the migration script

Download the script from the Stalwart repository:

```bash
$ curl -fLO https://raw.githubusercontent.com/stalwartlabs/stalwart/refs/heads/main/resources/scripts/migrate_v016.py
```

Review the script before running it. It is a single self-contained Python file and makes no changes to the running server:  it only reads configuration and principal data.

### Create a Python virtual environment

A virtual environment (`venv`) is a self-contained Python setup that keeps installed libraries out of the system-wide Python install. This avoids polluting the host Python and lets the script run on systems where `pip` installs are restricted.

```bash
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install requests urllib3
```

The first command creates the environment in a `.venv/` directory. The second activates it (the shell prompt usually gains a `(.venv)` prefix). The third installs the only two libraries the script needs.

### Dump the live `v0.15.x` settings

The script has two subcommands. The first, `dump`, connects to the running server and downloads its settings and principals into two files on disk:

```bash
(.venv) $ python migrate_v016.py dump \
    --url https://mail.example.com \
    --username admin \
    --password adminPassword \
    --settings settings.json \
    --principals principals.json
```

Replace the URL and credentials with those of the `v0.15.x` server. The admin account must have permission to read all settings and principals. Output files default to `settings.json` and `principals.json` in the current directory. These files are plain JSON:  opening them in a text editor to inspect their contents is encouraged.

### Convert the dump to the new format

The second subcommand, `convert`, reads the two dump files and produces the two files that the new server will consume:

```bash
(.venv) $ python migrate_v016.py convert \
    --settings settings.json \
    --principals principals.json \
    --config config.json \
    --output export.json
```

This produces:

- `config.json`:  the new on-disk datastore configuration. This is the file the `v0.16` server will be pointed at on startup. It is small, because it describes only the datastore (data store, blob store, search store, in-memory store).
- `export.json`:  a snapshot of every other piece of state the script could convert, in the format consumed by `stalwart-cli apply`. This file will be replayed against the `v0.16` server once it is running in recovery mode.

### What the script converts:  and what it does not

The conversion is intentionally conservative. Only the following settings are migrated, because the rest have changed enough that automatic mapping would do more harm than good:

- **User accounts, groups, and mailing lists** (with their aliases and memberships)
- **Tenants** (in multi-tenant deployments)
- **Domain names** (including those discovered by scanning the addresses of existing accounts)
- **Data store, blob store, full-text search store, and in-memory store** settings
- **DKIM signatures** (except `rsa-sha1`, which is obsolete and not supported in `v0.16`)
- **TLS certificates** (including those issued by the built-in ACME client)

**Everything else**:  SMTP listeners, mail routing rules, rate limits, connection limits, spam filter settings, logging and telemetry configuration, authentication backends other than the ones listed above, session scripts, Sieve preludes, milter/MTA hook configuration, etc.:  must be recreated on the new server.

This is the reason the test deployment recommended above is so useful: recreating the remaining settings on a test `v0.16` instance, then using [`stalwart-cli snapshot`](https://stalw.art/docs/management/cli/snapshot) to export them, turns what would otherwise be manual post-migration work into a second `apply` run. If the production deployment is close to the defaults, this is straightforward. If it has extensive customisation, plan for the time this takes.

## Step 2:  Back up the database

When the `v0.16` server starts for the first time, it will **wipe** the parts of the database that are no longer compatible with the new schema. No user mail is touched, but everything below is **deleted unconditionally**:

- **Directory records.** Every user, group, tenant, OAuth client, domain, and mailing list record is removed. The mail itself stays in place on disk:  it is the *directory entries* describing who owns it that are deleted. After the first `apply`, the new directory entries produced from `export.json` recreate these records with the same identities.
- **All settings.** Every stored setting is deleted. `export.json` replays what the script was able to convert; anything the script could not convert needs to be recreated manually (or via a snapshot from the test deployment).
- **DMARC, TLS, and ARF reports** (both incoming and outgoing). Report records are incompatible with the new schema.
- **Pending tasks.** Maintenance tasks queued for background execution are discarded. `v0.16` exposes a task panel in the WebUI, and the equivalent tasks can be triggered manually from there.
- **Telemetry data.** Metrics and trace spans are deleted.
- **Spam training samples** (but **not** the spam classifier model itself:  the model is preserved and migrated).
- **Disk quotas.** All per-account disk-usage counters are reset to zero and must be recalculated after the migration (see Step 4).

Because the wipe is irreversible, a full backup of the existing data must exist **before** the new server is started.

### Embedded databases (RocksDB, SQLite)

These store everything in a single directory on disk (typically `/var/lib/stalwart/data` or `/opt/stalwart/data`). A file-level copy while the server is stopped is sufficient:

```bash
$ sudo systemctl stop stalwart        # or the equivalent for the service manager in use
$ sudo cp -a /var/lib/stalwart /var/lib/stalwart.v015-backup
```

Record the path of the backup somewhere safe. If the migration fails, restoring this directory and starting the old binary returns the system to its previous state.

### PostgreSQL / MySQL backends

The database holds many tables, but only a subset needs to be captured to be able to undo the migration. Each table is a single ASCII character that corresponds to an internal Stalwart subspace. The destructive part of the migration touches the following tables:

| Table | Purpose | Priority |
|:---:|---|---|
| `s` | Settings | **Critical**:  contains all server configuration |
| `d` | Directory | **Critical**:  users, groups, domains, tenants, mailing lists, OAuth clients |
| `r` | Incoming reports (DMARC, TLS, ARF) | Recommended |
| `h` | Outgoing reports | Recommended |
| `b` | Legacy bitmap index | Recommended |
| `g` | Legacy full-text-search index | Recommended |
| `j` | Legacy blob-extra metadata | Recommended |
| `f` | Pending task queue | Recommended |
| `u` | Quotas (partially reset) | Recommended |
| `o` | Telemetry spans (traces) | Optional:  can be very large |
| `x` | Telemetry metrics | Optional:  can be very large |
| `w` | Legacy telemetry/spam-sample index | Optional:  can be very large |

The telemetry tables (`o`, `x`, `w`) can grow into tens of gigabytes on busy servers. Skipping them from the backup is reasonable unless there is a specific need to preserve historical metrics or traces.

For PostgreSQL, a per-table dump looks like this:

```bash
$ pg_dump -U stalwart -d stalwart \
    -t s -t d -t r -t h -t b -t g -t j -t f -t u \
    -f /var/backups/stalwart-v015-critical.sql
```

The equivalent with `mysqldump`:

```bash
$ mysqldump -u stalwart -p stalwart \
    s d r h b g j f u \
    > /var/backups/stalwart-v015-critical.sql
```

A full database dump (`pg_dump` / `mysqldump` without the `-t` flags, or `pg_dumpall`) is the safest option if disk space allows.

## Step 3:  Perform the migration

This step is the only one that requires downtime. The sequence has moving parts, and each moving part must complete before the next begins. Reading this entire section before starting is strongly encouraged.

> **Clustered deployments:** stop **every** node before beginning. Leaving a single `v0.15.x` node running while the migration is in progress will corrupt the database.

### Option A:  Binary deployments (systemd / init.d)

The following instructions assume the standard FHS layout (`/usr/local/bin/stalwart`, `/etc/stalwart/config.toml`, `/var/lib/stalwart`). Operators using a custom prefix (for example `/opt/stalwart`) should substitute their paths accordingly.

**1. Download the `v0.16` binary.** Grab the release matching the target platform from https://github.com/stalwartlabs/stalwart/releases/latest. Do **not** replace the running binary yet.

**2. Stop the old service.**

- On systems with systemd:
  ```bash
  $ sudo systemctl stop stalwart
  ```
- On SysV-style systems with init.d:
  ```bash
  $ sudo service stalwart stop
  ```

Verify the process is gone with `ps` before continuing. In a cluster, repeat this on every node.

**3. Back up the old binary and install the new one.**

```bash
$ sudo mv /usr/local/bin/stalwart /usr/local/bin/stalwart.v015
$ sudo mv /path/to/downloaded/stalwart /usr/local/bin/stalwart
$ sudo chmod 0755 /usr/local/bin/stalwart
$ sudo chown root:root /usr/local/bin/stalwart
```

**4. Install the new `config.json`.** The file produced by the migration script in Step 1 goes where the old TOML configuration used to live:

```bash
$ sudo mv /path/to/config.json /etc/stalwart/config.json
$ sudo chown stalwart:stalwart /etc/stalwart/config.json
$ sudo chmod 0640 /etc/stalwart/config.json
```

The old `config.toml` can be kept as a reference but is no longer read by the server.

**5. Start the new binary in recovery mode from the foreground.** Running the initial migration under the service manager is discouraged: if something goes wrong, the output scrolls past in `journalctl` and the restart loop masks the cause. Instead, run it directly as the `stalwart` user so that stdout and stderr are visible in the current terminal:

```bash
$ sudo -u stalwart env \
    STALWART_RECOVERY_MODE=1 \
    STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
    /usr/local/bin/stalwart --config=/etc/stalwart/config.json
```

`STALWART_RECOVERY_MODE=1` tells the server to enter the one-shot migration path: wipe the incompatible subspaces listed above, migrate the spam classifier model, and then bring up **only** the management HTTP endpoint on port `8080`. Mail ports stay closed. `STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword` provisions a temporary admin credential that the CLI can authenticate against:  this is needed because the converted `export.json` does not grant admin rights to any user (that is deliberate; admin assignment is a deployment decision). Replace `someTemporaryPassword` with a strong value; this account exists only until a real admin is created.

The migration output will scroll past. When it finishes, the process stays in the foreground, listening on port `8080`. Leave this terminal open.

**6. Apply the exported snapshot.** From a **second terminal** (on the same host or any machine that can reach the server on port `8080`), install the new CLI (*make sure to install v1.0.2 or later*):  instructions at https://stalw.art/docs/management/cli/overview:  and run:

```bash
$ export STALWART_URL=http://127.0.0.1:8080
$ export STALWART_USER=admin
$ export STALWART_PASSWORD=someTemporaryPassword
$ stalwart-cli apply --file /path/to/export.json
```

A summary similar to the following should appear:

```
Plan: 0 destroy, 5 update, 6 create (…)
✓ created Tenant (…)
✓ created Domain (…)
✓ created Account (…)
…
Done: 0 destroyed, 5 updated, … created (0 failed)
```

If any operation fails, the CLI stops immediately and prints the error. Fix the root cause (usually a conflict with an object created in an earlier attempt) and rerun. `apply` is re-entrant with `--continue-on-error` when needed.

At this point, snapshots exported from the test deployment with `stalwart-cli snapshot` can also be applied, in order:

```bash
$ stalwart-cli apply --file /path/to/test-deployment-snapshot.json
```

**7. Shut down recovery mode.** Return to the terminal running the foreground server and press `Ctrl+C`. The process will exit cleanly.

**8. Reconfigure the service manager.** The systemd unit or init.d script still references the old TOML path. Update it to point at the new JSON file:

- For systemd (typically `/etc/systemd/system/stalwart.service`), locate the `ExecStart=` line and change the `--config=` argument:
  ```ini
  ExecStart=/usr/local/bin/stalwart --config=/etc/stalwart/config.json
  ```
  Then reload the unit:
  ```bash
  $ sudo systemctl daemon-reload
  ```
- For init.d (typically `/etc/init.d/stalwart`), update the `DAEMON_ARGS` line similarly.

**9. Decide how to handle the recovery admin.** The recovery admin credential must be available the first time a real administrator logs in to create a proper admin account. Two options:

- **Preferred, if a test deployment was used:** the test-deployment snapshot applied in step 6 can already include an administrator account, in which case no further action is needed. Start the service normally.
- **Otherwise:** leave `STALWART_RECOVERY_ADMIN` in place until a real admin is created through the WebUI, then remove it. For systemd, set it via the environment file referenced by `EnvironmentFile=` in the service unit (the default Stalwart install creates `/etc/stalwart/stalwart.env` for exactly this purpose:  uncomment the `STALWART_RECOVERY_ADMIN` line and set the value). For init.d, export the variable in `/etc/default/stalwart` or the distribution's equivalent. Do **not** set `STALWART_RECOVERY_MODE=1`:  that is for the migration only and would put the server back into recovery mode at every restart.

**10. Start the service.**

```bash
$ sudo systemctl start stalwart     # or: sudo service stalwart start
```

Verify it comes up cleanly and is listening on its normal ports. The deployment is now on `v0.16`. For reference on how a fresh `v0.16` Linux install is expected to look, see https://stalw.art/docs/install/platform/linux.

### Option B:  Docker deployments

The new Docker image uses **different mount points** than the old one. Where the previous image mounted a single `/opt/stalwart` volume, the new image mounts two:

| Volume | Purpose |
|:---|---|
| `/etc/stalwart` | Configuration directory (contains `config.json`) |
| `/var/lib/stalwart` | Persistent application data (RocksDB, local blobs, bootstrap registry) |

The Docker migration uses the same recovery-mode pattern as the binary case: stop the old container, run a throwaway container in recovery mode, apply the snapshot, stop the throwaway, then start the real container.

> **Clustered deployments:** stop every container running `v0.15.x` before starting the migration on any node.

**1. Stop the old container.**

```bash
$ docker stop stalwart
```

**2. Prepare the new volumes.** Two named volumes (or two host directories, if bind-mounting) are required:

```bash
$ docker volume create stalwart-etc
$ docker volume create stalwart-data
```

For deployments where the embedded database holds user mail (RocksDB / SQLite), the contents of the old `/opt/stalwart/data` directory must be placed in the new `stalwart-data` volume before starting the recovery-mode container. The simplest way is a helper container:

```bash
$ docker run --rm \
    -v <OLD_STALWART_DIR>:/old \
    -v stalwart-data:/new \
    alpine sh -c 'cp -a /old/data/. /new/ && chown -R 2000:2000 /new'
```

Replace `<OLD_STALWART_DIR>` with the host path that the previous container had mounted at `/opt/stalwart`. The `chown` step is required because the new image runs as UID `2000`. For deployments that use external databases (PostgreSQL, MySQL, FoundationDB, S3, Azure, Redis, NATS), skip the copy:  the data already lives outside the container.

**3. Install `config.json` in the new config volume.**

```bash
$ docker run --rm \
    -v /path/to/local/config.json:/src/config.json:ro \
    -v stalwart-etc:/dst \
    alpine sh -c 'cp /src/config.json /dst/config.json && chown 2000:2000 /dst/config.json'
```

> **Update embedded paths inside `config.json` and `export.json` for the new mount points.** The migration script writes the on-disk paths it found in the v0.15 deployment, which on the previous Docker image typically pointed at `/opt/stalwart/data` (and `/opt/stalwart/data/blobs` for the filesystem [BlobStore](https://stalw.art/docs/ref/object/blob-store)). The new image mounts persistent data at `/var/lib/stalwart` instead, so any path referencing the old location must be rewritten before the recovery container is started; otherwise the container exits with `Permission denied: /opt/stalwart/data` because UID `2000` cannot create that directory inside the container's filesystem.
>
> The migration script ships with a `--patch-paths` flag that handles the rewrite during `convert`:
>
> ```bash
> $ python migrate_v016.py convert \
>     --settings settings.json --principals principals.json \
>     --config config.json --output export.json \
>     --patch-paths /opt/stalwart=/var/lib/stalwart
> ```
>
> `--patch-paths SOURCE=DEST` walks both emitted files and rewrites any string value beginning with the source prefix. The flag may be supplied multiple times for deployments that mount data under several legacy paths. When the script detects `/opt/stalwart` in the source settings and the flag was not passed, it prints a notice with the exact command to rerun.
>
> For deployments that already produced `config.json` and `export.json` without the flag, the equivalent in-place edit is:
>
> ```bash
> $ sed -i.bak \
>       -e 's|/opt/stalwart/data/blobs|/var/lib/stalwart/blobs|g' \
>       -e 's|/opt/stalwart/data|/var/lib/stalwart|g' \
>       config.json export.json
> $ grep -n /opt/stalwart config.json export.json   # verify clean
> ```
>
> The blob-path substitution runs first so the more general data-path rewrite does not double-rewrite it. The `.bak` files left behind by `-i.bak` are the rollback if the substitution went wrong.
>
> Skip this paragraph entirely on deployments that use external databases (PostgreSQL, MySQL, FoundationDB) and external blob backends; those deployments have no on-disk paths to rewrite.

**4. Start a temporary container in recovery mode.** This container exists only for the duration of the migration:

```bash
$ docker run -d --name stalwart-recovery \
    -e STALWART_RECOVERY_MODE=1 \
    -e STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
    -p 8080:8080 \
    -v stalwart-etc:/etc/stalwart \
    -v stalwart-data:/var/lib/stalwart \
    stalwartlabs/stalwart:v0.16
```

Only port `8080` (management API) is published:  mail ports stay closed in recovery mode. Watch the logs to confirm the migration completes successfully:

```bash
$ docker logs -f stalwart-recovery
```

Wait until the logs stop scrolling and settle on the message indicating the HTTP endpoint is listening.

**5. Apply the exported snapshot.** From the host (or any machine that can reach `http://<docker-host>:8080`):

```bash
$ export STALWART_URL=http://127.0.0.1:8080
$ export STALWART_USER=admin
$ export STALWART_PASSWORD=someTemporaryPassword
$ stalwart-cli apply --file /path/to/export.json
```

Follow with any snapshots captured from the test deployment:

```bash
$ stalwart-cli apply --file /path/to/test-deployment-snapshot.json
```

**6. Stop and remove the temporary container.**

```bash
$ docker stop stalwart-recovery
$ docker rm stalwart-recovery
```

**7. Start the production container.** Same image, without `STALWART_RECOVERY_MODE`, with all mail ports published:

```bash
$ docker run -d --name stalwart \
    --restart unless-stopped \
    -e STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
    -p 443:443 -p 8080:8080 \
    -p 25:25 -p 587:587 -p 465:465 \
    -p 143:143 -p 993:993 \
    -p 110:110 -p 995:995 \
    -p 4190:4190 \
    -v stalwart-etc:/etc/stalwart \
    -v stalwart-data:/var/lib/stalwart \
    stalwartlabs/stalwart:v0.16
```

The `STALWART_RECOVERY_ADMIN` variable is retained deliberately so that a real administrator account can still be created through the WebUI after the first login. Once a permanent admin exists, restart the container without that environment variable to remove the back-door credential. If the test deployment snapshot applied in step 5 already includes an administrator account, the variable can be omitted from this step entirely.

For reference on the standard Docker deployment, see https://stalw.art/docs/install/platform/docker.

## Step 4:  Post-migration tasks

With the server running on `v0.16`, a few follow-up actions are required to complete the upgrade.

### Log in to the admin panel

Open a browser and navigate to:

```
https://mail.example.org/admin
```

Replace `mail.example.org` with the server's hostname. Log in either with the recovery admin credential (if it is still active) or with the administrator account created via the test-deployment snapshot.

A few behavioural changes from `v0.15.x` are worth flagging before the first sign-in:

- **The WebUI is reached over HTTPS on the configured hostname only.** The OAuth, OIDC, and JMAP discovery documents `v0.16` publishes use `https://<defaultHostname>/...` exclusively in normal mode. Loading the WebUI by IP address, by container name, or over plain HTTP (for example `http://192.168.1.10:8080/admin`) will appear to load the sign-in page but will fail at the OAuth callback. Use the same hostname that was entered in Step 1 of the wizard, or that already lives on `defaultHostname` from the migrated settings.
- **`http://...:8080` is no longer the right URL for day-to-day administration.** Port `8080` carries the recovery / bootstrap HTTP listener and is intended for the migration window; once the server is running normally it stops being a valid sign-in entry point.
- **When the public HTTPS port is not `443`** (for example a reverse proxy on `:8443`, or a Docker host port mapping that diverges from the container's `443`), set the [`STALWART_HTTPS_PORT`](https://stalw.art/docs/configuration/environment-variables#public-urls) environment variable to that port and restart the server. Without it, the discovery documents will publish `https://<host>/...` (port `443` implied) and clients will be sent to a port the proxy is not listening on.
- **Plain-text mail listeners (port `587` submission, port `143` IMAP) are no longer added by default.** This is required for compliance with the PACC autoconfig draft, which only advertises implicit-TLS ports. Mail clients that were configured to connect over `587` STARTTLS will silently stop working until either the listener is recreated through the WebUI / CLI or the clients are pointed at the implicit-TLS ports (`465` for submission, `993` for IMAPS).

#### Reverse-proxy deployments

If the deployment sits behind a reverse proxy (NGINX, Traefik, Caddy, HAProxy, or similar), this is the part of the migration where proxy-related issues most often surface. The migrated `defaultHostname`, the proxy's public hostname, the proxy's listening port, and the proxy's TLS configuration all have to line up before the first sign-in completes; if any of them is off, the OAuth flow fails partway through with errors that are hard to relate back to the proxy.

The most reliable way through this step is to **bypass the proxy temporarily** for the duration of the recovery-mode `apply`, the first sign-in, and the creation of a permanent administrator. Concretely:

1. While running `stalwart-cli apply` and creating the permanent admin, point the CLI and the browser at Stalwart directly: `http://<stalwart-host>:8080` for the recovery-mode CLI session, then `https://<stalwart-host>/admin` (accepting any self-signed certificate warning) for the first WebUI sign-in.
2. Once a permanent administrator account exists and the WebUI is confirmed working, restore the reverse-proxy configuration. From this point forward, end users reach Stalwart through the proxy and the discovery documents already point at the public hostname over HTTPS.

A full description of how `v0.16` composes the published URLs, how the proxy can talk to Stalwart on either HTTP or HTTPS, and where to set `STALWART_HTTPS_PORT` for non-standard public ports lives at https://stalw.art/docs/server/reverse-proxy/overview.

### Recalculate disk quotas

Disk quotas were reset to zero during the wipe and need to be rebuilt from the actual mailbox contents. Navigate to the **Tasks** section of the admin panel and trigger the **"Recalculate disk quotas"** task. This spawns one subtask per user account, each of which scans that user's storage and updates the counter. On large deployments this may take a while to complete:  progress is visible in the Tasks panel.

### Recalculate tenant quotas (multi-tenant deployments)

Only applicable when per-tenant disk quotas are in use. After the per-account recalculation has finished for every user, trigger a second task from the Tasks panel: **"Recalculate tenant quotas"**. This rolls the per-account totals up into the tenant-level counters.

### Create a permanent administrator

If the migration was performed without a snapshot from a test deployment, the only administrative credential at this point is the recovery admin defined by `STALWART_RECOVERY_ADMIN`. This credential is a back door:  as long as the environment variable is set, the username and password it specifies can log in regardless of directory state. Create a real administrator account through **Management → Accounts**, verify the new account can log in, and then remove `STALWART_RECOVERY_ADMIN` from:

- the systemd environment file (for example `/etc/stalwart/stalwart.env`) and restart the service, or
- the init.d defaults file, and restart the service, or
- the Docker container's `-e` flag (redeploy the container without it).

### Review the rest of the configuration

The migration script converts directory, domain, storage, DKIM, and certificate state. Everything else:  SMTP listeners, mail routing, spam rules, rate limits, retention policies, ACME, authentication backends other than those listed above:  must be reviewed in the WebUI and either recreated by hand, applied from a test-deployment snapshot, or validated against the defaults that `v0.16` ships with.

## Troubleshooting

### The server fails to start after installing `config.json`

Confirm the file is valid JSON (`python -m json.tool config.json` or `jq . config.json`) and that the datastore described in it is reachable with the provided credentials. The daemon's logs will name the offending field when a field is missing or malformed.

### `stalwart-cli apply` fails partway through

Most failures come from trying to create an object whose parent does not exist yet (for example, a `DkimSignature` referencing a `Domain` that is missing from the plan). The error message names the object and the missing reference. Either edit the plan to include the missing parent, or split the `apply` into two runs using the individual snapshot files produced by the script and the test deployment.

### Recovering from a partial `apply`

`apply` runs operations in plan order and stops on the first error. When a `create` fails halfway through, every prior `create` in the same run has already been committed to the database. Re-running the same plan now fails with `primaryKeyViolation` (the objects exist) or `invalidForeignKey` (a parent that did not get created the first time is still missing).

> **Do not bulk-delete `Account` objects to recover.** The migration plan creates each account with its original v0.15 account id (the `restore-<id>` mechanism), so a migrated `Account` points at the existing v0.15 mailbox data in the data store. Deleting that `Account` schedules account destruction, which unlinks and erases all mail, calendars, and contacts stored under that id. On the community edition this runs immediately, with no retention window. Never run `delete Account` against a data store that already contains v0.15 mail.

Recovery does not require deleting accounts. An account that a partial `apply` already created is correct and is reused as-is on the next run; the only objects that need clearing are the registry-only ones that carry no mailbox data and whose re-creation would otherwise raise `primaryKeyViolation`. While the server is still in recovery mode:

```bash
$ stalwart-cli query DkimSignature --json | jq -r '.[].id' \
    | stalwart-cli delete DkimSignature --stdin
$ stalwart-cli query Certificate --json | jq -r '.[].id' \
    | stalwart-cli delete Certificate --stdin
$ stalwart-cli query Domain --json | jq -r '.[].id' \
    | stalwart-cli delete Domain --stdin
$ stalwart-cli query Tenant --json | jq -r '.[].id' \
    | stalwart-cli delete Tenant --stdin
```

`Domain` and `Tenant` hold only directory metadata and are safe to delete and recreate; `Account` is deliberately omitted. Then fix the underlying cause in `export.json` (most often a domain that fails the v0.16 hostname check, an account whose local-part contains `@`, or a stale `/opt/stalwart` path embedded by the migration script), remove from `export.json` the `create` operation for `Account` (and any other object that already committed before the failure, so re-applying it does not raise `primaryKeyViolation`), and rerun:

```bash
$ stalwart-cli apply --file export.json
```

If you must start over with the accounts as well, do not delete them: point the new deployment at an empty data store (or restore the v0.15 data-store backup) before re-running `apply`, so that destroying and recreating accounts cannot reach live mail.

If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from `main` before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing `@`, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends, and `%{file:...}%` / `%{env:...}%` macros in DKIM private keys and certificates, which are now expanded by the script instead of being passed through verbatim and aborting the `apply`).

For deployments where individual objects are easier to identify than to wipe wholesale, use `stalwart-cli query <type>` to list ids and `stalwart-cli delete <type> --ids <id>` to remove a specific one. The same warning applies: deleting an `Account` destroys the mail stored under it. Only `Domain`, `Tenant`, `DkimSignature`, and `Certificate` are safe to delete and recreate during recovery.

### Bootstrapping a real administrator from the CLI

When the WebUI is unreachable for any reason (TLS not yet in place, reverse proxy misconfigured, OAuth callback failing), the CLI is the supported escape hatch for promoting the first real administrator. Authenticate as the recovery admin and run:

```bash
$ export STALWART_URL='http://127.0.0.1:8080'
$ export STALWART_USER='admin'
$ export STALWART_PASSWORD='someTemporaryPassword'

$ stalwart-cli query Domain --fields id,name
$ stalwart-cli create account/user \
        --field name=admin \
        --field domainId=<domain-id>
$ stalwart-cli query Account --where name=admin --fields id
$ stalwart-cli update Account <account-id> \
        --field 'credentials={"0":{"@type":"Password","secret":"<NEW-PASSWORD>"}}'
$ stalwart-cli update Account <account-id> \
        --field 'roles={"@type":"Admin"}'
```

Once the new account can sign in to the WebUI, remove `STALWART_RECOVERY_ADMIN` from the service environment and restart the service.

### Common questions

- **`primaryKeyViolation` on a rerun of `apply`**: see *Recovering from a partial `apply`* above.
- **`Domain: create failed for create-N: invalidPatch | Invalid domain name`**: the domain in `export.json` does not pass the v0.16 hostname check (typically a missing or non-public TLD). Either correct the domain in v0.15 before redumping, or hand-edit the offending block in `export.json`.
- **`/admin` redirects to `http://<random>:8080/`**: fixed in v0.16.x; upgrade to the latest patch release.
- **"Recalculate disk quotas" not visible in the WebUI**: open *Tasks → Scheduled → Create task*, choose the *Quota recalculation* maintenance type at the per-account scope, and pick a near-future timestamp.

### `Data corruption detected` after migration

This error means one node in a cluster was left running on `v0.15.x` while another was being migrated, and the old node wrote records in the obsolete format into the shared database. Stop every node in the cluster, ensure every binary is on `v0.16`, and restart. If corruption persists, the logs name the offending keys and they can be removed with the `stalwart-cli delete` command.

### `/admin` (or `/account`) returns `404 Not Found`

The WebUI is delivered as a downloadable [Application](https://stalw.art/docs/applications/overview) bundle that the server fetches from `https://github.com/stalwartlabs/webui/releases/latest/` on first start, and then refreshes on a schedule. When the very first download fails, no bundle has been unpacked locally and every request to a WebUI mount path returns `404 Not Found`. This is the most common cause of "the server is running, port `8080` answers, but `/admin` returns 404" reports during the migration.

The fix is to make sure outbound HTTPS from the Stalwart host can reach GitHub's release storage (`github.com` and `objects.githubusercontent.com`). On a host that genuinely cannot reach the public internet, stage the WebUI bundle on an internal HTTPS server and update the [`resourceUrl`](https://stalw.art/docs/ref/object/application#resourceurl) field on the WebUI's [Application](https://stalw.art/docs/ref/object/application) record to point at the internal location. After the first successful download, subsequent failures are non-fatal: the previously installed bundle stays in service and `/admin` keeps working until the next successful refresh. The full description, including the precise hosts involved, is at https://stalw.art/docs/management/webui/overview#outbound-network-requirement.

### Rolling back

If the migration cannot be completed within the available maintenance window, the database backup captured in Step 2 can be restored and the old binary (preserved as `/usr/local/bin/stalwart.v015` in the example) started again. The `v0.16` binary will refuse to start a second time against a database that has already been migrated, so restoring the pre-migration backup is the only path back to `v0.15.x`.

## Questions

If any part of this migration is unclear, or if something does not behave as documented, please post in the dedicated upgrade discussion at https://support.stalw.art. Include:

- The deployment type (binary / Docker / clustered)
- The datastore backend (RocksDB / SQLite / PostgreSQL / MySQL / FoundationDB)
- The exact version being upgraded from (`stalwart --version`)
- Any error messages from the server log or the CLI, verbatim

We would rather answer a question than watch a deployment break. There is no such thing as an obvious question for a migration of this size.
