skip to content
szczecin 53.43N 14.55E ./start-project

symfony-nats-messenger 5.8: NKey logins and community fixes

(php, symfony, nats, release-notes) 12 min read by Bartosz Pachołek

5.8.0 and 5.8.1 add NKey and JWT logins, keep failed messages in nats retry mode, keep a worker alive while its server goes away, and stop an idle worker's heap from growing. Two of those came from outside contributors.

symfony-nats-messenger 5.8.0 and 5.8.1 are out. Between them, the Symfony Messenger transport for NATS JetStream learned to log in with NKeys and JWTs, stopped losing failed messages in nats retry mode on recent Symfony releases, keeps a worker alive while its server goes away, and no longer lets an idle worker's heap creep up. Two of those changes started outside IDCT, and two more outside pull requests are open.

This post goes through what changed since 5.7, what to check when you upgrade, and who to thank. We ran the examples below on PHP 8.5.9 against NATS 2.15.2 in Docker, and the output is pasted as it came out.

Upgrading in one minute

5.8.0 adds options and refuses a few configurations that could never work; 5.8.1 leaves the public PHP API unchanged. 5.8.1 requires idct/php-nats-jetstream-client 2.29.0 or later, which Composer pulls in:

bash
composer update idct/symfony-nats-messenger

Then check four things:

  • nkey or jwt without nkey_seed. Such a transport could never connect, and now it fails when it is built, with an InvalidArgumentException naming nkey_seed. That shows even where nothing uses the transport: a plain messenger:stats exits 1. Add the seed or remove the option.
  • retry_handler: nats with Symfony's retries on. The transport now logs a warning once. Set max_retries: 0 on that transport.
  • A failed message costs one round trip more. Its NAK or TERM first checks the connection with a PING. ping_after_idle: 0 turns that off.
  • A pull that gets 503 costs one round trip more. The client asks the server about the consumer ($JS.API.CONSUMER.INFO.<stream>.<consumer>). If a user may pull but not ask for consumer info, allow that subject, or a missing consumer ends the worker only after request_timeout.

The README's upgrade notes for 5.7.1 to 5.8.0 and 5.8.0 to 5.8.1 have the details.

NKey and JWT logins

A NATS user can log in with an NKey instead of a password. The server keeps only the user's public key (U...), and the client proves it holds the matching seed (SU...) by signing a nonce the server sends. The transport has had nkey and jwt options since 4.0.0, but they never worked: it passed them to the client without a signer, and the client throws before it writes the CONNECT. The new nkey_seed option supplies the signer.

Give the transport the seed and nothing else:

yaml
framework:
    messenger:
        transports:
            orders:
                dsn: 'nats-jetstream://nats.example.com:4222/orders/orders.created'
                options:
                    nkey_seed: '%env(NATS_NKEY_SEED)%'

On the server, the user is the public key of a pair you generate, for example with nk -gen user -pubout from the natsio/nats-box image: the first line is the seed, the second the public key. This is the whole server config we ran:

text
jetstream { store_dir: /tmp/js }
authorization {
  users = [ { nkey: U...your public key... } ]
}

To see both sides, we generated two key pairs, listed only the first on the server, and ran this script against it:

php
<?php
// nkey-demo.php: log in with an NKey seed, then see what 5.8 refuses up front.
declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use IDCT\NatsMessenger\NatsTransport;
use Symfony\Component\Messenger\Envelope;

$dsn = 'nats-jetstream://127.0.0.1:14222/orders/orders.created';
$seed = getenv('NATS_NKEY_SEED');

function show(string $label, \Throwable $e): void
{
    $class = (new \ReflectionClass($e))->getShortName();
    printf("%s: %s\n  %s\n", $label, $class, wordwrap($e->getMessage(), 80, "\n  "));
}

// 1. The seed alone: the transport signs the server's nonce with it.
$transport = new NatsTransport($dsn, ['nkey_seed' => $seed]);
$transport->setup();
$transport->send(new Envelope(new \stdClass()));
foreach ($transport->get() as $envelope) {
    $transport->ack($envelope);
    echo "logged in with the seed: sent and received one message\n";
}

// 2. What can never log in is refused when the transport is built.
$refused = [
    'nkey alone' => ['nkey' => getenv('NATS_NKEY_PUBLIC')],
    'seed + password' => ['nkey_seed' => $seed, 'username' => 'a', 'password' => 'b'],
    'mistyped seed' => ['nkey_seed' => substr($seed, 0, -1) . 'X'],
];
foreach ($refused as $label => $options) {
    try {
        new NatsTransport($dsn, $options);
        echo "$label: built (unexpected)\n";
    } catch (\Throwable $e) {
        show($label, $e);
    }
}

// 3. A valid seed of a user this server does not list: the server says no.
try {
    (new NatsTransport($dsn, ['nkey_seed' => getenv('OTHER_NKEY_SEED')]))->setup();
    echo "unknown seed: logged in (unexpected)\n";
} catch (\Throwable $e) {
    show('unknown seed', $e);
}
text
logged in with the seed: sent and received one message
nkey alone: InvalidArgumentException
  The nkey option needs nkey_seed: the client must sign the server's nonce with
  the seed, so nkey alone can never connect. Put the seed (SU...) in nkey_seed and
  leave nkey unset: the public key is derived from it.
seed + password: InvalidArgumentException
  The nkey_seed option cannot be combined with a username, password or token, from
  the DSN or the options: a server with NKey users would log the connection in
  with the NKey and ignore them, even wrong ones, and on a server that sends no
  nonce the client would refuse the login, as there is nothing to sign. Remove
  them, or remove the seed.
mistyped seed: InvalidArgumentException
  The NKey seed cannot be used: Invalid trailing bits in NKey encoding. A user
  seed starts with SU, and a .creds file holds it under its BEGIN USER NKEY SEED
  line.
unknown seed: RuntimeException
  Failed to setup NATS stream 'orders': Server rejected authentication during
  connect: 'Authorization Violation'

Three of those four failures happen in the constructor, before any connection. That is deliberate. A seed next to a password would be silently ignored by a server with NKey users, so the DSN could name one user while the connection is another. None of the errors repeats the seed, and the transport does not keep its text in its options. Signing needs the sodium extension; the paragonie/sodium_compat polyfill does not replace it.

JWT users in operator mode

On a server in operator mode, a user has a JWT as well as an NKey. Pass both: jwt for the token and nkey_seed for that user's seed. The transport does not read .creds files yet, so take the two values out of one:

bash
export NATS_USER_JWT=$(awk '/BEGIN NATS USER JWT/ { getline; print }' user.creds)
export NATS_NKEY_SEED=$(awk '/BEGIN USER NKEY SEED/ { getline; print }' user.creds)

The transport keeps the JWT it was built with. When the JWT expires, the server closes the connection with User Authentication Expired, so restart the workers with the new value.

NKey support was contributed by Adrian Brajković in pull request #89 and adapted in #101, which added the build-time refusals and real NKey, JWT and token logins in the functional suite and the examples, run on every CI job.

GitHub pull request #89 by Brajk19, merged: add support for nkey authentication with the nkey_seed option

Reading .creds files directly, with a rotated JWT picked up without a restart (#104), and external signers that keep the seed out of the PHP process (#105) are planned.

Failed messages in nats retry mode

With retry_handler: nats, NATS redelivers a failed message itself after a NAK, and Symfony's retry strategy is ignored. Symfony Messenger 7.4.22 and 8.1.10 changed how a worker settles a message it sends for retry: it now acknowledges it (symfony/symfony#67038). Up to 5.7.1 the transport passed that ACK on to NATS. With Symfony's retries left on, a message whose handler failed once was gone: never redelivered, and not in a failure transport either.

Maxime Huran reported it in #90. Since 5.8.0 the transport NAKs that delivery, so NATS delivers it again. A transport in nats mode whose retry strategy still retries now logs a warning once through your logger, because Symfony would otherwise log retries it never makes. The intended setup bounds the attempts on the NATS side and switches Symfony's off:

yaml
framework:
  messenger:
    transports:
      nats_transport:
        dsn: 'nats-jetstream://localhost:4222/my-stream/my-topic'
        options:
          retry_handler: nats
          max_deliver: 5   # attempts before NATS gives up
          nak_delay: 10    # seconds before a NAK'd message comes back
        retry_strategy:
          max_retries: 0

max_deliver matters here: without it, NATS redelivers a message that always fails forever.

Losing the server mid-handler

5.8.0 also covers a server that goes away while a handler runs, which is what a failover in a cluster looks like from a worker:

  • With reconnect: true, the ACK of a message handled at that moment, and Symfony's retry copy of one that failed, go out on the connection the client recovers. The ACK used to be lost, so the message ran again after ack_wait, and the retry copy timed out and stopped the worker.
  • The new resend_on_reconnect option, off by default, sends a publish again on the recovered connection when its server went away before answering. It needs a message id, from deduplicate: true or a DeduplicationIdStamp, so that JetStream drops the copy if the first one was stored.
  • setup() now waits for a consumer that NATS is moving after a change of stream_replicas, instead of failing.

A server going away

A NATS server that is shutting down answers a pull with 503 for a consumer it has already stopped. A consumer that does not exist gets the same 503. On NATS 2.15 that window lasts about 10 ms per durable consumer on a file stream during a graceful stop, and in lame duck mode it lasts the whole grace period, 10 seconds by default. 5.8.0 read every such 503 as a missing consumer and ended the worker. This was the known issue of 5.8.0, tracked as #106.

Client 2.29.0 added diagnoseNoResponders(), and 5.8.1 calls it after a 503. A server that is staying answers the question, with the consumer or with a 404, and the 503 keeps its old meaning. A server that is going away leaves it unanswered, so that pull becomes an empty batch, and the next operation checks the connection first.

To see the difference, we started one NATS 2.15.2 server with this config:

text
jetstream { store_dir: /tmp/js }
lame_duck_duration: "30s"
lame_duck_grace_period: "10s"

Then we ran the same idle consumer on 5.8.0 and on 5.8.1. Each pulled once a second, printed a line whenever the outcome changed, and stopped on an exception, as a Symfony worker does:

php
while (microtime(true) - $start < 55) {
    try {
        $n = 0;
        foreach ($transport->get() as $_) {
            $n++;
        }
        $line = "pull ok, $n messages";
    } catch (\Throwable $e) {
        $line = (new \ReflectionClass($e))->getShortName() . ': ' . $e->getMessage();
    }
    // print $line when it changes; break after an exception
}

About five seconds in, we put the server into lame duck mode with docker kill -s USR2:

text
app-5.8.0   1.0s pull ok, 0 messages
app-5.8.0   6.0s JetStreamException: JetStream pull request ended with status 503
text
app-5.8.1   1.0s pull ok, 0 messages
app-5.8.1  15.9s ConnectionException: Socket closed by peer (EOF)

The server log shows the sequence: lame duck mode and the JetStream shutdown at once, then Closing existing clients ten seconds later. 5.8.0 ended its worker within a second, on a 503 it took for a missing consumer, while the connection was still good. 5.8.1 kept pulling empty batches through the grace period and stopped only when the server really closed the socket.

With one server there is nowhere else to go, so the run still ends. In a cluster with reconnect: true, that closed socket is the point where the client can move to another server. We did not run the cluster case for this post; the project's functional suite has a scenario that keeps a worker consuming across a server that leaves in lame duck mode.

An idle worker's memory

The second fix in 5.8.1 came from Matko Anđelinić, who found it and fixed it in pull request #114.

A pull that finds no messages ends with a JetStreamException (408) raised in the event loop. With zend.exception_ignore_args off, PHP records each frame's arguments in the exception's trace. One of those frames waited on the pull's Future, so the trace held the Future, and the Future held the exception: a reference cycle that only PHP's cycle collector can free. A failed publish, ACK, NAK, TERM, message count or setup call left the same kind of cycle. 5.8.1 passes each call to the connection as a closure that starts it, so no frame takes the Future as an argument.

Whether you were affected depends on your php.ini. zend.exception_ignore_args is 1 in php.ini-production, but PHP's own default is 0, and the official Docker images load no php.ini at all. On the machine we tested with, and in the official image:

bash
php -r 'var_dump(ini_get("zend.exception_ignore_args"));'
docker run --rm php:8.5-cli php -r \
  'echo PHP_VERSION, " ";
   var_export(ini_get("zend.exception_ignore_args"));
   echo PHP_EOL;'
text
string(1) "1"
8.5.12 '0'

To measure it, we made 60 empty pulls with default options, one a second, and recorded the heap after each. Two warm-up pulls come first, and the values go into preallocated SplFixedArrays, so recording them allocates nothing:

php
$n = (int) $pulls;
$heap = new SplFixedArray($n + 1); // preallocated: recording a value must not allocate
for ($i = 1; $i <= $n; $i++) {
    foreach ($transport->get() as $_) {} // the pull ends empty after max_batch_timeout
    $heap[$i] = memory_get_usage() - $base;
}
$collected = gc_collect_cycles();

We ran it four times in parallel, with both releases on client 2.29.0, so only the transport differed:

Release exception_ignore_args Heap growth, 60 pulls Objects in cycles Growth after gc_collect_cycles()
5.8.0 0 752,800 B 2,100 4,480 B
5.8.0 1 4,480 B 0 4,480 B
5.8.1 0 4,480 B 0 4,480 B
5.8.1 1 4,480 B 0 4,480 B

Chart: an idle worker's heap over 60 empty pulls. With symfony-nats-messenger 5.8.0 it grows to 752,800 bytes; with 5.8.1 it stays at 4,480 bytes

That is about 12.5 KB and 35 objects per empty pull, close to the 13 KB and 37 objects Matko measured with a full Symfony Worker. The memory is not lost for good: the cycle collector frees it once its root buffer fills, or after the next handled message on Symfony 7.3 and later. But an idle worker can sit on it for tens of minutes, and the process keeps the peak it grew to. With 5.8.1 the heap settles after the first pulls and stays flat.

GitHub pull request #114 by matkomat, merged: an empty pull leaves no reference cycle behind

Pull request #114 came with a test for every operation that waits on the connection, plus one that pins the order of the call and the connection check. The release also records a rule for such tests: an exception passed to Future::error() is created before the wait, so a test built that way passes on the broken code too. A test of what an exception leaves behind has to raise it inside the event loop (EventLoop::defer()).

Contributions

Outside contributors shaped this cycle:

  • Adrian Brajković contributed NKey logins (#89). He also opened #100, per-message subjects with a NatsSubjectStamp, and reported #109: WorkerMessageFailedEvent::willRetry() always gives the same answer with retry_handler: nats.
  • Matko Anđelinić found and fixed the idle worker's reference cycle (#114, #115).
  • Maxime Huran reported the lost message in nats retry mode (#90), which became the main fix of 5.8.0.
  • Petar Fedorovsky opened #113, an opt-in mode that NAKs with Symfony's retry delay in nats mode and leaves a message whose retries ran out in the stream, building on #109.

Thank you, all of you. #100 and #113 are open. Reviewing #100 turned up three issues, filed as #110 to #112: the stamp applies on every transport a message is routed to, a setup() with another configuration can narrow the consumer's filters and lose stamped messages, and on NATS 2.9 setup() with additional_subjects leaves the consumer unfiltered before it fails.

Testing

5.8.1 has 1068 unit tests at 99.80% statement coverage. Infection kills 1179 of 1197 mutants (98.5%), and docs/TESTS.md lists the 18 survivors with the reason for each. CI now also runs the unit tests with random pauses (composer test:unit:paused), so that no test can come to depend on how fast its process runs.

The transport is free under the MIT license. The symfony-nats-messenger software page has the overview, and the source, issues and full changelog are on GitHub. It runs on the php-nats-jetstream-client.