symfony-nats-messenger 5.8: NKey logins and community fixes
(php, symfony, nats, release-notes) 12 min read by Bartosz Pachołek
5.8.0 and 5.8.1 add NKey and JWT logins, keep failed messages in nats retry mode, keep a worker alive while its server goes away, and stop an idle worker's heap from growing. Two of those came from outside contributors.
symfony-nats-messenger 5.8.0 and 5.8.1 are out. Between them, the Symfony Messenger transport for NATS JetStream learned to log in with NKeys and JWTs, stopped losing failed messages in nats retry mode on recent Symfony releases, keeps a worker alive while its server goes away, and no longer lets an idle worker's heap creep up. Two of those changes started outside IDCT, and two more outside pull requests are open.
This post goes through what changed since 5.7, what to check when you upgrade, and who to thank. We ran the examples below on PHP 8.5.9 against NATS 2.15.2 in Docker, and the output is pasted as it came out.
Upgrading in one minute
5.8.0 adds options and refuses a few configurations that could never work; 5.8.1 leaves the public PHP API unchanged. 5.8.1 requires idct/php-nats-jetstream-client 2.29.0 or later, which Composer pulls in:
composer update idct/symfony-nats-messenger
Then check four things:
nkeyorjwtwithoutnkey_seed. Such a transport could never connect, and now it fails when it is built, with anInvalidArgumentExceptionnamingnkey_seed. That shows even where nothing uses the transport: a plainmessenger:statsexits 1. Add the seed or remove the option.retry_handler: natswith Symfony's retries on. The transport now logs a warning once. Setmax_retries: 0on that transport.- A failed message costs one round trip more. Its NAK or TERM first checks the connection with a PING.
ping_after_idle: 0turns that off. - A pull that gets 503 costs one round trip more. The client asks the server about the consumer (
$JS.API.CONSUMER.INFO.<stream>.<consumer>). If a user may pull but not ask for consumer info, allow that subject, or a missing consumer ends the worker only afterrequest_timeout.
The README's upgrade notes for 5.7.1 to 5.8.0 and 5.8.0 to 5.8.1 have the details.
NKey and JWT logins
A NATS user can log in with an NKey instead of a password. The server keeps only the user's public key (U...), and the client proves it holds the matching seed (SU...) by signing a nonce the server sends. The transport has had nkey and jwt options since 4.0.0, but they never worked: it passed them to the client without a signer, and the client throws before it writes the CONNECT. The new nkey_seed option supplies the signer.
Give the transport the seed and nothing else:
framework:
messenger:
transports:
orders:
dsn: 'nats-jetstream://nats.example.com:4222/orders/orders.created'
options:
nkey_seed: '%env(NATS_NKEY_SEED)%'
On the server, the user is the public key of a pair you generate, for example with nk -gen user -pubout from the natsio/nats-box image: the first line is the seed, the second the public key. This is the whole server config we ran:
jetstream { store_dir: /tmp/js }
authorization {
users = [ { nkey: U...your public key... } ]
}
To see both sides, we generated two key pairs, listed only the first on the server, and ran this script against it:
<?php
// nkey-demo.php: log in with an NKey seed, then see what 5.8 refuses up front.
declare(strict_types=1);
require __DIR__ . '/vendor/autoload.php';
use IDCT\NatsMessenger\NatsTransport;
use Symfony\Component\Messenger\Envelope;
$dsn = 'nats-jetstream://127.0.0.1:14222/orders/orders.created';
$seed = getenv('NATS_NKEY_SEED');
function show(string $label, \Throwable $e): void
{
$class = (new \ReflectionClass($e))->getShortName();
printf("%s: %s\n %s\n", $label, $class, wordwrap($e->getMessage(), 80, "\n "));
}
// 1. The seed alone: the transport signs the server's nonce with it.
$transport = new NatsTransport($dsn, ['nkey_seed' => $seed]);
$transport->setup();
$transport->send(new Envelope(new \stdClass()));
foreach ($transport->get() as $envelope) {
$transport->ack($envelope);
echo "logged in with the seed: sent and received one message\n";
}
// 2. What can never log in is refused when the transport is built.
$refused = [
'nkey alone' => ['nkey' => getenv('NATS_NKEY_PUBLIC')],
'seed + password' => ['nkey_seed' => $seed, 'username' => 'a', 'password' => 'b'],
'mistyped seed' => ['nkey_seed' => substr($seed, 0, -1) . 'X'],
];
foreach ($refused as $label => $options) {
try {
new NatsTransport($dsn, $options);
echo "$label: built (unexpected)\n";
} catch (\Throwable $e) {
show($label, $e);
}
}
// 3. A valid seed of a user this server does not list: the server says no.
try {
(new NatsTransport($dsn, ['nkey_seed' => getenv('OTHER_NKEY_SEED')]))->setup();
echo "unknown seed: logged in (unexpected)\n";
} catch (\Throwable $e) {
show('unknown seed', $e);
}
logged in with the seed: sent and received one message
nkey alone: InvalidArgumentException
The nkey option needs nkey_seed: the client must sign the server's nonce with
the seed, so nkey alone can never connect. Put the seed (SU...) in nkey_seed and
leave nkey unset: the public key is derived from it.
seed + password: InvalidArgumentException
The nkey_seed option cannot be combined with a username, password or token, from
the DSN or the options: a server with NKey users would log the connection in
with the NKey and ignore them, even wrong ones, and on a server that sends no
nonce the client would refuse the login, as there is nothing to sign. Remove
them, or remove the seed.
mistyped seed: InvalidArgumentException
The NKey seed cannot be used: Invalid trailing bits in NKey encoding. A user
seed starts with SU, and a .creds file holds it under its BEGIN USER NKEY SEED
line.
unknown seed: RuntimeException
Failed to setup NATS stream 'orders': Server rejected authentication during
connect: 'Authorization Violation'
Three of those four failures happen in the constructor, before any connection. That is deliberate. A seed next to a password would be silently ignored by a server with NKey users, so the DSN could name one user while the connection is another. None of the errors repeats the seed, and the transport does not keep its text in its options. Signing needs the sodium extension; the paragonie/sodium_compat polyfill does not replace it.
JWT users in operator mode
On a server in operator mode, a user has a JWT as well as an NKey. Pass both: jwt for the token and nkey_seed for that user's seed. The transport does not read .creds files yet, so take the two values out of one:
export NATS_USER_JWT=$(awk '/BEGIN NATS USER JWT/ { getline; print }' user.creds)
export NATS_NKEY_SEED=$(awk '/BEGIN USER NKEY SEED/ { getline; print }' user.creds)
The transport keeps the JWT it was built with. When the JWT expires, the server closes the connection with User Authentication Expired, so restart the workers with the new value.
NKey support was contributed by Adrian Brajković in pull request #89 and adapted in #101, which added the build-time refusals and real NKey, JWT and token logins in the functional suite and the examples, run on every CI job.

Reading .creds files directly, with a rotated JWT picked up without a restart (#104), and external signers that keep the seed out of the PHP process (#105) are planned.
Failed messages in nats retry mode
With retry_handler: nats, NATS redelivers a failed message itself after a NAK, and Symfony's retry strategy is ignored. Symfony Messenger 7.4.22 and 8.1.10 changed how a worker settles a message it sends for retry: it now acknowledges it (symfony/symfony#67038). Up to 5.7.1 the transport passed that ACK on to NATS. With Symfony's retries left on, a message whose handler failed once was gone: never redelivered, and not in a failure transport either.
Maxime Huran reported it in #90. Since 5.8.0 the transport NAKs that delivery, so NATS delivers it again. A transport in nats mode whose retry strategy still retries now logs a warning once through your logger, because Symfony would otherwise log retries it never makes. The intended setup bounds the attempts on the NATS side and switches Symfony's off:
framework:
messenger:
transports:
nats_transport:
dsn: 'nats-jetstream://localhost:4222/my-stream/my-topic'
options:
retry_handler: nats
max_deliver: 5 # attempts before NATS gives up
nak_delay: 10 # seconds before a NAK'd message comes back
retry_strategy:
max_retries: 0
max_deliver matters here: without it, NATS redelivers a message that always fails forever.
Losing the server mid-handler
5.8.0 also covers a server that goes away while a handler runs, which is what a failover in a cluster looks like from a worker:
- With
reconnect: true, the ACK of a message handled at that moment, and Symfony's retry copy of one that failed, go out on the connection the client recovers. The ACK used to be lost, so the message ran again afterack_wait, and the retry copy timed out and stopped the worker. - The new
resend_on_reconnectoption, off by default, sends a publish again on the recovered connection when its server went away before answering. It needs a message id, fromdeduplicate: trueor aDeduplicationIdStamp, so that JetStream drops the copy if the first one was stored. setup()now waits for a consumer that NATS is moving after a change ofstream_replicas, instead of failing.
A server going away
A NATS server that is shutting down answers a pull with 503 for a consumer it has already stopped. A consumer that does not exist gets the same 503. On NATS 2.15 that window lasts about 10 ms per durable consumer on a file stream during a graceful stop, and in lame duck mode it lasts the whole grace period, 10 seconds by default. 5.8.0 read every such 503 as a missing consumer and ended the worker. This was the known issue of 5.8.0, tracked as #106.
Client 2.29.0 added diagnoseNoResponders(), and 5.8.1 calls it after a 503. A server that is staying answers the question, with the consumer or with a 404, and the 503 keeps its old meaning. A server that is going away leaves it unanswered, so that pull becomes an empty batch, and the next operation checks the connection first.
To see the difference, we started one NATS 2.15.2 server with this config:
jetstream { store_dir: /tmp/js }
lame_duck_duration: "30s"
lame_duck_grace_period: "10s"
Then we ran the same idle consumer on 5.8.0 and on 5.8.1. Each pulled once a second, printed a line whenever the outcome changed, and stopped on an exception, as a Symfony worker does:
while (microtime(true) - $start < 55) {
try {
$n = 0;
foreach ($transport->get() as $_) {
$n++;
}
$line = "pull ok, $n messages";
} catch (\Throwable $e) {
$line = (new \ReflectionClass($e))->getShortName() . ': ' . $e->getMessage();
}
// print $line when it changes; break after an exception
}
About five seconds in, we put the server into lame duck mode with docker kill -s USR2:
app-5.8.0 1.0s pull ok, 0 messages
app-5.8.0 6.0s JetStreamException: JetStream pull request ended with status 503
app-5.8.1 1.0s pull ok, 0 messages
app-5.8.1 15.9s ConnectionException: Socket closed by peer (EOF)
The server log shows the sequence: lame duck mode and the JetStream shutdown at once, then Closing existing clients ten seconds later. 5.8.0 ended its worker within a second, on a 503 it took for a missing consumer, while the connection was still good. 5.8.1 kept pulling empty batches through the grace period and stopped only when the server really closed the socket.
With one server there is nowhere else to go, so the run still ends. In a cluster with reconnect: true, that closed socket is the point where the client can move to another server. We did not run the cluster case for this post; the project's functional suite has a scenario that keeps a worker consuming across a server that leaves in lame duck mode.
An idle worker's memory
The second fix in 5.8.1 came from Matko Anđelinić, who found it and fixed it in pull request #114.
A pull that finds no messages ends with a JetStreamException (408) raised in the event loop. With zend.exception_ignore_args off, PHP records each frame's arguments in the exception's trace. One of those frames waited on the pull's Future, so the trace held the Future, and the Future held the exception: a reference cycle that only PHP's cycle collector can free. A failed publish, ACK, NAK, TERM, message count or setup call left the same kind of cycle. 5.8.1 passes each call to the connection as a closure that starts it, so no frame takes the Future as an argument.
Whether you were affected depends on your php.ini. zend.exception_ignore_args is 1 in php.ini-production, but PHP's own default is 0, and the official Docker images load no php.ini at all. On the machine we tested with, and in the official image:
php -r 'var_dump(ini_get("zend.exception_ignore_args"));'
docker run --rm php:8.5-cli php -r \
'echo PHP_VERSION, " ";
var_export(ini_get("zend.exception_ignore_args"));
echo PHP_EOL;'
string(1) "1"
8.5.12 '0'
To measure it, we made 60 empty pulls with default options, one a second, and recorded the heap after each. Two warm-up pulls come first, and the values go into preallocated SplFixedArrays, so recording them allocates nothing:
$n = (int) $pulls;
$heap = new SplFixedArray($n + 1); // preallocated: recording a value must not allocate
for ($i = 1; $i <= $n; $i++) {
foreach ($transport->get() as $_) {} // the pull ends empty after max_batch_timeout
$heap[$i] = memory_get_usage() - $base;
}
$collected = gc_collect_cycles();
We ran it four times in parallel, with both releases on client 2.29.0, so only the transport differed:
| Release | exception_ignore_args |
Heap growth, 60 pulls | Objects in cycles | Growth after gc_collect_cycles() |
|---|---|---|---|---|
| 5.8.0 | 0 | 752,800 B | 2,100 | 4,480 B |
| 5.8.0 | 1 | 4,480 B | 0 | 4,480 B |
| 5.8.1 | 0 | 4,480 B | 0 | 4,480 B |
| 5.8.1 | 1 | 4,480 B | 0 | 4,480 B |

That is about 12.5 KB and 35 objects per empty pull, close to the 13 KB and 37 objects Matko measured with a full Symfony Worker. The memory is not lost for good: the cycle collector frees it once its root buffer fills, or after the next handled message on Symfony 7.3 and later. But an idle worker can sit on it for tens of minutes, and the process keeps the peak it grew to. With 5.8.1 the heap settles after the first pulls and stays flat.

Pull request #114 came with a test for every operation that waits on the connection, plus one that pins the order of the call and the connection check. The release also records a rule for such tests: an exception passed to Future::error() is created before the wait, so a test built that way passes on the broken code too. A test of what an exception leaves behind has to raise it inside the event loop (EventLoop::defer()).
Contributions
Outside contributors shaped this cycle:
- Adrian Brajković contributed NKey logins (#89). He also opened #100, per-message subjects with a
NatsSubjectStamp, and reported #109:WorkerMessageFailedEvent::willRetry()always gives the same answer withretry_handler: nats. - Matko Anđelinić found and fixed the idle worker's reference cycle (#114, #115).
- Maxime Huran reported the lost message in nats retry mode (#90), which became the main fix of 5.8.0.
- Petar Fedorovsky opened #113, an opt-in mode that NAKs with Symfony's retry delay in nats mode and leaves a message whose retries ran out in the stream, building on #109.
Thank you, all of you. #100 and #113 are open. Reviewing #100 turned up three issues, filed as #110 to #112: the stamp applies on every transport a message is routed to, a setup() with another configuration can narrow the consumer's filters and lose stamped messages, and on NATS 2.9 setup() with additional_subjects leaves the consumer unfiltered before it fails.
Testing
5.8.1 has 1068 unit tests at 99.80% statement coverage. Infection kills 1179 of 1197 mutants (98.5%), and docs/TESTS.md lists the 18 survivors with the reason for each. CI now also runs the unit tests with random pauses (composer test:unit:paused), so that no test can come to depend on how fast its process runs.
The transport is free under the MIT license. The symfony-nats-messenger software page has the overview, and the source, issues and full changelog are on GitHub. It runs on the php-nats-jetstream-client.