Whenever you are in a very specific need of spreading a single RabbitMQ queue which you cannot reconfigure as it is your data-source from a 3rd party provider so you cannot set the streams or delivery groups, restreaming it to multiple clients to solve for example a problem of having multiple environments or simply delivering messages also to your developers you of course have multiple choices how to implement it. At some point in history I have been using a simple Redis Pub/Sub proxy tool which you can also find on my private GitHub rabbit-2-redis yet it caused me to lose the backlog of messages: anything not read was lost. This was especially problematic when I needed to do a restart or deployment which caused a short downtime.

So my environment was set up like this:

Setup of the environment, Rabbit source, tool re-sending to redis from which clients subscribe.

Redis has the benefit of being very snappy with minimum configuration, the default one is already fine enough for most production environments, yet due to the previously mentioned issue I was looking for a different approach. I also wanted to have a bit more control over authorization and access limitation.

Tried with Apache Kafka and Pulsar which both are great, but both require much more effort regarding DevOps configuration and maintenance which is not a major problem itself, yet when one of your problems is lack of enough resources in that area of your business you keep looking for a much simpler solution which will remove your dependency on what you have limited access to.

I came across NATS (NATS.io which existence was new to me. Decided to give it a try. I have made myself a small benchmark which results I will share in another article, yet to make it short: they were more than amazing + has users and permissions control and persistence layer: JetStream. Out of the box installation seemed enough for production-grade performance and stability which won for me, yet it had one serious issue: lack of RabbitMQ source provider (RabbitMQ plugin).

Since NATS is an open protocol and has libraries for most popular programming languages made a simple tool: rabbit-going-nats which already proven itself for us in a production environment, utilizes minimum of resources and can be executed in Docker or natively on any glibc-based linux by just execution of the binary. It probably can run on Windows (even without WSL2, works within it) and MacOSX if compiled for these systems or as a .NET executable. You can compile it to a .NET CLR application running on top of .NET or a native binary thanks to the NativeAOT compilation.

In a perfect world I should have made a plugin for NATS.io itself using its internal API, yet at the time of me looking for the solution that would require too much time and effort, yet possibly that will happen in the future and then this tool may become obsolete.

If useful for you check it out: rabbit-going-nats on GitHub

Rabbit-Going-NATS

Allows to pass-through messages fetched from RabbitMQ's queue to NATS PubSub/JetStream. Useful if you receive a data feed through RabbitMQ, but you need to redistribute it further to multiple clients in the most efficient way. Allows to create an infrastructure as in the diagram below:

Setup schema using the tool and NATS.

Installation

Application supports any .NET compatible environment. May be executed as .NET CLR application with .NET 8+ Runtime or precompiled to a standalone native binary using the NativeAOT functionality.

Repository features two precompiled applications:

Native Linux AMD64 (as this is the most popular environment)
Native Linux ARM64 (as this is my environment)

You may compile for your environment using simply dotnet publish command.

To run the application just: 1) compile yourself or download the binary from latest Release. 2) place the binary in a desired folder. 3) create appsettings.json as instructed below.

NOTE: I personally suggest to run this using supervisor to be sure it gets restarted in case of any failure.

Configuration

Application expects configuration sections file to be present in the config file, standard appsettings.json in the workdir:

{
    "RabbitMq": {
        "HostName": "localhost",
        "UserName": "user",
        "Port": 5000,
        "Password": "password",
        "VirtualHost": "",
        "QueueName": "test"
    },
    "Nats": {
        "Url": "nats://localhost:4222",
        "Subject": "test-subject",
        "Secret": "s3cr3t"
    }
}

You may also use username and password:

{
    "Nats": {
        "Url": "nats://localhost:4222",
        "Username": "<your username>",
        "Password": "<your password>"
    }
}

VirtualHost defaults to "/". If your NATS server does not have any protection leave Secret and Username/Password fields empty.

This file may be used also to set config details for Nlog Logging.

For example adding there:

{
  "Logging": {
      "NLog": {
          "IncludeScopes": false,
          "ParseMessageTemplates": true,
          "CaptureMessageProperties": true
      }
  },
  "NLog": {
      "autoreload": true,
      "internalLogLevel": "Debug",
      "internalLogFile": "/var/log/rabbit-going-nats.log",
      "throwConfigExceptions": true,
      "targets": {
          "console": {
              "type": "Console",
              "layout": "${date}|${level:uppercase=true}|${message} ${exception:format=tostring}|${logger}|${all-event-properties}"
          },
          "file": {
              "type": "AsyncWrapper",
              "target": {
                  "wrappedFile": {
                      "type": "File",
                      "fileName": "/var/log/rabbit-going-nats.log",
                      "layout": {
                          "type": "JsonLayout",
                          "Attributes": [
                              { "name": "timestamp", "layout": "${date:format=o}" },
                              { "name": "level", "layout": "${level}" },
                              { "name": "logger", "layout": "${logger}" },
                              { "name": "message", "layout": "${message:raw=true}" },
                              { "name": "properties", "encode": false, "layout": { "type": "JsonLayout", "includeallproperties": "true" } }
                          ]
                      }
                  }
              }
          }
      },
      "rules": [
          {
              "logger": "*",
              "minLevel": "Trace",
              "writeTo": "File,Console"
          }
      ]
  }
}

Will cause logging of everything in Debug mode to /var/log/rabbit-going-nats.log.

NOTE: Be sure to set permissions to the log file, user which runs the application must be able to write to it.

You may use different Nlog targets, be sure to check Nlogs config reference: https://nlog-project.org/config/

Warning: this build includes Sentry.IO logging target support.

Any form of testing at the moment?

If you have Docker installed you may launch first:

setup-test.sh which will create an instance of RabbitMQ, queue and NATS.
build: dotnet publish
run the binary with config from appsettings.json.dist (place it in the same folder and rename ot appsettings.json).
execute run-test.sh which will send few messages to the queue.

Proper tests are in the TODOs.

TODO

Unit Tests.
Functional tests.
Utilize the stopping token.
Support multiple sources.
Support more authorization methods.
Strip ExecuteAsync into proper sections, submethods, organise code better.
Option to support scripted events: for example to execute a command or different application when connection gets regained.

Contribution

Feel free to create any issues or pull requests, they are more than welcome!

Link

rabbit-going-nats on GitHub

Rabbit-going-NATS