Hemera 5 has just been released and we’d like to give you an overview of why message queues are great for your Node.js application and how you can apply them easily in your code to optimise performance and maintainability.
Projects get bigger all the time, features are added along the way and you might end up with a big application (monolith). The application becomes difficult to understand and modify, difficult to implement new features and over time, code quality decreases and development speed slows down. To avoid this issue from the start, you can structure your application by Domain (Domain Driven Design), using separate services (needn’t necessarily be *micro *services) for separate logical clusters or business purposes. For example: Products, Users and the Shopping Cart. Each service has different needs and requirements: you might have a database for millions of products and highly volatile eCommerce transactions. It would not make much sense to use the same database and system resources for such different purposes and you can save a lot of money by optimising usage. Service-based architecture also helps to exchange parts in your technology stack to evolve with your requirements.
Coming from a single application, you would probably import/require
products.js and use something like products.list() to retrieve a list of
products, maybe even enhanced with pagination and other metadata. products.js
is maintained by another developer and you trust it to work as you expect it.
There are several ways to do this in a service-based architecture. We decided to
use NATS as message queue system. It’s very simple and
incredibly fast. There’s an official
NATS docker image that you can pull in
docker-compose.
There are alternatives like HTTP, RabbitMQ etc. which all have their pros and cons which we won’t discuss in this article.
NATS Remote Functions with Hemera
Hemera is a Node.js toolkit for NATS and
simplifies messaging in distributed systems. The pattern we use is called RPC —
Remote Procedure Call
(named Request & Reply
in Hemera). Instead of the applications internal products.list() example
above, the call is wrapped in a hemera function and executed by a different
service. The behaviour is exactly the same, you can use async/await, Promises or
Callbacks, only the function call changes a little bit:
hemera.act({topic: “users”, cmd: “list”})
To register the function in the distributed system, you add the topic and command to the messaging system:
hemera.add({topic: “users”, cmd: “list”}, function() { … })
“act” calls for a function, “add” registers a function on the queue.
On each service, you add available functions: users.list(), users.get(id),
product.update({}) etc. From the API (or anywhere else) you call (act) that
function with the required parameters.
The full command to register a function on the queue with hemera requires a topic, a command and the function. Further attributes are optional:
Congratulations, you wrote your service-to-service Hello World App. Let’s move on.
Joi
A neat addition to mention at this point is
Joi Schema Validation. As the name says, a
defined schema is used for input validation. The function above doesn’t accept
“Hello” as value for a but instead throws an error. There’s no need to
re-validate or check the input parameters, for example
if (!req.a) {...}, if (!req.b) {...} , which is great.
We’re working on a way to
automatically extract the Joi validation of a Hemera function into a JSDocs for
better documentation.
Improved Hemera handlers
The first step when writing something other than “Hello World” is to use a named function instead of the inline function above. To keep definition and implementation together, we export both from a single file:
In the main file (ie. index.js), those are added to the queue:
hemera.add(getProduct.params, getProduct.handler)
This pattern has worked well for us, the functions are all registered in one place. And the definition, validation rules and implementation are in a single file, which makes it easy to focus on the function. There’s no need to switch through multiple files.
Demo Time
Service to Service communication with Hemera and NATS
To demonstrate how all this works together, we have prepared a repository with all the necessary files here: hemera-nats-demo. Every service runs in a docker container, all services connect to the central NATS system, register their functions and can then be called by the API.
sequence
diagram of a distributed function call
Summary
NATS and hemera have helped us simplify a monolithic system into many different, small services. We subsequently increased our productivity, as developers are more confident while working on smaller services. Decisions and changes result in less technical impact and more user value.
Special thanks to the team behind NATS and Dustin Deus, the lead author of hemera.
We will write about unit and integration testing of distributed services in the next article. Follow us to get notified about the publication.
If you have any questions or comments, please reach out on Twitter or start a discussion on GitHub.
