Building radical reactivity into your web application

What is reactivity?

In September 2014 The Reactive Manifesto was written by Jonas Bonér, Dave Farley, Roland Kuhn, and Martin Thompson and signed by over 32k people from the community by September 2022. It describes the changes we have seen over the years in the way software is used, developed, and run. It proposes four key properties that modern systems need to fulfill: Responsiveness, resilience, elasticity and a message driven communication system.

These provide three values: Responsiveness, which in and of itself is valuable so we don't waste our time when interacting with a system that is slow to respond. Maintainablity means that we can easily adapt our system to provide the same service in changing conditions. And extensibility, so that we can easily expand it to integrate new functionality in existing infrastructure.

Visit https://www.reactivemanifesto.org/ for a more thorough explanation.

What you'll learn

This article explains advantages and potential problems discovered when implementing a PoC for radical reactivity using the MEAN stack (MongoDB, ExpressJS, Angular, NodeJS). A focus was laid on the message driven nature of data querying and pushing, using tools such as RxJS as a base technology for a functional programming, which is already embedded into the Angular Framework, NgRx for reactive state management and a message driven, modular query system and the somewhat hidden Change Stream feature of the MongoDB to extend the philosophy of decoupled data push and query strategies (Flux) right into the database layer.

The result is the user experience of a real time application that reflects changes made by an arbitrary user with only millisecond delay in every other active client.

Strong coding guidelines ("best practices") regarding the design of all frontend logic have been adapted from previous projects and sanity checked in the concrete setting of an Angular (→ component based) frontend application. These guidelines have thus been validated, and the limitations imposed by the Angular framework identified.

Lessons learned

Designing a highly reactive application using the MEAN stack is possible, and technologies exist that tremendously facilitate such a development. They provide an opinionated approach to effectively structure your application code for modularity, adaptablity and maintainability. However, these technologies need to be understood well to be used effectively. Reactivity is often associated with a functional programming style, which can be challenging to adapt to, because code is structured in a fundamentally different way than in imperative or object oriented approaches.

When should I use this technology?

Teams must therefore decide whether their product is valuable enough to warrant the overhead that learning and applying this technology brings. Things that should be considered are (the list is not exhaustive):

• Is my team ready to learn and adapt to a new programming style?
• Is stablity, maintainability and extensibility (→ Reactivity) more important than quick initial development?
• Is an experienced lead developer available, who can initially supervise the correct application of RxJS code? This is important to avoid common anti-patterns.
• Does the product require instant updates when state changes in the backend? Is state shared across clients?

If you can answer enough of these questions with yes, then the technologies described in this article, or a subset of them, might be the way to go for you.

Technical details

The architecture from 1000 feet

Every system can be broken into parts/modules/layers that encapsulate a specific part of the functionality. Each part may reach reactivity differently, use different technology, and be faced with different challenges. The most clearly distinct parts of a MEAN application with regards to reactivity have been identified to be

• The database, which requires an automated feedback system that notifies of data changes
• The backend layer, which in this case is mostly direct pass-through between frontend and backend, providing network connectivity, auth and minor mapping and validation logic
• The store layer in the frontend, which provides data caching, reactively querying (→ subscribing to) distinct parts of the state, and a structured location for all business logic, as well as a message driven push and query system for internal operations and backend interaction.
• RxJS to reactively transport data and events in the frontend, apply mapping logic and define cascades for subsequent steps during this journey.

MongoDB

MongoDB, like any other database, is built to be as responsive as possible given it's other properties such as complexity/cost, simplicity of usage, and general feature set. The application used in the POC did not store large amounts of data, so responsiveness was never a problem.

Resilience in MongoDB can be implemented by running it as a a cluster of docker images in a container orchestration tool like Kubernetes. This way no data is lost when a container / node goes down and a flexible connection between database and backend allows reconnecting after failure and restore.

The same setup also provides elasticity, by scaling the amount of DB containers according to current load and applying a load balancer to distribute queries evenly.

MongoDB is message driven in the way that the backend code can be notified of every data change that occurred using the Change Stream feature. It can then process and act upon these events however needed.

Modularity and extensibility can be achieved by an intelligent design of documents in the database. It has been found to be useful to use an object relational mapper (ORM) like Mongoose for access and validation, and to separate object properties from object relations by using dedicated binding objects to express relations of individual objects. This separation is maintained up until the frontend store layer, to minimize cache maintenance in the client when relations change.

Backend

The backend achieves responsiveness by using functional, asynchronous coding principles which results in non-blocking code with high pass-through rates. This is possible because no expensive calculations need to be performed and requests are usually directly mapped to database objects. If your system has similar properties, NodeJS is a good choice. Most important is the way in which connectivity is implemented. By using the Websocket protocol, a symmetric connection between client and server is established, enabling both sides to initiate communication, which eliminates the need for the client to continuously query for data updates. Instead, the backend actively pushes that information the instant it becomes available.

The backend is resilient because it is largely state-less, scalable, and applies retry-strategies in case of failures of dependencies (the DB).

It is elastic for the same reasons that it is resilient.

It is message driven through the implementation of a fixed protocol for communication in both directions.

The store

The NgRx store or logic layer of the frontend is built using the data stream library RxJS which provides a highly responsive data flow primitve: The Observable. In specific events, the data no longer needs to be queried, processed and distributed actively, which introduces spaghetti-code and uncertainty. Instead, data arrives as it becomes available or changes, automatically triggers processing logic and is distributed to any part of the application that has requested it.

The store is resilient by applying the same automation facilities to errors as to common events. It remains your responsibility to handle these errors, but they arise at a predictable location and can be handled with a level of granularity that you choose.

Elasticity can be enabled by using feature stores that become active only when a certain part of your application is activated. This modular design allows the store to be easily extended as new requirements arise, without compromising performance when the feature is not active.

A message driven API is implemented by NgRx's Actions, which are used to trigger every event and communication within the app. Any number of modular logic blocks can be triggered when one or a combination of Actions has been dispatched. This includes communication over the network.

The view layer

The Angular frontend is used to display information on the page. Angular also uses RxJS internally and exposes most of it's APIs in two versions: A simple, snapshot based version for classical imperative programming, and an observable based interface for reactively responding to changes. While this design philosophy is not fully applied, yet, it allows most code to run highly responsively.

The system can be made resilient by correctly handling and recovering from errors, which are also passed along the observable event pipeline. Correct error handling is vital, yet somewhat unintuitive. To prevent logical networks from collapsing this will have to be understood by each developer working on RxJS code.

The system may be designed to be elastic using RxJS primitives such as the debounce operator to reduce the amount of processing triggered by quickly repeating events. Optimizing these bottlenecks when needed is the developer's task.

Everyithng in RxJS is message/event driven through the nature of Observables. Any event or data change can easily be made available to the entire application, and consumers decide what they are interested in.

A click's journey

Let's assume that the user is currently filling in a form that allows him to comment on a post made by another user.

While he is typing a flicker in the application occurs and the text that he is commenting on changes, as the other user just edited it to remove a typo. A warning sign pops up that informs him that the content he is commenting on has changed. He continues typing.

As he clicks the submit button a submit clicked event carrying no further data is fired. In the component class of the comment editor, this event is received, enriched with the user's ID and the input's text, and published as a comment posted event.

This event is again listened to and transformed into an NgRx action, which is a standardized and optimized format for such events.

It triggers logic that closes the edit form, increases the comments posted today counter and sends a comment published message to the backend.

In the backend, the ID is checked for validity, forbidden characters are removed from the text, and it is saved as a Comment document. Alongside, a UserCommentBinding is created, which associates the new comment with the user that posted it, and the post to which the comment was attached.

Immediately, two Change Stream events are fired and received by the backend, which report the creation of the two new documents and their contents.

This message is mapped to the transport format and pushed to all clients that are currently connected.

In each client, receiving that data triggers a update received Action that is received by logic blocks called reducers that process the data and update the internal state accordingly.

The changes in internal state trigger the RxJS message system and are forwarded to each part of the application that has subscribed to this particular part of the state, triggering processing logic defined there.

The user who wrote the comment now sees it pop up under the post. So does the user who wrote the original post, and anyone else who currently visits the same page.

But is it fun?

The greatest architecture and most ingenious algorithms may allow us to design high quality applications. But what's the use, if it takes ages to implement, is not well understood by your developers, and ends up as a great idea that was miserably implemented?

A common mantra in software development is KISS (Keep it simple, stupid!). It is meant to motivate everybody to solve a problem in a way that, while addressing it adequately, keeps the solution as simple (≠ easy) as possible.

So the question we need to ask is whether the tech stack discussed here is the simplest (≠ easiest) solution to the problem.

I believe that, while using these tools is not easy (≠ simple), we have to acknowledge that the proposed application type (→ highly reactive) is inherently complex (≠ complicated). So setting out to implement such an app will require adequate tooling, to simply find a complex solution, instead of easily finding a complicated one.

The learning curve will be steep, but the quality of the code you'll produce will improve drastically. For the first time, you will be able to find elegant solutions that allow you to leverage TypeScript's capabilities fully, even in challenging environments such as Angular's components, where messy things like @Input()s and @ViewChild()ren are ubiquitous. No more !, and ? only when the business logic requires it. The flow of data apparent on first glance, even in forms that use [(ngModel)]. And you in the middle of it, honing your network of data pipelines, instead of debugging your mess of imperative spaghetti code.

It takes time. But it's addictive. And you'll never go back!

Gimme more

The greatest challenge when trying to implement this type of project is to write proper RxJS. Unfortunately, learning RxJS is a macro task. It's not enough to study the docs and learn all the operators. You have to understand how those operators are meaningfully applied in the context of your application. Few people know this well.

Even on StackOverflow, you'll often find the quick and dirty solution, instead of the wholistic one, because it's very difficult to write a minimal working example of something that is connected to your entire application. Yet that greater view is what's required to really shine with RxJS. So it takes a lot of practice, ideally on a green playing field. Unless you are lucky enough to be working in a project that already follows the RxJS philosophy, its architecture is likely to work against you.

I have written an article series on Medium about the most common challenges when working with RxJS. It uses excerpts from my example project which shows different levels of RxJS code quality, and can serve as a template for other applications.