Make loosely coupled Applications with Apache NiFi
Once upon a time (well, three years from 2018 to be precise) one of our solutions looked like this:
We were receiving real-time feed data from Smart Electric Meters. This data was from hundreds of locations, each having tens of such meters. We needed to process this data which meant filtration, aggregation and transformation on some business rules. The end results were to be served to business users.
This system had become a success and a useful tool for the business. After a year, the use cases increased tremendously and the solution looked like below:
We started receiving data of water, gas and temperature meters too. This was huge pressure on our ingestion and processing pipeline. The real-time predictive analysis was also included in our processing pipeline. We needed to develop new workflows for real-time and batch processing. As we needed to maintain the same latency and throughput, we introduced some new elements and decoupled the system into more logical and functional modules.
Here, the Apache NiFi came to our rescue. We did a POC where we transported our pre-processing logic to Apache NiFi and made some architectural changes. Let me be clear that NiFi did not magically make the system better. However, it allows us the flexibility to make a loosely coupled system.
We made our Nginx server to send any POST request received to a Kafka topic. In NiFi, we started performing the following operations:
- Consuming the message from Kafka topic
- Storing the raw message into a database, in parallel, for audit and re-run purposes
- Downloading file from the URL received in the message
- Validating the file according to some business rules
- Filtering, splitting and attributing the data
- Distributing the files to multiple Kafka topics according to filter type and load (number of files in the queue)
- Logging using a common processor group
Alerting or calling external Rest APIs during processing was now moved to NiFi. Communication started happening through Kafka. Our daily reconciliation reports of ingestion and processing were now generated by workflows designed in NiFi. We designed our data flows in such a way that continuous real time processing of data does not stop and all the slow and non-critical operations are done in parallel.
Introducing Kafka was meant for decoupling our different applications so that they can produce and consume messages on their own speed.
We could write complex flows easily, like this one in which we are load balancing our messages to different systems on the basis of current pending sites.
We were already using Apache Storm for stream processing, so we could easily scale up on the processing part by fine tuning the flow as needed.
Over time, NiFi has helped us to enable new functionalities, add new features, build custom workflows, and debug in difficult times (thanks to data provenance).
It has become a key component of most of our products- a true Data Orchestrator.