Data analysis using Data Lake

Just like the scenario of fragmented logs and monitoring, fragmented data is another challenge in microservice architecture. Fragmented data poses challenges in data analytics. This data may be used for simple business event monitoring, data auditing, or even for deriving business intelligence out of the data.

Data Lake or a data hub is an ideal solution to handle such scenarios. The event-sourced architecture pattern is generally used to share state and state changes as events with an external data store. When there is a state change, microservices publish the state change as events. Interested parties may subscribe to these events and process them based on their requirements. A central event store can also subscribe to these events and store them in a big data store for further analysis.

One of the commonly followed architectures for such data handling is shown in the following diagram:

The state change events generated from the microservices, in our case, Search...