Enterprise Integration has existed for many, many years. Although it might seem like an old set of patterns, the reality is that more and more data silos, protocols, and systems have been created in recent times, which increases the need for the capabilities of an Enterprise Integration platform.
Orchestra
Orchestra is a module within GraphAware Hume that helps address complex data integration and orchestration challenges.
The natural human approach taken to solve a problem is to decompose it into simple steps, adapt some of them, get help or advice from others, and orchestrate all those steps in the correct manner to achieve a final result.
The key concept in the previous definition is the verb “orchestrate” (which is also the reasoning behind the name Orchestra), and it’s exactly the approach Orchestra uses to help us achieve the desired result.
The power of Orchestra is its simplicity: Orchestra allows the integration of different types of data coming from multiple data sources (like S3 buckets, Azure BlobStorage, third-party APIs, RDBMS tables, and so on). Also, it allows for normalization of ingested data, transforming it if necessary, and processing it with special services called Skills. Skills are where magic happens: unstructured data is transformed into knowledge.
Workflow
The key element of Orchestra is the workflow. Think of it as a set of processing steps called components. Each component reads incoming data (structured or unstructured), processes it, and produces an enriched version of the input. Together, these steps form a continuous knowledge enrichment pipeline.
A concrete example
A financial institution is storing data in different formats across a multitude of silos (RSS feeds, JSON files on S3 buckets, CSV on Azure BlobStorage and third party APIs) and needs to retrieve key information about the top revenue coming from their products promoted in different countries and store the results of the analysis in the form of a graph in Neo4j in order to provide a 360-degree customer view.
The above picture represents the integration of many heterogeneous data silos into one unified form that will be used to create a knowledge graph around it.
Enterprise data integration is not a trivial task. Orchestra abstracts the complexity of different connection mechanisms and protocols, as well as the orchestration between them, so humans can focus on the value of data transformation and enrichment.
To summarise, Orchestra lets you create complex workflows in minutes with drag-and-drop instead of weeks of programming.
Orchestra can handle any amount of data and leverages cloud infrastructure constructs such as Kubernetes.
If you think nothing will go wrong, it will. Several issues can happen during the lifecycle of a workflow.
For example, the workflow illustrated above includes an RSS Feed component as a source input. The following non-exhaustive list enumerates various situations where an error can occur :
- The text size of the feed content exceeds the Neo4j max size of text with a particular indexing strategy
- The RSS feed becomes unreachable
- The virtual machine on which Orchestra is installed runs out of disk space or memory
Errors can be predictable or unpredictable, but they have, unfortunately, a common factor:
They impact a critical operational capability of the business.
Orchestra provides components that can manage errors (failure handlers). They use retry logic to send alerts to several channels, like email, Slack, and so on. Failure handlers can also be configured as a chain, adding redundancy to error handling.
Error handling is, of course, the first layer of defense in application reliability, but is limited to logic executed during the application’s runtime. When an application shuts down for an unmanaged reason, external systems need to know about this event to take appropriate action. This is where “system observability” comes into place.
System observability
Metrics are a numeric representation of data measured over intervals of time. Metrics can harness the power of mathematical modeling and prediction to derive knowledge of the behavior of a system over intervals of time in the present and future.
Distributed Systems Observability, Cindy Sridharan
This makes metrics well suited to building dashboards that reflect historical trends, with metrics, traces, and logs as the three pillars.
The next section explains how Orchestra fits in a system-wide monitoring infrastructure.
Hume monitoring
GraphAware Hume includes a module called “Hume Monitoring”, which addresses these challenges. It uses Prometheus as the time-series database and Grafana as a metrics watcher.
Prometheus supports key metric types, including counters, gauges, histograms, and summaries.
In the Hume Monitoring module, new custom metrics have been introduced to represent the concept of workflows.
Metric customizations allow Hume Monitoring to produce a set of dynamic values, like the name of the workflow and the kind of event produced (error or message), fitting perfectly with the concept of templating and repetition in Grafana.
A new panel is automatically generated every time a new workflow is started in Orchestra, and removed when the workflow is stopped.
Hume Monitoring includes customizable dashboards with extremely useful indicators about:
- System loads
- CPU utilization
- Request throughput
- Number of workflows running across all Orchestra instances
- A detailed view for each running workflow that shows messages and errors over time
The biggest benefit of adopting this approach is that we have a single point where we can verify how Orchestra is performing and be alerted when something goes wrong, regardless of the physical location of Orchestra and whether it’s running or not.
Here are some dashboard examples:
In the picture below, we can easily identify how many workflows are currently running, the total number of errors and messages for all workflows. For each workflow, a linear graph is displayed for the number of messages/errors over time.
In the following dashboard, we can see the impact in terms of resources consumed, number of requests per second, JVM statistics, CPU utilization, and error logs.
Monitoring is an essential module for the entire lifecycle of an infrastructure. It gives insights into our running applications, how they’re being utilized, and how they’re growing. Thanks to Hume Monitoring, we can prevent problems, take action immediately, and ensure our business doesn’t suffer losses.
Conclusion
Learn how GraphAware Hume helps you build knowledge from connected data while monitoring workflows to identify unexpected errors.
