This is the second post in a two-part series on monitoring the Neo4j graph database with popular enterprise solutions such as Prometheus and Grafana. Monitoring the status and performance of connected data processes is crucial to deploying graph-based applications. In part 1, we saw how to expose the graph database internals and custom metrics to Prometheus, where they are stored as multi-dimensional time series.
Now it’s time to query those metrics and render results in a beautiful, integrated Grafana dashboard. This will help you establish 24/7 monitoring and alerting for your Neo4j setup so you can respond to operational events, gain useful insights to develop faster, better solutions to any bottleneck or incident, and maintain stable performance and good graph health.
Get ready, we are going on a journey through a forest of metrics, chart types, panel configurations, and query tricks. Many examples and solutions tailored for Neo4j will help us stay on the path, and eventually, we are going to land on a great Neo4j Dashboard, as masters of Neo4j monitoring.
The Grafana connection
Grafana is a popular, powerful open-source time-series analytics platform. For production use, we suggest you check out Grafana Cloud, a fully managed SaaS platform. For testing, you can install and run the Grafana server locally (instructions here).
Grafana, like Neo4j, also has built-in support for Prometheus, so you just need to add a Prometheus data source. To do so, open the Grafana UI at localhost:3000/datasources, add a new source, choose Prometheus from the option list, and fill in the Prometheus server URL (e.g. http://localhost:9090).
To verify everything works as expected, use the Explore page from the Grafana sidebar menu, select the Prometheus data source, and pick a query from the Neo4j namespace. The time series you picked should be rendered instantly in a beautiful chart, and you can even benefit from Grafana’s suggestions, such as Time series is monotonously increasing. Fix by adding rate().
Query Neo4j metrics with PromQL in Grafana dashboards
Now everything is in place. Neo4j and our procedure expose metrics to Prometheus, which stores them, and Grafana queries them using PromQL. Prometheus Query Language allows you to select, filter, aggregate, and perform vector operations on time series data in real time.
Following the documentation, which can be found here and here, we are going to craft a new dashboard, one piece at a time. Start by creating a new blank dashboard using the plus button in the left bar. A new blank panel is added by default, and you can add a query to it.
Up
Let’s start with the most basic, but important query: up. Type up in the query editor; it will render as a line chart by default. If you didn’t remove the default Prometheus scrape configuration, the first thing you will notice is that there are two lines, one for each job (neo4j-prometheus and prometheus itself). Type up{job="neo4j-prometheus"} to see only the Neo4j job series. In Prometheus, “every time series is uniquely identified by its metric name and a set of key-value pairs, also known as labels” (from Prometheus documentation). job is a built-in label, and we set its values in the YML configuration above.
The Prometheus Up metric returns 1 if the instance is reachable, or 0 if the scrape failed. The resulting line chart shows the time dimension well, but it’s not well suited to this kind of binary value. So go to the visualization tab (left side) of the panel editor and choose Singlestat visualization type, select Current from the stat dropdown, add some text mapping so that 1 becomes UP and 0 becomes DOWN. You can also colorise the displayed text by using suitable threshold levels such as 1,1.
Now the result looks clear and shiny, and when you or someone else opens this dashboard in times of need, they will get the database status immediately. And if you want to keep the time dimension, just add a spark to this visualization.
Counters
Let’s now add another basic query: a counter. Create a new panel and type neo4j_ids_in_use_node in the query editor (here, the job filter is not needed since this metric is Neo4j-specific). This metric gives us a good proxy for the total number of nodes stored in the database, which is something very useful to monitor. You will notice that when you add a query to a visualization, the legend is not very readable, being something like metric-name{job="<job-name>", instance="<instance-id>"} by default. You can easily display the metric name only using the built-in __name__ label by typing {{__name__}} in the legend template field.
Note that you can inspect all the available labels by using the Query Inspector on the right side of the query panel. If you expand the result object, you will find the metric object, looking something like this:
"metric": {
"__name__": "neo4j_ids_in_use_node",
"instance": "localhost:2004",
"job": "Neo4-Prometheus"
}
Now we could create similar panels to add the other metrics from ids_in_use namespace, which includes relationship, relationship_type, and property counts. Or we can create new queries within the same panel, which would result in three more series – three new lines – and Grafana will automatically assign different colors to each. Either of these options is good and gives you full control over each query. But there is also another way to get multiple metrics in the same chart, by using a regex expression matching in metric names, leveraging their namespace structure. The syntax for this is very straightforward: {__name__=~"neo4j_ids_in_use_.+"}.
Legends
You can go even fancier with legends using Prometheus’ label_replace() function and rewriting your query like this:
label_replace({__name__=~"neo4j_ids_in_use_.+"}, "label", "$1", "__name__", "neo4j_ids_in_use_(.+)")This will provide you with a new variable called “label” which you can use in your legend template, filled in with the second capturing group in the last regex. So, using {{label}} count will print Node count, Relationship count and so on, for each metric. Note that this new field is also visible in the metrics object revealed through the Query Inspector.
This is not vital, of course, but cleaner charts are more readable. For the same reason, another good addition is to display the current value of each metric directly on the label itself, so that users do not need to check the y-axis or hover the mouse at the last point to find out. To achieve this, go to the visualization tab, in the Legend fieldset, and turn on the Current value option.
Derive new dimensions from existing metrics: Speed
PromQL has many useful functions which let you manipulate your metrics for your needs; see the complete list. An extremely useful one is irate, which will calculate the “per-second instant rate of increase of the time series” in a chosen range. So, given our instant vector neo4j_ids_in_use_node, we can get a range vector by simply appending a time range. neo4j_ids_in_use_node[5m]. For example, we will select the values of the counter metric within ranges of five minutes, according to the actual function used on the range vector (the range vector can indeed be rendered only using functions).
We can pass the 5-minute range vector to the instant rate function with: irate(neo4j_ids_in_use_node{job='$job'}[5m]), and get a nice representation of the per-second variation of the total number of nodes in the database, or in other words, the node creation speed.
Now speed calls for Gauge, right? Grafana’s built-in Gauge visualization makes this easy. We will not go into details since the configuration is pretty straightforward. If you care about ingestion speed, one important tip is to choose meaningful thresholds. To do so, test your Neo4j setup in a production-like environment, measure the maximum speed, and set thresholds accordingly. Remember to invert the colors, because red means the database is slow 🙂 Repeat the same steps for relationship and property creation speed, and you’ll have a complete dashboard for your Neo4j racing car!
Bolt queue length using operators
PromQL operators can also be very useful because you can perform arithmetic, comparison, and logical operations with vectors. Take, for example, Neo4j Bolt metrics. neo4j.bolt.messages_received returns the “total number of messages received via Bolt since this instance started”, and neo4j.bolt.messages_started “tracks how many of the received messages have been taken on by a worker thread” (see Neo4j manual). So, every time the database receives a new Bolt message, it can start a thread or pause while waiting for one.
Neo4j provides a specific metric to monitor “the accumulated time messages have spent waiting for a worker thread” (neo4j.bolt.accumulated_queue_time), but it is also interesting to monitor the length of the message queue: how many bolt messages are waiting? Well, you can plot the difference between the two vectors neo4j_bolt_messages_received_total - neo4j_bolt_messages_started_total , and you have the bolt queue length!
In this case, you might even want to render the queue line on top of the counter lines; since they are on different scales, you can use a series override (from the visualization tab). We suggest increasing the line width and assigning the queue series to the right y-axis scale.
Filtering with Grafana variables: $job
Ok, we know how to query Neo4j metrics with PromQL and render them in Grafana. So far, we focused on a single job, but one of Grafana’s big advantages is combining many different jobs in the same dashboard. As an example, we can imagine having three different environments: development, stage, and production, each with Neo4j reporting statistics, each with a prometheus.yml file defining a scrape job. In this case, the time series are completely independent from one job to another, since what happens at a specific point in time in production has nothing to do with what happens at the same moment in development or stage. So we want to switch between environments while keeping the same dashboard.
We have seen that the job name is automatically added as a label to each time series, but we don’t want to duplicate our queries manually. What we need is a variable. In Grafana, go to Dashboard settings, open the Variables tab, and add a new variable. Name the variable (e.g., neo4j_job) and choose a type. Here we have 2 options. We could create a custom variable and hardcode the values of our job names, or better, in this case, we could use one of the query functions that the Prometheus data source plugin for Grafana provides us: label_values(label).
So, select the Query type, select the Prometheus data source and type label_values(job). At the bottom of the variable page, you can see a preview of the values generated. As you can see, there is also the Prometheus job itself, but since we only want Neo4j jobs here, we can add a simple regex like /neo4j/ to filter out the others.
Grafana displays variables as dropdown filters at the top of the page automatically. But to make our new filter work, we need to bind the queries to it using the $ symbol. Our query would look like this: up{job=”$neo4j_job”} and we can use {{job}} in legend templates. This way, we can easily switch between Neo4j environments, and all the panels will re-render with the chosen environment data. Neat!
Since we are on the multi-job topic, remember that Grafana can manage multiple data sources, so you can monitor Prometheus and non-Prometheus metrics in the same dashboard. This is useful when you want to monitor other processes side by side that connect to Neo4j in your platform and already export metrics to other systems.
Filtering with $instance and other tricks for Neo4j cluster dashboards
If Neo4j is running in cluster mode, you will have different machines. In order to collect statistics from all of them, each Neo4j instance must be configured to publish its own Prometheus endpoint, and the scrape job prometheus.yml must be edited with a target list including all the endpoints. For example, our test configuration looks like this:
- job_name: 'neo4j-prometheus'
static_configs:
- targets: ['neo4j-core1:2004', 'neo4j-core2:2005', 'neo4j-core3:2006', 'neo4j-replica1:2007']This will be reflected in the instance label provided by Prometheus. If you do nothing, you will get one series per instance for each metric. This might be exactly what you need. But if you want to filter by instance, simply repeat the steps above and add a label_values(instance) query variable and a suitable regex to populate a neo4j_instance variable.
You can enable Multi-value and Include All options as they are very useful. Just remember that Grafana converts labels to regex-compatible strings in this case, so you need to use the =~ syntax, such as {instance=~"$neo4j_instance"}.
While the default line chart supports multiple time series, this is not true for singlestat or gauge charts. If you want to keep those, either set them to a specific instance or use Grafana’s Repeat feature to multiply them dynamically based on the variable selection. Our initial status singlestat panel is a good example. It doesn’t work with multiple instances, and it is not meaningful with only one, since we want to know the status of the full cluster.
Open the panel editor and go to the third tab called General. Fill in the Repeating section with the newly created instance variable and other settings as you see fit. Now update the query to support multiple instances, like this up{job='$job', instance=~"$neo4j_instance"} , and go back to your dashboard. Now you can change the instance dropdown filter at the top, and when you select all or multiple, your status panel will repeat accordingly.
Neat, but all panels look the same, and there is no mention of the instance being tracked. Well, we cannot use Prometheus labels in the panel title or in the prefix/suffix of the single stat chart, but we can use Grafana variables, so if you add $neo4j_instance as a prefix template in the visualization tab, you’ll get exactly what you needed: each panel would have the tracked instance stamp on it!
Get the leader!
If you have a Neo4j cluster, you can also use the specific causal clustering metrics. Instances in a Neo4j cluster can have three different roles: leader (only one at a specific time), follower (in the core group together with the leader), and read replica. Even though the leader can change during core elections, a common monitoring request is to plot only the current leader.
Our starting point to achieve this is the Boolean metric called neo4j.causal_clustering.core.is_leader. Let’s create a new Grafana query variable with the following details: query: query_result(neo4j_causal_clustering_core_is_leader == 1) regex: /instance="(.+)",/
Query_result is another function offered by the Grafana Prometheus plugin, which can return results of a query. We combine it with a Prometheus comparison operator, which by default filters the time series, and a suitable capturing-group regex. This returns the leader instance and stores it in a variable.
Now, let’s say that we want to display the number of active transactions for the leader. We could use a query like this: neo4j_transaction_committed_total{job="$job", instance="$leader"}, maybe pairing this up with its “opposite” neo4j_transaction_committed_total{job=”$job”, instance!="$leader"} to identify the followers’ and replicas’ transactions.
Custom timer metric as a histogram
In part 1, we took all the effort to publish custom metrics, and we should render them! You’ve got the hang of counters, so let’s focus on timers.
Timers are great because you can measure the time performance of any method of your custom database logic. For example, imagine data being ingested into Neo4j from event processor queues or other sources. In this case, you would enforce your graph model in middleware, where you could, for example, split the original events into nodes and relationships to speed up transactions and limit write locks, based on your data model. With a timer metric, you could monitor how long this method takes to execute, and you could control what level of detail to include, for example, adding some business categories as a Prometheus label to be leveraged in queries. This could be the category of entities, or the category of an event source, for example.
The timer metric returns quantiles. As you can see from the output in the custom metric section of our previous post, a 0.5 quantile with a value of 1.3296 means that for the 20 runs (_count) of the monitored procedure, in 50% of the cases, the method took 1.3296 seconds. This is the median of our timer.
Go on and add my-timer{job=”$job”} as a panel query. Each quantile will show up as a different line. Note that quantile is a new label you can use as you see fit. This visualization lets you monitor how the size of these quantiles varies over time.
Another option is to build a histogram out of this metric. If you wish to do so, go to the visualization tab and set the x-axis mode to histogram instead of time. The result will show the current state of the quantile distribution. On the x-axis, you will have the values – how many seconds did your code take to run – and on the y-axis, you will have the count – how many events happened with that timing.
Putting it all together
Following the examples we covered so far, you can create panels for all the Neo4j metrics you need, including the very important transaction metrics, page cache metrics, and don’t forget checkpoint ones either.
Most of the time, the metric name is enough to render a metric, and you can pick a visualization from the many Grafana offers out of the box. When you need more advanced transformations, PromQL functions and operators can be extremely useful for getting the values you need, and visualization options help you render them in a meaningful, informative way.
Once you put all your panels in place, you get a beautiful Neo4j Dashboard like this one.
We published an example docker-compose project to help anybody test this in no time. Simply clone the repository and type docker-compose up. The script will install in the main repository folder a Neo4j cluster of 3 core members and 1 read replica, Prometheus, and Grafana. Everything is set up to work out of the box, including the provisioned dashboard. A really good starting point for experimenting! For more information, read our Causal Cluster Quickstart post.
Update: The example Neo4j dashboard used in this short series is publicly available on grafana.com dashboard sharing service, so it is even easier to import it into your own running Grafana instance.
Annotations and alerts
Even if we call it that, we don’t think there is a “perfect” Neo4j Dashboard for every use case. Especially with custom logic, you need to tailor it to your needs. In this tutorial, we share how to do that. You also want to pick the internal metrics you care about and avoid showing everything in your dashboard. There are also other techniques to improve monitoring clarity and efficiency.
We will not go into details, but it’s worth noting that the Grafana/Prometheus monitoring solution also offers annotations and alerts.
Grafana lets users add manual annotations to time series charts and includes a tagging system and drop-down filters. For example, after you change your custom database logic and redeploy a new version of your Neo4j plugins, don’t forget to add an annotation with the “deployment” tag and a note about the version or any other useful information. When you check out the dashboard again, it will be easy to spot the moment of change or to know what happened to justify a change in metric trends.
Alerts are extremely useful for 24/7 monitoring and automation. You can set simple or complex rules, using all the features of PromQL, to trigger alerts and send notifications through any medium, including email, Slack, Kafka queues, and more. Both Prometheus and Grafana support alerts.
Conclusion
The Neo4j graph database is often a core component in complex architectures, providing graph features like traversing, path-finding, community detection, recommendations, and many more to other services and applications. For this reason, monitoring its status and performance is even more important. This practice enables effective optimization, supports proactive monitoring, and speeds incident recovery.
In this short series (part 1), we saw how many useful Neo4j internal metrics can be exposed to standard open-source monitoring systems like Prometheus and Grafana, which in some cases are a preferable choice to ad hoc solutions. They not only manage different processes and sources in one place, but also offer powerful query and graphical tools.
When you use general-purpose tools, you need some knowledge to get specific insights from your graph database. So we showed some examples of how to use the PromQL query language and Grafana panels to create beautiful Neo4j Dashboards.
Finally, we took monitoring Neo4j one level up by showing how to publish custom metrics that let you monitor your custom database logic. Here lies the true power of monitoring your graph database: you can place side-by-side internals and business-tailored metrics and release truly useful insights.
Special thanks to all GraphAware experts and to David Allen, Partner Solution Architect at Neo4j, for their valuable feedback on this topic.
This article was originally published in 2019 and reflects the Neo4j, Prometheus, and Grafana versions available at the time. Some configuration steps, metrics, and interface features may have changed in subsequent releases. We retain these examples for historical reference, and they may not reflect current best practices. For up-to-date guidance, please refer to the relevant product documentation.
