![]() ![]() Burrow gives you visibility into Kafka’s offsets, topics, and consumers.īy consuming the special internal Kafka topic _consumer_offsets, Burrow can act as a centralized service, separate from any single consumer, giving you an objective view of consumers based on both their committed offsets (across topics) and broker state. For that, there is Burrow.īurrow is a specialized monitoring tool developed by LinkedIn specifically for Kafka consumer monitoring. In addition to the key metrics mentioned in Part 1 of this series, you may want more detailed metrics on your consumers. The configuration steps depend greatly on the particular monitoring tools you choose, but JMX is a fast route to viewing Kafka performance metrics using the MBean names mentioned in Part 1 of this series. ![]() Fortunately, many monitoring services and tools can collect JMX metrics from Kafka, whether via JMX plugins via pluggable metrics reporter libraries or via connectors that write JMX metrics out to StatsD, Graphite, or other systems. To answer these kinds of questions, you need a more sophisticated monitoring system. JConsole is a great lightweight tool that can provide metrics snapshots very quickly, but is not so well-suited to the kinds of big-picture questions that arise in a production environment: What are the long-term trends for my metrics? Are there any large-scale patterns I should be aware of? Do changes in performance metrics tend to correlate with actions or events elsewhere in my environment? Collect Kafka performance metrics via JMX ![]() To collect JMX metrics from your consumers and producers, follow the same steps outlined above, replacing port 9999 with the JMX port for your producer or consumer, and the node’s IP address. All the JMX paths for Kafka’s key metrics can be found in Part 1 of this series. Edit Kafka’s startup script- bin/kafka-run-class.sh-to include the value of the JMX port by adding the following parameters to the KAFKA_JMX_OPTS variable:Īs you can see in the screenshot above, Kafka aggregates metrics by source. Because JConsole can be resource-intensive, you should run it on a dedicated host and collect Kafka metrics remotely.įirst, you need to designate a port that JConsole can use to collect JMX metrics from your Kafka host. It provides an interface for exploring the full range of metrics Kafka emits via JMX. JConsole is a simple Java GUI that ships with the JDK. Collect Kafka performance metrics with JConsole For host-level metrics, you should consider installing a monitoring agent. ![]() JConsole and JMX can collect all of the native Kafka performance metrics outlined in Part 1 of this series, while Burrow is a more specialized tool that allows you to monitor the status and offsets of all your consumers. JMX with external graphing and monitoring tools and services.JConsole, a GUI that ships with the Java Development Kit (JDK).In this post, we’ll show you how to use the following tools to collect metrics from Kafka and ZooKeeper: Like Tomcat, Cassandra, and other Java applications, both Kafka and ZooKeeper expose metrics on availability and performance via Java Management Extensions (JMX). This post covers some different options for collecting Kafka and ZooKeeper metrics, depending on your needs. You’ve also seen that no Kafka performance monitoring solution is complete without also monitoring ZooKeeper. If you’ve already read our guide to key Kafka performance metrics, you’ve seen that Kafka provides a vast array of metrics on performance and resource utilization, which are available in a number of different ways. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |