Application Monitoring Guide

https://prometheus.io/

Prometheus fundamentally stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Besides stored time series, Prometheus may generate temporary derived time series as the result of queries. Every time series is uniquely identified by its metric name and a set of key-value pairs, also known as labels.

The metric name specifies the general feature of a system that is measured (e.g. http_requests_total - the total number of HTTP requests received). It may contain ASCII letters and digits, as well as underscores and colons. It must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*. Labels enable Prometheus’s dimensional data model: any given combination of labels for the same metric name identifies a particular dimensional instantiation of that metric (for example: all HTTP requests that used the method POST to the /api/tracks handler). The query language allows filtering and aggregation based on these dimensions. Changing any label value, including adding or removing a label, will create a new time series.

https://grafana.com/grafana

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB. It is used to graph the prometheus metrics.

https://github.com/prometheus/node_exporter

On each hardware node which runs a LogIsland related service (Zookeeper, Kafka, HBase, Yarn) we want to grab system metrics. Prometheus was developed for the purpose of monitoring web services. In order to monitor the metrics of your linux server, you should install a tool called Node Exporter. Node Exporter, as its name suggests, exports lots of metrics (such as disk I/O statistics, CPU load, memory usage, network statistics, and more) in a format Prometheus understands.

Node exporter can be either installed manually or launched as a Docker container :

We will use the jmx_prometheus_javaagent tool to publish zookeeper metrics on a given port ($ZK_JMX_PORT here). Prometheus will then scrap the metrics here.

We will use the jmx_prometheus_javaagent tool to publish kafka metrics on a given port ($KAFKA_JMX_PORT here). Prometheus will then scrap the metrics here.

Spark has a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics system is configured via a configuration file that Spark expects to be present at $SPARK_HOME/conf/metrics.properties. A custom file location can be specified via the spark.metrics.conf configuration property. By default, the root namespace used for driver or executor metrics is the value of spark.app.id. However, often times, users want to be able to track the metrics across apps for driver and executors, which is hard to do with application ID (i.e. spark.app.id) since it changes with every invocation of the app. For such use cases, a custom namespace can be specified for metrics reporting using spark.metrics.namespace configuration property. If, say, users wanted to set the metrics namespace to the name of the application, they can set the spark.metrics.namespace property to a value like ${spark.app.name}. This value is then expanded appropriately by Spark and is used as the root namespace of the metrics system. Non driver and executor metrics are never prefixed with spark.app.id, nor does the spark.metrics.namespace property have any such affect on such metrics.

Spark’s metrics are decoupled into different instances corresponding to Spark components. Within each instance, you can configure a set of sinks to which metrics are reported. The following instances are currently supported:

Here is a list of arbitrary ports for prometheus data scrapping. there are many web services by host so that can a good idea to carefully note every port number for each of them and to keep the same ports on each host.

https://github.com/justwatchcom/elasticsearch_exporter

this tool is used to get metrics from elasticsearch nodes through the REST api and to serve them in the prometheus format

make sure to edit the $LOGISLAND_HOME/monitoring/.env file with the correct ES_HOST and ES_PORT values.

Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service.

https://github.com/linkedin/Burrow

additionnal configuration can be set in $LOGISLAND_HOME/monitoring/burrow/conf/burrow.cfg but you can leave the default

edit $LOGISLAND_HOME/monitoring/prometheus/conf/prometheus.yml with the following (according to the previous port number list)