Elastic Cloud Monitoring and Alerting

In this blog post, I will show you how to:

Enable logging and monitoring that will let us monitor our deployments in Kibana.
Integrate Elastic Alert with PagerDuty.

Enabling Logging and Monitoring

In Production, the best practice is to send our deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into Elasticsearch and these indexes consume storage, memory, and CPU cycles like any other index. We can avoid affecting other production deployments and view the logs and metrics, even when production deployment is unavailable, by using a separate monitoring deployment.We need a minimum of three monitoring nodes to make monitoring highly available. Read More

Steps:

The Monitoring deployment must have the same major version and must be in the same region as your production deployments. Once the monitoring deployment has been setup, below are the steps to enable monitoring and logging.

Go to one of your deployments then go to Logs and Metrics.

2. Choose your Monitoring deployment where you want to ship data to. Choose Logs and Metrics. Click Save.

3. Repeat these steps for all your deployments.

Viewing Cluster Listing

Let’s view our clusters.

Steps:

1. Go to your Monitoring deployment.

2. Open Kibana.

3. Go to Stack Monitoring, which is under the Management section. When you open Stack Monitoring for the first time, you will be asked to acknowledge the creation of these default rules. They are initially configured to detect and notify on various conditions across your monitored clusters. You can view notifications for: Cluster health, Resource utilization, and Errors and exceptions for Elasticsearch in real time.

4. Click on one cluster to see its overview.

Review and Modify Existing Stack Monitoring Rules

Elastic Stack monitoring feature provides Kibana alerting rules, which is an out-of-the-box monitoring feature. These rules are preconfigured based on the best practices recommended by Elastic, but we can modify them to meet our requirements

Steps:

Go to Alerts and rules > Manage rules.

2. You may choose to edit or retain their default values.

Setting up Alerts Using PagerDuty Connector and Action

The PagerDuty connector uses the v2 Events API to trigger, acknowledge, and resolve PagerDuty alerts.

Creating PagerDuty Service and Intergration

Create a service and add integrations to begin receiving incident notifications.

Steps:

In Pager Duty, go to Services -> Service Directory and click New Service. On the next screen you will be guided through several steps.

2. Name: Enter a Name and Description based on the function that the service provides and click Next to continue.

3. Assign: Select Generate a new Escalation Policy or Select an existing Escalation Policy. Click Next to continue.

4. Integrations: Select the integration(s) you use to send alerts to this service from the search bar, dropdown, or from the list of our most popular integrations. In this case, we will select Elastic Alerts.