Using Cribl Search to Monitor Instances in Google Cloud Platform (GCP)

One recurring challenge in managing cloud environments is the tendency for lab and development instances to remain active long after they’re needed. While it might seem like a small oversight, the impact can be significant. These idle instances rack up unnecessary costs, drain valuable resources, and open the door to security vulnerabilities. Configuring effective monitoring to notify about the running instances is a good way to address this problem.

In this blog post, we’ll explore how to leverage Cribl Search to efficiently monitor running instances in a Google Cloud Platform (GCP) environment and configure alerts based on the collected data.

GCP Configuration

As a first step, we need to create a service account in GCP that will be used with the dataset provider configuration in Cribl Search. 

• Create a service account either using the GCP console or gcloud commandline. Detailed steps are available in the Google Cloud Documentation. 
• The service account needs to be created within the GCP project that has the compute instances of interest. 
• Go to IAM Admin and grant access for the service account to list the compute instances in the GCP project. For our use case,  we have custom role with the following permissions attached to the service account
  • compute.instances.get
  • compute.instances.list 
• Create a key for the service account created. Steps for creating the key are available here. The JSON content of the key is needed configuring the dataset provider in Cribl

Cribl Configuration

The next step is to configure Dataset Provider and Dataset in Cribl Search.

• Select ‘Data’ from the left pane of Cribl Search UI. 
• Select ‘Dataset Providers’ and add the provider. Select ‘Google Cloud Platform API’ as the dataset provider type. Use the JSON content of the GCP service account key in the account configuration. Detailed instructions for adding the dataset provider is available here
• Create a new ‘Dataset’ using the dataset provider created in the earlier steps. For the endpoint, select ‘v1.instances’ and select a region where the virtual machines are located. Add more endpoints if virtual machines are created in additional regions.
• Use the search option in the dataset to execute search. Adjust search time range to get expected results.

In our use case, Lab/Dev instances are typically not used outside of business hours. However, there is a possibility they may unintentionally remain in a running state. We use the following search to identify compute instances that are still in running state.

dataset="di_gcp_compute_vms" status="RUNNING"
| project name, status, lastStartTimestamp,labels.owner,labels.env

The search then is scheduled to run on a regular basis. The scheduled search has an alert action configured to notify the team about running instances.

Cribl Search offers several notification targets that can be used with a scheduled search to suit your needs and available resources. In our case, we use Slack for notifications. However, you can also configure alerts to be sent via email, as an SNS notification, or to Splunk HEC through a webhook connection. More details about Cribl Search notification options are available here.