You are currently viewing Complete Guide: Installing and Setting up ELK Stack (Elasticsearch, Logstash, Kibana) on a GCP Machine

Complete Guide: Installing and Setting up ELK Stack (Elasticsearch, Logstash, Kibana) on a GCP Machine

To install and set up the ELK Stack (Elasticsearch, Logstash, Kibana) on a GCP machine, you can follow these steps:

  1. Provision a GCP virtual machine: Create a virtual machine instance on Google Cloud Platform. Ensure that the machine meets the minimum requirements for running ELK Stack (sufficient CPU, memory, and disk space).
  2. Connect to the GCP machine: Use SSH to connect to the virtual machine instance. You can use the gcloud command-line tool or connect through the GCP Console.
  3. Install Java: ELK Stack requires Java to run. Install Java Development Kit (JDK) using the appropriate package manager for your Linux distribution. For example, on Ubuntu, you can use the following command:
sudo apt update
sudo apt install default-jdk

Install Elasticsearch:

  • Add the Elasticsearch repository key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add the Elasticsearch repository:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

Update and install Elasticsearch:

sudo apt update
sudo apt install elasticsearch

Start and enable the Elasticsearch service:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Install Logstash:

Add the Logstash repository key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add the Logstash repository:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

Update and install Logstash:

sudo apt update
sudo apt install logstash

Install Kibana:

  • Add the Kibana repository key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add the Kibana repository:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

Update and install Kibana:

sudo apt update
sudo apt install kibana

Configure Kibana to listen on the external network interface by editing the Kibana configuration file:

sudo nano /etc/kibana/kibana.yml
  • Uncomment the line #server.host: "localhost" and change it to server.host: "0.0.0.0".
  • Start and enable the Kibana service:
sudo systemctl start kibana
sudo systemctl enable kibana

Once the installation is complete, you should be able to access Kibana by navigating to http://<your-server-ip>:5601 in a web browser. From there, you can configure and manage your ELK Stack for log management, analysis, and visualization.

Please note that the above steps provide a basic installation of ELK Stack on a GCP machine. For production environments, you may need to consider additional configuration and security measures.

For more detailed information and advanced configurations, refer to the official Elastic documentation for Elasticsearch, Logstash, and Kibana.

Step-by-Step Guide: Provisioning a GCP Virtual Machine for Running ELK Stack with Terraform

  1. Install Terraform: Download and install Terraform on your local machine by following the official installation instructions for your operating system: https://learn.hashicorp.com/tutorials/terraform/install-cli
  2. Create a new directory: Create a new directory on your local machine where you’ll store your Terraform configuration files.
  3. Create a Terraform configuration file: Inside the directory, create a file named main.tf and add the following content:
provider "google" {
  project = "<your-project-id>"
  region  = "us-central1"
}

resource "google_compute_instance" "elk" {
  name         = "elk-instance"
  machine_type = "n1-standard-2"

  boot_disk {
    initialize_params {
      image = "ubuntu-os-cloud/ubuntu-2004-lts"
    }
  }

  network_interface {
    network = "default"
  }

  metadata_startup_script = <<-EOF
    #!/bin/bash
    sudo apt update
    sudo apt install default-jdk -y
    sudo apt install elasticsearch logstash kibana -y
    sudo systemctl enable elasticsearch logstash kibana
    sudo systemctl start elasticsearch logstash kibana
  EOF
}

Make sure to replace <your-project-id> with your actual GCP project ID.

Initialize the Terraform configuration: Open a terminal or command prompt, navigate to the directory where you saved the main.tf file, and run the following command:

terraform init

This command initializes Terraform and downloads the necessary providers.

Preview the changes: Run the following command to preview the changes that Terraform will apply:

terraform plan

Review the output to ensure that the resources to be created match your expectations.

Apply the Terraform configuration: To create the virtual machine and provision the ELK Stack, run the following command:

terraform apply

You’ll be prompted to confirm the creation of resources. Type “yes” and press Enter to proceed.

Terraform will provision the GCP virtual machine, install Java, Elasticsearch, Logstash, and Kibana on the instance, and start the necessary services.

Once the process completes, you can access Kibana by navigating to http://<your-vm-ip>:5601 in a web browser.

Remember to manage your Terraform state files properly and consider additional configuration and security measures for production environments.

For more advanced configurations or customization, you can refer to the Terraform documentation and GCP provider documentation: https://www.terraform.io/docs/ and https://registry.terraform.io/providers/hashicorp/google/latest/docs

Leave a Reply