Blog

Elasticsearch Meets Kubernetes, Part 2 of 2

This post is part 2 of a 2-part series on setting up an Elasticsearch on Kubernetes. In Part 1, I discussed how to create a Kubernetes ready Elasticsearch Docker image and set up the front-end service. In part 2 I will finish the job of building out an Elasticsearch cluster by showing you how to create the data and master nodes, and write scripts to start, test, and stop the cluster.

Elasticsearch Kubernetes Cluster

Recall from Part 1 that the cluster we are building consists of separate client, data, and master nodes which provides better Elasticsearch performance than clusters that combine these functions in each node. There will be 2 client nodes, 4 data nodes, and 3 master nodes.  We have the client node, so let’s move on the data and master nodes

 website-elasticsearch-on-kubernetes-stack_graphic

 

Data Nodes

The Kubernetes service file for Elasticsearch data nodes is es_data.yaml.

1   apiVersion: extensions/v1beta1

2   kind: Deployment

3   metadata:

4  name: es-data

5  namespace: es-cluster

6  labels:

7 component: elasticsearch

8 role: data

9   spec:

10 template:

11 metadata:

12   labels:

13 component: elasticsearch

14 role: data

15 spec:

16   containers:

17   – name: es-data

18 securityContext:

19   privileged: true

20   capabilities:1

21 add:

22   – IPC_LOCK

23 image: gcr.io/my_gke_project/elasticsearch:latest

24 imagePullPolicy: Always

25 env:

26 – name: NAMESPACE

27   valueFrom:

28 fieldRef:

29   fieldPath: metadata.namespace

30 – name: “CLUSTER_NAME”

31   value: “my_es_cluster”

32 – name: NODE_MASTER

33   value: “false”

34 – name: NODE_DATA

35   value: “true”

36 – name: HTTP_ENABLE

37   value: “false”

38 – name: ES_HEAP_SIZE

39   value: “1g”

40 – name: NUMBER_OF_SHARDS

41   value: “4”

42 – name: NUMBER_OF_REPLICAS

43   value: “1”

44 ports:

45 – containerPort: 9300

46   name: transport

47   protocol: TCP

48 resources:

49   limits:

50 memory: 2Gi

51 volumeMounts:

52 – name: storage

53   mountPath: /data

54   volumes:

55   – emptyDir:

56   medium: “”

57 name: “storage”


[Lines 32-35] To create an Elasticsearch data node set the NODE_MASTER to false and the NODE_DATA to true.

[Lines 36-37] HTTP_ENABLE is set to false to allow only TCP network I/O between nodes.

[Lines 38-39] The heap size is set to 1 GB which means the size of the client container should be 2 GB. These values are too small for most clusters, so you will have to increase them as necessary. 

 

[Lines 40-41] The number of primary shards. The rule of thumb is to create 1 primary shard for each data node. You can also set this to a higher number if you anticipate adding data nodes to the cluster for increased capacity. If you do set a higher value, then when you do add nodes, Elasticsearch will rebalance the primary and replica nodes to make sure there is an even distribution of shards across the cluster. 

[Lines 42-43] The number of replica shards. Note this is a replica set, meaning 1 replica shard per primary shard. Usually 1 is a good number, but you can also set this to a higher value if you want to add more data nodes later.

[Lines 45-47] TCP port 9300 is used for inter-node network traffic to handle Elasticsearch internal command communication and is not exposed as a service. The data and master nodes will only use this port and not 9200.

[Line 50] Set the pod memory size to twice the ES_HEAP_SIZE. 

Master Nodes

The Kubernetes service file for Elasticsearch master nodes is es_master.yaml.

 

1   apiVersion: extensions/v1beta1

2   kind: Deployment

3   metadata:

4  name: es-master

5  namespace: es-cluster

6  labels:

7 component: elasticsearch

8 role: master

9   spec:

10 template:

11 metadata:

12   labels:

13 component: elasticsearch

14 role: master

15 spec:

16   containers:

17   – name: es-master

18 securityContext:

19   privileged: true

20   capabilities:

21 add:

22   – IPC_LOCK

23 image: gcr.io/my_gke_project/elasticsearch:latest

24 imagePullPolicy: Always

25 env:

26 – name: NAMESPACE

27   valueFrom:

28 fieldRef:

29   fieldPath: metadata.namespace

30 – name: “CLUSTER_NAME”

31   value: “my_es_cluster”

32 – name: NODE_MASTER

33   value: “true”

34 – name: NODE_DATA

35   value: “false”

36 – name: HTTP_ENABLE

37   value: “false”

38 – name: ES_HEAP_SIZE

39   value: “512m”

40 – name: NUMBER_OF_SHARDS

41   value: “4”

42 – name: NUMBER_OF_REPLICAS

43   value: “1”

44 – name: NUMBER_OF_MASTERS

45   value: “2”

46 ports:

47 – containerPort: 9300

48   name: transport

49   protocol: TCP

50 resources:

51   limits:

52 memory: 1Gi

53 volumeMounts:

54 – name: storage

55   mountPath: /data

56   volumes:

57   – emptyDir:

58   medium: “”

59 name: “storage” 

 

[Lines 32-35] To create an Elasticsearch master node set the NODE_MASTER to true and the NODE_DATA to false.

[Lines 38-39] The heap size is set to 512 MB which means the size of the client container should be 1 GB. These values are too small for most clusters, so you will have to increase them as necessary. 

[Lines 40-41] The number of primary shards. For this setting to take effect it must match the corresponding setting in es-data.yaml.

[Lines 42-43] The number of replica shards. For this setting to take effect it must match the corresponding setting in es-data.yaml.

[Line 44-45] The minimum number of master nodes must be set to 2 for a master node cluster of 3. This setting prevents split brain syndrome when one of the master nodes fails, drops out of the (master node) cluster, then rejoins the cluster.

 

[Lines 47-49] TCP port 9300 is used for inter-node network traffic to handle Elasticsearch internal command communication and is not exposed as a service. The data and master nodes will only use this port and not 9200.

[Line 52] Set the pod memory size to twice the ES_HEAP_SIZE. 

Run the Elasticsearch Cluster

Startup Script

To start the cluster requires  series of kubectl commands which is more easily done in a script. The following script –start-es.sh – starts the deployments, waits for each to become active, scales the cluster to the desired size, and finally waits for a public Elasticsearch service IP. 

1   #!/bin/bash

2

3   echo “Creating Elasticsearch services…”

4   kubectl create -f es-discovery.yaml

5   kubectl create -f es-svc.yaml

6   kubectl create -f es-master.yaml –validate=false

7   kubectl create -f es-client.yaml –validate=false

8   kubectl create -f es-data.yaml –validate=false

9

10  # Check to see if the deployments are running

11  while true; do

12 active=kubectl get deployments --all-namespaces | grep es-master | awk '{print $6}'

13 if [ “$active” == “1” ]; then

14 break

15 fi

16 sleep 2

17  done

18  while true; do

19 active=kubectl get deployments --all-namespaces | grep es-client | awk '{print $6}'

20 if [ “$active” == “1” ]; then

21 break

22 fi

23 sleep 2

24  done

25  while true; do

26 active=kubectl get deployments --all-namespaces | grep es-data | awk '{print $6}'

27 if [ “$active” == “1” ]; then

28 break

29 fi

30 sleep 2

31  done

32   

33  # Scale the cluster to 3 master, 4 data, and 2 client nodes

34  kubectl scale deployment es-master –replicas 3

35  kubectl scale deployment es-client –replicas 2

36  kubectl scale deployment es-data –replicas 4

37   

38  echo “Waiting for Elasticsearch public service IP…”

39  while true; do

40 es_ip=kubectl get svc elasticsearch | grep elasticsearch | awk '{print $3}'

41 if [ “$es_ip” != “<pending>” ]; then

42 break

43 fi

44 sleep 2

45  done   

46  echo “Elasticsearch public IP: “$es_ip”

 

You can start with a larger number of data nodes if you want by changing the number of es-data replicas. If you do this, make sure the NUMBER_OF_SHARDS in the data and master node deployment files are set to that number. 

Run the script, then wait for it to finish. The output should look something like this:

$ ./start-es.sh

Creating Elasticsearch services…

namespace “es-cluster” created

service “elasticsearch-discovery” created

service “elasticsearch” created

deployment “es-master” created

deployment “es-client” created

deployment “es-data” created

Scaling cluster…

deployment “es-master” scaled

deployment “es-client” scaled

deployment “es-data” scaled

Waiting for Elasticsearch public service IP…

Elasticsearch public IP: 104.196.121.33

 

Check to make sure you got all the pods running.

$ kubectl get pods –namespace=es-cluster

NAME READY STATUS RESTARTS   AGE

es-client-1692123692-ff8ms   1/1    Running   0   4m

es-client-1692123692-nfjpl   1/1    Running   0   4m

es-data-2061351218-a14hh 1/1    Running   0   4m

es-data-2061351218-a7jr1 1/1    Running   0   4m

es-data-2061351218-dasnd 1/1    Running   0   4m

es-data-2061351218-p8ws1 1/1    Running   0   4m

es-master-2164439597-4t5z8   1/1    Running   0   4m

es-master-2164439597-je66g   1/1    Running   0   4m

es-master-2164439597-uzf02   1/1    Running   0   4m

 

Test the Cluster

To test out cluster, let’s add some data. This set is taken from the Aggregation Test-Drive chapter in Elasticsearch: The Definitive Guide 2.x:

$ curl ‘http://104.196.121.33:9200/cars/transactions/_bulk‘ -d ‘

{ “index”: {}}

{ “price” : 10000, “color” : “red”, “make” : “honda”, “sold” : “2014-10-28” }

{ “index”: {}}

{ “price” : 20000, “color” : “red”, “make” : “honda”, “sold” : “2014-11-05” }

{ “index”: {}}

{ “price” : 30000, “color” : “green”, “make” : “ford”, “sold” : “2014-05-18” }

{ “index”: {}}

{ “price” : 15000, “color” : “blue”, “make” : “toyota”, “sold” : “2014-07-02” }

{ “index”: {}}

{ “price” : 12000, “color” : “green”, “make” : “toyota”, “sold” : “2014-08-19” }

{ “index”: {}}

{ “price” : 20000, “color” : “red”, “make” : “honda”, “sold” : “2014-11-05” }

{ “index”: {}}

{ “price” : 80000, “color” : “red”, “make” : “bmw”, “sold” : “2014-01-01” }

{ “index”: {}}

{ “price” : 25000, “color” : “blue”, “make” : “ford”, “sold” : “2014-02-12” }’

Then run a query to make sure the data was indexed:

$ curl ‘http://104.196.121.33:9200/cars/transactions‘ -d ‘

{ “query”: { “match”: { “color”: “red” } } }’

You should get 4 hits back.

Stop the Cluster

Normally you would leave Elasticsearch cluster running. However, if you are just doing testing you can tear down the cluster with this script (contained in stop-es.sh):

 

1   #!/bin/bash

2

3   echo “Tearing down cluster services…”

4   kubectl delete svc –namespace=es-cluster elasticsearch

5   kubectl delete svc –namespace=es-cluster elasticsearch-discovery

6   kubectl delete deployment –namespace=es-cluster es-master

7   kubectl delete deployment –namespace=es-cluster es-client

8   kubectl delete deployment –namespace=es-cluster es-data

9   kubectl delete namespace es-cluster

10  sleep 60

11  echo “Done”

 

When you run the script, the output will look like this:

$ ./stop-es.sh

Tearing down cluster services…

service “elasticsearch” deleted

service “elasticsearch-discovery” deleted

deployment “es-master” deleted

deployment “es-client” deleted

deployment “es-data” deleted

namespace “es-cluster” deleted

Done

 

Persistent Storage

If you gave gotten this far, you have a working Elasticsearch cluster running on Kubernetes. So what’s to do now. Primarily you’ll need a persistent form of disk storage to make sure you don’t lose your data between restarts of your cluster. There are several ways to do that, so I did not cover that here. The Kubernetes online documentation have a chapter on Persistent Volumes that can get you going with that. 

 

Author: Vic Hargrave

About Solinea

Solinea services help enterprises build step-by-step modernization plans to evolve from legacy infrastructure and processes to modern cloud and open source infrastructure driven by DevOps and Agile processes.

Better processes and tools equals better customer (and employee) satisfaction, lower IT costs, and easier recruiting, with fewer legacy headaches.

Solinea specializes in 3 areas: 

  • Containers and Microservices –  Now enterprises are looking for ways to drive even more efficiencies, we help organizations with Docker and Kubernetes implementations – containerizing applications and orchestrating the containers in production.
  • DevOps and CI/CD Automation –  Once we build the infrastructure, the challenge is to gain agility from the environment, which is the primary reason people adopt cloud. We work at the process level and tool chain level, meaning that we have engineers that specialize in technologies like Jenkins, Git, Artifactory, Cliqr and we build these toolchains and underlying processes so organizations can build and move apps more effectively to the cloud.
  • Cloud Architecture and Infrastructure –  We are design and implementation experts, working with a variety of open source and proprietary, and have built numerous private, public, and hybrid cloud platforms for globally-recognized enterprises for over three years.