Logging in Kubernetes with Elasticsearch Kibana and Fluentd

Logging in Kubernetes with Elasticsearch, Kibana, and Fluentd

Minikube

By default, the Minikube VM is configured to use 1GB of memory and 2 CPU cores. This is not sufficient for Elasticsearch, so be sure to increase the memory in your Docker client (for HyperKit) or directly in VirtualBox. Then, when you start Minikube, pass the memory and CPU options to it:

$ minikube start --vm-driver=hyperkit --memory 8192 --cpus 4

or

$ minikube start --memory 8192 --cpus 4

Once up, make sure you can view the dashboard:

$ minikube dashboard

Then, create a new namespace:

$ kubectl create namespace logging

namespace/logging created

If you run into problems with Minikube, it’s often best to remove it completely and start over.

For example:

$ minikube stop; minikube delete
$ rm /usr/local/bin/minikube
$ rm -rf ~/.minikube
# re-download minikube
$ minikube start

Elastic

We’ll start with Elasticsearch.

kubernetes/elastic.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
component: elasticsearch
template:
metadata:
labels:
component: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.4
env:
- name: discovery.type
value: single-node
ports:
- containerPort: 9200
name: http
protocol: TCP
resources:
limits:
cpu: 500m
memory: 4Gi
requests:
cpu: 500m
memory: 4Gi

---

apiVersion: v1
kind: Service
metadata:
name: elasticsearch
labels:
service: elasticsearch
spec:
type: NodePort
selector:
component: elasticsearch
ports:
- port: 9200
targetPort: 9200

So, this will spin up a single node Elasticsearch pod in the cluster along with a NodePort service to expose the pod to the outside world.

Create:

$ kubectl create -f kubernetes/elastic.yaml -n logging

deployment.extensions/elasticsearch created
service/elasticsearch created

Verify that both the pod and service were created:

$ kubectl get pods -n logging

NAME                          READY   STATUS    RESTARTS   AGE
elasticsearch-bb9f879-d9kmg   1/1     Running   0          75s


$ kubectl get service -n logging

NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
elasticsearch   NodePort   10.102.149.212   <none>        9200:30531/TCP   86s

Take note of the exposed port–e.g, 30531. Then, grab the Minikube IP and make sure you can cURL that pod:

$ curl $(minikube ip):30531

You should see something similar to:

{
"name" : "9b5Whvv",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "-qMwo61ATv2KmbZsf2_Tpw",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

Kibana

Next, let’s get Kibana up and running.

kubernetes/kibana.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kibana
spec:
selector:
matchLabels:
run: kibana
template:
metadata:
labels:
run: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:6.5.4
env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch:9200
- name: XPACK_SECURITY_ENABLED
value: "true"
ports:
- containerPort: 5601
name: http
protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
name: kibana
labels:
service: kibana
spec:
type: NodePort
selector:
run: kibana
ports:
- port: 5601
targetPort: 5601

Like before, this deployment will spin up a single Kibana pod that gets exposed via a NodePort service. Take note of the two environment variables:

  1. ELASTICSEARCH_URL – URL of the Elasticsearch instance
  2. XPACK_SECURITY_ENABLED – enables X-Pack security
  • ELASTICSEARCH_URL – URL of the Elasticsearch instance
  • XPACK_SECURITY_ENABLED – enables X-Pack security
  • Refer to the Running Kibana on Docker guide for more info on these variables.

    Create:

    $ kubectl create -f kubernetes/kibana.yaml -n logging
    

    Verify:

    $ kubectl get pods -n logging
    
    NAME                          READY   STATUS    RESTARTS   AGE
    elasticsearch-bb9f879-d9kmg   1/1     Running   0          17m
    kibana-7f6686674c-mjlb2       1/1     Running   0          60s
    
    
    $ kubectl get service -n logging
    
    NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
    elasticsearch   NodePort   10.102.149.212   <none>        9200:30531/TCP   17m
    kibana          NodePort   10.106.226.34    <none>        5601:32683/TCP   74s
    

    Test this in your browser at http://MINIKUBE_IP:KIBANA_EXPOSED_PORT.

    First, we need to configure RBAC (role-based access control) permissions so that Fluentd can access the appropriate components.

    kubernetes/fluentd-rbac.yaml:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: fluentd
    namespace: kube-system
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: fluentd
    namespace: kube-system
    rules:
    - apiGroups:
    - ""
    resources:
    - pods
    - namespaces
    verbs:
    - get
    - list
    - watch
    
    ---
    
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
    name: fluentd
    roleRef:
    kind: ClusterRole
    name: fluentd
    apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
    name: fluentd
    namespace: kube-system
    

    In short, this will create a ClusterRole which grants get, list, and watch permissions on pods and namespace objects. The ClusterRoleBinding then binds the ClusterRole to the ServiceAccount within the kube-system namespace. Refer to the Using RBAC Authorization guide to learn more about RBAC and ClusterRoles.

    Create:

    $ kubectl create -f kubernetes/fluentd-rbac.yaml
    

    Now, we can create the DaemonSet.

    kubernetes/fluentd-daemonset.yaml:

    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
    name: fluentd
    namespace: kube-system
    labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    spec:
    template:
    metadata:
    labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    spec:
    serviceAccount: fluentd
    serviceAccountName: fluentd
    tolerations:
    - key: node-role.kubernetes.io/master
    effect: NoSchedule
    containers:
    - name: fluentd
    image: fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch
    env:
    - name:  FLUENT_ELASTICSEARCH_HOST
    value: "elasticsearch.logging"
    - name:  FLUENT_ELASTICSEARCH_PORT
    value: "9200"
    - name: FLUENT_ELASTICSEARCH_SCHEME
    value: "http"
    - name: FLUENT_UID
    value: "0"
    resources:
    limits:
    memory: 200Mi
    requests:
    cpu: 100m
    memory: 200Mi
    volumeMounts:
    - name: varlog
    mountPath: /var/log
    - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
    terminationGracePeriodSeconds: 30
    volumes:
    - name: varlog
    hostPath:
    path: /var/log
    - name: varlibdockercontainers
    hostPath:
    path: /var/lib/docker/containers
    

    Be sure to review Kubernetes Logging with Fluentd along with the sample Daemonset. Make sure FLUENT_ELASTICSEARCH_HOST aligns with the SERVICE_NAME.NAMESPACE of Elasticsearch within your cluster.

    Deploy:

    $ kubectl create -f kubernetes/fluentd-daemonset.yaml
    

    If you’re running Kubernetes as a single node with Minikube, this will create a single Fluentd pod in the kube-system namespace.

    $ kubectl get pods -n kube-system
    
    coredns-576cbf47c7-mhxbp                1/1     Running   0          120m
    coredns-576cbf47c7-vx7m7                1/1     Running   0          120m
    etcd-minikube                           1/1     Running   0          119m
    fluentd-kxc46                           1/1     Running   0          89s
    kube-addon-manager-minikube             1/1     Running   0          119m
    kube-apiserver-minikube                 1/1     Running   0          119m
    kube-controller-manager-minikube        1/1     Running   0          119m
    kube-proxy-m4vzt                        1/1     Running   0          120m
    kube-scheduler-minikube                 1/1     Running   0          119m
    kubernetes-dashboard-5bff5f8fb8-d64qs   1/1     Running   0          120m
    storage-provisioner                     1/1     Running   0          120m
    

    Take note of the logs:

    $ kubectl logs fluentd-kxc46 -n kube-system
    

    You should see that Fluentd connect to Elasticsearch within the logs:

    Connection opened to Elasticsearch cluster =>
    {:host=>"elasticsearch.logging", :port=>9200, :scheme=>"http"}
    

    To see the logs collected by Fluentd in Kibana, click “Management” and then select “Index Patterns” under “Kibana”.

    Click the “Create index pattern” button. Select the new Logstash index that is generated by the Fluentd DaemonSet. Click “Next step”.

    Set the “Time Filter field name” to “@timestamp”. Then, click “Create index pattern”.

    Click “Discover” to view your application logs.

    Sanity Check

    Let’s spin up a quick Node.js app to test.

    Point your local Docker client at the Minikube Docker daemon, and then build the image:

    $ eval $(minikube docker-env)
    $ docker build -t fluentd-node-sample:latest -f sample-app/Dockerfile sample-app
    

    Create the deployment:

    $ kubectl create -f kubernetes/node-deployment.yaml
    

    Take a quick look at the app in sample-app/index.js:

    const SimpleNodeLogger = require('simple-node-logger');
    const opts = {
    timestampFormat:'YYYY-MM-DD HH:mm:ss.SSS'
    };
    const log = SimpleNodeLogger.createSimpleLogger(opts);
    
    (function repeatMe() {
    setTimeout(() => {
    log.info('it works');
    repeatMe();
    }, 1000)
    })();
    

    So, it just logs 'it works' to stdout every second.

    Back in “Discover” on the Kibana dashboard, add the following filter:

    1. Field: kubernetes.pod_name
    2. Operator: is
    3. Value: node
  • Field: kubernetes.pod_name
  • Operator: is
  • Value: node
  • Now, you should be able to see the it works log in the stream.

    Leave a Reply

    Your email address will not be published. Required fields are marked *