Home / linux / kubenetes / loggging

Logging, Monitoring & Troubleshooting

Understanding Resource Monitoring

kubectl get can be used on any resource and shows generic resource health.
If metrics are collected, use kubectl top pods and kubectl top nodes to get performance-related information about Pods and Nodes.
Consider using advanced tools like Prometheus and Grafana for more details about application usage.

Resources are first created in the Kubernetes etcd database.
Use kubectl describe and kubectl events to see how that has been going.
After adding the resources to the database, the Pod application is started on the nodes where it is scheduled.
Before it can be started, the Pod image needs to be fetched:
- Use sudo crictl images to get a list
Once the application is started, use kubectl logs to read the output of the application.
On running applications, use an interactive shell for further investigation.

The first step is to use kubectl get, which will give a generic overview of Pod states
A Pod can be in any of the following states:
- Pending: the Pod has been created in etcd, but is waiting for an eligible node
- Running: the Pod is in a healthy state
- Succeeded: the Pod has done its work and there is no need to restart it
- Failed: one or more containers in the Pod have ended with an error code and will not be restarted
- Unknown: the state could not be obtained, often related to network issues
- Completed: the Pod has run to completion
- CrashLoopBackOff: one or more containers in the Pod have generated an error, but the scheduler is still trying to run them

If kubectl get indicates that there is an issue, the next step is to use kubectl describe to get more information.
kubectl describe shows API information about the resource and often has good indicators of what is going wrong.
If kubectl describe shows that a Pod has an issue starting its primary container, use kubectl logs to investigate the application logs.
If the Pod is running, but not behaving as expected, open an interactive shell on the Pod for further troubleshooting: kubectl exec -it mypod -- sh

Use kubectl cluster-info for a generic impression of cluster health.
Use kubectl cluster-info dump for (too much) information coming from all the cluster log files.
kubectl get nodes will give a generic overview of node health.
kubectl get pods -n kube-system shows Kubernetes core services running on the control node.
kubectl describe node nodename shows detailed information about nodes, check the "Conditions" section for operational information.
sudo systemctl status kubelet will show current status information about the kubelet.
sudo systemctl restart kubelet allows you to restart it.
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -text allows you to verify kubelet certificates and verify they are still valid.
The kube-proxy Pods are running to ensure connectivity with worker nodes, use kubectl get pods -n kube-system for an overview.

To access applications running in the Pods, Services and Ingress are used.
The Service resource uses a selector label to connect to Pods with a matching label.
The Ingress resource connects to a Service and picks up its selector label to connect to the backend Pods directly.
To troubleshoot application access, check the labels in all of these resources.