Kubernetes Advanced: Operators, Helm, and Service Mesh Guide
December 18, 2024
9 min read
Tushar Agrawal
Master advanced Kubernetes concepts including Helm charts, custom operators, service mesh with Istio, GitOps patterns, and production best practices.
Introduction
Once you've mastered basic Kubernetes deployments, the next level involves Helm for package management, Operators for complex application lifecycle management, and Service Mesh for advanced networking. These tools are essential for running production workloads at scale.
In this guide, we'll cover:
- Helm charts creation and best practices
- Kubernetes Operators with Operator SDK
- Service Mesh with Istio
- GitOps with ArgoCD
- Production patterns and optimization
Helm Charts Deep Dive
Chart Structure
mychart/
├── Chart.yaml # Chart metadata
├── values.yaml # Default configuration
├── values-prod.yaml # Production overrides
├── templates/
│ ├── _helpers.tpl # Template helpers
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── configmap.yaml
│ ├── secrets.yaml
│ ├── hpa.yaml
│ └── NOTES.txt # Post-install notes
├── charts/ # Dependencies
└── .helmignore
Chart.yaml
apiVersion: v2
name: myapp
description: A Helm chart for MyApp
type: application
version: 1.0.0
appVersion: "2.0.0"
dependencies:
- name: postgresql
version: "12.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
- name: redis
version: "17.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: redis.enabled
maintainers:
- name: Tushar Agrawal
email: tushar@example.com
Templates with Best Practices
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "myapp.fullname" . }}
labels:
{{- include "myapp.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "myapp.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
labels:
{{- include "myapp.selectorLabels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "myapp.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.targetPort }}
env:
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
value: {{ $value | quote }}
{{- end }}
envFrom:
- configMapRef:
name: {{ include "myapp.fullname" . }}-config
- secretRef:
name: {{ include "myapp.fullname" . }}-secrets
livenessProbe:
httpGet:
path: {{ .Values.health.liveness.path }}
port: http
initialDelaySeconds: {{ .Values.health.liveness.initialDelaySeconds }}
readinessProbe:
httpGet:
path: {{ .Values.health.readiness.path }}
port: http
initialDelaySeconds: {{ .Values.health.readiness.initialDelaySeconds }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
values.yaml
# Default values for myapp
replicaCount: 2
image:
repository: myregistry.io/myapp
pullPolicy: IfNotPresent
tag: "" # Defaults to Chart.appVersion
service:
type: ClusterIP
port: 80
targetPort: 8080
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: myapp.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: myapp-tls
hosts:
- myapp.example.com
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 128Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
health:
liveness:
path: /health/live
initialDelaySeconds: 10
readiness:
path: /health/ready
initialDelaySeconds: 5
env:
LOG_LEVEL: info
NODE_ENV: production
postgresql:
enabled: true
auth:
database: myapp
username: myapp
redis:
enabled: true
architecture: standalone
Helm Commands
# Create new chart
helm create myapp
# Install chart
helm install myapp ./myapp -n production --create-namespace
# Install with values file
helm install myapp ./myapp -f values-prod.yaml -n production
# Upgrade release
helm upgrade myapp ./myapp -f values-prod.yaml -n production
# Rollback
helm rollback myapp 1 -n production
# Template rendering (debug)
helm template myapp ./myapp -f values-prod.yaml
# Package chart
helm package ./myapp
# Push to OCI registry
helm push myapp-1.0.0.tgz oci://myregistry.io/charts
Kubernetes Operators
Operator Pattern
Operator Architecture
=====================
┌────────────────────────────────────────┐
│ Kubernetes API │
└───────────────────┬────────────────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Watch │ │ Watch │ │ Watch │
│ CRD │ │ Pods │ │ Services │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└──────────────┼──────────────┘
│
▼
┌─────────────────┐
│ Controller │
│ (Reconcile) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Business │
│ Logic │
└─────────────────┘
Custom Resource Definition
# api/v1/database_types.go
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.myapp.io
spec:
group: myapp.io
names:
kind: Database
listKind: DatabaseList
plural: databases
singular: database
shortNames:
- db
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required:
- engine
- version
properties:
engine:
type: string
enum: [postgresql, mysql, mongodb]
version:
type: string
replicas:
type: integer
minimum: 1
maximum: 5
default: 1
storage:
type: string
default: "10Gi"
status:
type: object
properties:
phase:
type: string
readyReplicas:
type: integer
conditions:
type: array
items:
type: object
properties:
type:
type: string
status:
type: string
lastTransitionTime:
type: string
subresources:
status: {}
additionalPrinterColumns:
- name: Engine
type: string
jsonPath: .spec.engine
- name: Version
type: string
jsonPath: .spec.version
- name: Status
type: string
jsonPath: .status.phase
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
Controller (Go)
// controllers/database_controller.go
package controllers
import (
"context"
"fmt"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log"
myappv1 "myapp.io/api/v1"
)
type DatabaseReconciler struct {
client.Client
Scheme *runtime.Scheme
}
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// Fetch the Database instance
var database myappv1.Database
if err := r.Get(ctx, req.NamespacedName, &database); err != nil {
if errors.IsNotFound(err) {
return ctrl.Result{}, nil
}
return ctrl.Result{}, err
}
// Create or update StatefulSet
statefulSet := r.statefulSetForDatabase(&database)
if err := ctrl.SetControllerReference(&database, statefulSet, r.Scheme); err != nil {
return ctrl.Result{}, err
}
found := &appsv1.StatefulSet{}
err := r.Get(ctx, client.ObjectKeyFromObject(statefulSet), found)
if err != nil && errors.IsNotFound(err) {
logger.Info("Creating StatefulSet", "name", statefulSet.Name)
if err := r.Create(ctx, statefulSet); err != nil {
return ctrl.Result{}, err
}
} else if err != nil {
return ctrl.Result{}, err
}
// Update status
database.Status.Phase = "Running"
database.Status.ReadyReplicas = found.Status.ReadyReplicas
if err := r.Status().Update(ctx, &database); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
func (r *DatabaseReconciler) statefulSetForDatabase(db *myappv1.Database) *appsv1.StatefulSet {
replicas := int32(db.Spec.Replicas)
return &appsv1.StatefulSet{
ObjectMeta: metav1.ObjectMeta{
Name: db.Name,
Namespace: db.Namespace,
},
Spec: appsv1.StatefulSetSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{"app": db.Name},
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{"app": db.Name},
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{{
Name: "database",
Image: fmt.Sprintf("%s:%s", db.Spec.Engine, db.Spec.Version),
}},
},
},
},
}
}
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&myappv1.Database{}).
Owns(&appsv1.StatefulSet{}).
Complete(r)
}
Service Mesh with Istio
Architecture
Istio Service Mesh Architecture
===============================
┌─────────────────────────────────────────────────────────────┐
│ Control Plane │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Pilot │ │ Citadel │ │ Galley │ │
│ │ (Traffic) │ │ (Security) │ │ (Config) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Data Plane │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Pod A │ │ Pod B │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ App │ │ ──── │ │ App │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ Envoy │ │ │ │ Envoy │ │ │
│ │ │ (Sidecar)│ │ │ │ (Sidecar)│ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │
│ └──────────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Traffic Management
# VirtualService for traffic routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapp
spec:
hosts:
- myapp.example.com
gateways:
- myapp-gateway
http:
# Canary deployment - 10% to v2
- match:
- headers:
x-canary:
exact: "true"
route:
- destination:
host: myapp
subset: v2
- route:
- destination:
host: myapp
subset: v1
weight: 90
- destination:
host: myapp
subset: v2
weight: 10
retries:
attempts: 3
perTryTimeout: 2s
timeout: 10s
---
# DestinationRule for subsets and load balancing
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: myapp
spec:
host: myapp
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
h2UpgradePolicy: UPGRADE
http1MaxPendingRequests: 100
http2MaxRequests: 1000
loadBalancer:
simple: LEAST_CONN
outlierDetection:
consecutive5xxErrors: 5
interval: 30s
baseEjectionTime: 30s
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
mTLS Configuration
# PeerAuthentication for mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT
---
# AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: myapp-authz
namespace: production
spec:
selector:
matchLabels:
app: myapp
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/production/sa/api-gateway"]
to:
- operation:
methods: ["GET", "POST"]
paths: ["/api/*"]
GitOps with ArgoCD
Application Definition
# ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: myapp-production
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/myapp-config
targetRevision: main
path: environments/production
helm:
valueFiles:
- values.yaml
- values-production.yaml
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ApplicationSet for Multi-Environment
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: myapp
namespace: argocd
spec:
generators:
- list:
elements:
- env: development
cluster: dev-cluster
- env: staging
cluster: staging-cluster
- env: production
cluster: prod-cluster
template:
metadata:
name: 'myapp-{{env}}'
spec:
project: default
source:
repoURL: https://github.com/myorg/myapp-config
targetRevision: main
path: 'environments/{{env}}'
destination:
server: '{{cluster}}'
namespace: '{{env}}'
syncPolicy:
automated:
prune: true
selfHeal: true
Production Best Practices
Resource Management
# LimitRange for namespace defaults
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
spec:
limits:
- default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 128Mi
type: Container
---
# ResourceQuota for namespace limits
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
pods: "50"
Pod Disruption Budget
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: myapp-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: myapp
Conclusion
Advanced Kubernetes features enable:
- Helm: Standardized, versioned deployments
- Operators: Automated application management
- Service Mesh: Secure, observable microservices
- GitOps: Declarative, auditable infrastructure
Related Articles
- Docker & Kubernetes Deployment Guide - K8s fundamentals
- GitHub Actions CI/CD Complete Guide - CI/CD pipelines
- Apache Kafka Deep Dive - Event streaming
- Microservices with Go and FastAPI - Build microservices