Back to Blog
Kubernetes Advanced: Operators, Helm, and Service Mesh Guide

Kubernetes Advanced: Operators, Helm, and Service Mesh Guide

December 18, 2024
9 min read
Tushar Agrawal
KubernetesHelmIstioService MeshOperatorsDevOpsCloud NativeGitOps

Master advanced Kubernetes concepts including Helm charts, custom operators, service mesh with Istio, GitOps patterns, and production best practices.

Introduction

Once you've mastered basic Kubernetes deployments, the next level involves Helm for package management, Operators for complex application lifecycle management, and Service Mesh for advanced networking. These tools are essential for running production workloads at scale.

In this guide, we'll cover:

  • Helm charts creation and best practices
  • Kubernetes Operators with Operator SDK
  • Service Mesh with Istio
  • GitOps with ArgoCD
  • Production patterns and optimization

Helm Charts Deep Dive

Chart Structure

mychart/
├── Chart.yaml          # Chart metadata
├── values.yaml         # Default configuration
├── values-prod.yaml    # Production overrides
├── templates/
│   ├── _helpers.tpl    # Template helpers
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   ├── secrets.yaml
│   ├── hpa.yaml
│   └── NOTES.txt       # Post-install notes
├── charts/             # Dependencies
└── .helmignore

Chart.yaml

apiVersion: v2
name: myapp
description: A Helm chart for MyApp
type: application
version: 1.0.0
appVersion: "2.0.0"
dependencies:
  - name: postgresql
    version: "12.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: postgresql.enabled
  - name: redis
    version: "17.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: redis.enabled
maintainers:
  - name: Tushar Agrawal
    email: tushar@example.com

Templates with Best Practices

templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "myapp.fullname" . }}
  labels:
    {{- include "myapp.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      labels:
        {{- include "myapp.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ include "myapp.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.targetPort }}
          env:
            {{- range $key, $value := .Values.env }}
            - name: {{ $key }}
              value: {{ $value | quote }}
            {{- end }}
          envFrom:
            - configMapRef:
                name: {{ include "myapp.fullname" . }}-config
            - secretRef:
                name: {{ include "myapp.fullname" . }}-secrets
          livenessProbe:
            httpGet:
              path: {{ .Values.health.liveness.path }}
              port: http
            initialDelaySeconds: {{ .Values.health.liveness.initialDelaySeconds }}
          readinessProbe:
            httpGet:
              path: {{ .Values.health.readiness.path }}
              port: http
            initialDelaySeconds: {{ .Values.health.readiness.initialDelaySeconds }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

values.yaml

Default values for myapp

replicaCount: 2
image:
  repository: myregistry.io/myapp
  pullPolicy: IfNotPresent
  tag: ""  # Defaults to Chart.appVersion
service:
  type: ClusterIP
  port: 80
  targetPort: 8080
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: myapp.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: myapp-tls
      hosts:
        - myapp.example.com
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80
health:
  liveness:
    path: /health/live
    initialDelaySeconds: 10
  readiness:
    path: /health/ready
    initialDelaySeconds: 5
env:
  LOG_LEVEL: info
  NODE_ENV: production
postgresql:
  enabled: true
  auth:
    database: myapp
    username: myapp
redis:
  enabled: true
  architecture: standalone

Helm Commands

Create new chart

helm create myapp
Install chart

helm install myapp ./myapp -n production --create-namespace
Install with values file

helm install myapp ./myapp -f values-prod.yaml -n production
Upgrade release

helm upgrade myapp ./myapp -f values-prod.yaml -n production
Rollback

helm rollback myapp 1 -n production
Template rendering (debug)

helm template myapp ./myapp -f values-prod.yaml
Package chart

helm package ./myapp
Push to OCI registry

helm push myapp-1.0.0.tgz oci://myregistry.io/charts

Kubernetes Operators

Operator Pattern

Operator Architecture
=====================
     ┌────────────────────────────────────────┐
     │           Kubernetes API               │
     └───────────────────┬────────────────────┘
                         │
          ┌──────────────┼──────────────┐
          │              │              │
          ▼              ▼              ▼
    ┌──────────┐   ┌──────────┐   ┌──────────┐
    │  Watch   │   │  Watch   │   │  Watch   │
    │   CRD    │   │  Pods    │   │ Services │
    └────┬─────┘   └────┬─────┘   └────┬─────┘
         │              │              │
         └──────────────┼──────────────┘
                        │
                        ▼
               ┌─────────────────┐
               │   Controller    │
               │   (Reconcile)   │
               └────────┬────────┘
                        │
                        ▼
               ┌─────────────────┐
               │    Business     │
               │     Logic       │
               └─────────────────┘

Custom Resource Definition

api/v1/database_types.go

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.myapp.io
spec:
  group: myapp.io
  names:
    kind: Database
    listKind: DatabaseList
    plural: databases
    singular: database
    shortNames:
      - db
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required:
                - engine
                - version
              properties:
                engine:
                  type: string
                  enum: [postgresql, mysql, mongodb]
                version:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 5
                  default: 1
                storage:
                  type: string
                  default: "10Gi"
            status:
              type: object
              properties:
                phase:
                  type: string
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                      lastTransitionTime:
                        type: string
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Version
          type: string
          jsonPath: .spec.version
        - name: Status
          type: string
          jsonPath: .status.phase
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

Controller (Go)

// controllers/database_controller.go
package controllers
import (
    "context"
    "fmt"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"
    myappv1 "myapp.io/api/v1"
)
type DatabaseReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)
    // Fetch the Database instance
    var database myappv1.Database
    if err := r.Get(ctx, req.NamespacedName, &database); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }
    // Create or update StatefulSet
    statefulSet := r.statefulSetForDatabase(&database)
    if err := ctrl.SetControllerReference(&database, statefulSet, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }
    found := &appsv1.StatefulSet{}
    err := r.Get(ctx, client.ObjectKeyFromObject(statefulSet), found)
    if err != nil && errors.IsNotFound(err) {
        logger.Info("Creating StatefulSet", "name", statefulSet.Name)
        if err := r.Create(ctx, statefulSet); err != nil {
            return ctrl.Result{}, err
        }
    } else if err != nil {
        return ctrl.Result{}, err
    }
    // Update status
    database.Status.Phase = "Running"
    database.Status.ReadyReplicas = found.Status.ReadyReplicas
    if err := r.Status().Update(ctx, &database); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}
func (r DatabaseReconciler) statefulSetForDatabase(db myappv1.Database) *appsv1.StatefulSet {
    replicas := int32(db.Spec.Replicas)
    return &appsv1.StatefulSet{
        ObjectMeta: metav1.ObjectMeta{
            Name:      db.Name,
            Namespace: db.Namespace,
        },
        Spec: appsv1.StatefulSetSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": db.Name},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": db.Name},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "database",
                        Image: fmt.Sprintf("%s:%s", db.Spec.Engine, db.Spec.Version),
                    }},
                },
            },
        },
    }
}
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&myappv1.Database{}).
        Owns(&appsv1.StatefulSet{}).
        Complete(r)
}

Service Mesh with Istio

Architecture

Istio Service Mesh Architecture
===============================
┌─────────────────────────────────────────────────────────────┐
│                      Control Plane                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │    Pilot    │  │   Citadel   │  │   Galley    │         │
│  │  (Traffic)  │  │  (Security) │  │  (Config)   │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                       Data Plane                            │
│                                                             │
│  ┌──────────────────┐      ┌──────────────────┐           │
│  │     Pod A        │      │     Pod B        │           │
│  │  ┌──────────┐   │      │  ┌──────────┐   │           │
│  │  │   App    │   │ ──── │  │   App    │   │           │
│  │  └──────────┘   │      │  └──────────┘   │           │
│  │  ┌──────────┐   │      │  ┌──────────┐   │           │
│  │  │  Envoy   │   │      │  │  Envoy   │   │           │
│  │  │ (Sidecar)│   │      │  │ (Sidecar)│   │           │
│  │  └──────────┘   │      │  └──────────┘   │           │
│  └──────────────────┘      └──────────────────┘           │
└─────────────────────────────────────────────────────────────┘

Traffic Management

VirtualService for traffic routing

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
    - myapp.example.com
  gateways:
    - myapp-gateway
  http:
    # Canary deployment - 10% to v2
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: myapp
            subset: v2
    - route:
        - destination:
            host: myapp
            subset: v1
          weight: 90
        - destination:
            host: myapp
            subset: v2
          weight: 10
      retries:
        attempts: 3
        perTryTimeout: 2s
      timeout: 10s
---

DestinationRule for subsets and load balancing

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    loadBalancer:
      simple: LEAST_CONN
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

mTLS Configuration

PeerAuthentication for mTLS

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
---

AuthorizationPolicy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: myapp-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: myapp
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/api-gateway"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]

GitOps with ArgoCD

Application Definition

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/myapp-config
    targetRevision: main
    path: environments/production
    helm:
      valueFiles:
        - values.yaml
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

ApplicationSet for Multi-Environment

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: development
            cluster: dev-cluster
          - env: staging
            cluster: staging-cluster
          - env: production
            cluster: prod-cluster
  template:
    metadata:
      name: 'myapp-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/myapp-config
        targetRevision: main
        path: 'environments/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: '{{env}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Production Best Practices

Resource Management

LimitRange for namespace defaults

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      type: Container
---

ResourceQuota for namespace limits

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"

Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp

Conclusion

Advanced Kubernetes features enable:

  • Helm: Standardized, versioned deployments
  • Operators: Automated application management
  • Service Mesh: Secure, observable microservices
  • GitOps: Declarative, auditable infrastructure
Mastering these tools is essential for production Kubernetes at scale.

Related Articles

Share this article

Related Articles

Microservices Security: Zero Trust Architecture Implementation Guide

Complete guide to implementing Zero Trust security in microservices. mTLS, service mesh security, API authentication, secrets management, and OWASP best practices for production systems.

Docker and Kubernetes: Complete Deployment Guide for Production

Master containerization with Docker and orchestration with Kubernetes. Learn to build, deploy, and scale applications with practical examples, best practices, and production-ready configurations.

Observability Stack: Prometheus, Grafana & Jaeger Complete Guide

Build production-grade observability with Prometheus metrics, Grafana dashboards, and Jaeger distributed tracing. Complete setup guide with alerting, custom metrics, and troubleshooting patterns.