PM2 & Linux Services: Production Node.js Deployment Guide

Introduction

Running Node.js applications in production requires robust process management to handle crashes, enable zero-downtime deployments, and utilize multiple CPU cores effectively. Two primary approaches dominate the Linux ecosystem: PM2 (a Node.js-specific process manager) and systemd (the Linux system and service manager).

In this guide, we'll explore:

PM2 fundamentals and advanced features
Linux systemd service creation and management
When to use each approach
Production-ready configurations
Monitoring and logging strategies

PM2 Fundamentals

Installation and Basic Usage

# Install PM2 globally
npm install -g pm2

# Or with yarn
yarn global add pm2

# Verify installation
pm2 --version

Starting Applications

# Basic start
pm2 start app.js

# With a custom name
pm2 start app.js --name "my-api"

# Start with arguments
pm2 start app.js -- --port 3000

# Start a script from package.json
pm2 start npm --name "my-app" -- start

# Start with environment variables
pm2 start app.js --env production

Process Management Commands

# List all processes
pm2 list
pm2 ls
pm2 status

# Detailed process info
pm2 show <app-name>
pm2 describe <app-name>

# Restart processes
pm2 restart <app-name>
pm2 restart all

# Stop processes
pm2 stop <app-name>
pm2 stop all

# Delete processes from PM2
pm2 delete <app-name>
pm2 delete all

# Reload with zero downtime (graceful reload)
pm2 reload <app-name>
pm2 reload all

PM2 Cluster Mode

PM2's cluster mode allows you to run multiple instances of your Node.js application, utilizing all available CPU cores:

PM2 Cluster Mode Architecture
=============================

         ┌─────────────────────────────────────┐
         │            PM2 Master               │
         │     (Load Balancer + Monitor)       │
         └──────────────┬──────────────────────┘
                        │
        ┌───────────────┼───────────────┐
        │               │               │
        ▼               ▼               ▼
   ┌─────────┐    ┌─────────┐    ┌─────────┐
   │ Worker  │    │ Worker  │    │ Worker  │
   │   0     │    │   1     │    │   2     │
   │ (CPU 0) │    │ (CPU 1) │    │ (CPU 2) │
   └─────────┘    └─────────┘    └─────────┘
        │               │               │
        └───────────────┼───────────────┘
                        │
                        ▼
              Incoming Requests
              (Round-robin LB)

Starting in Cluster Mode

# Start with all available CPUs
pm2 start app.js -i max

# Start with specific number of instances
pm2 start app.js -i 4

# Start with CPUs minus 1 (leave one for OS)
pm2 start app.js -i -1

Cluster Mode Application Example

// app.js - Express application optimized for cluster mode
const express = require('express');
const process = require('process');

const app = express();
const PORT = process.env.PORT || 3000;

// Health check endpoint
app.get('/health', (req, res) => {
  res.json({
    status: 'healthy',
    pid: process.pid,
    uptime: process.uptime(),
    memory: process.memoryUsage(),
  });
});

// Your application routes
app.get('/api/data', (req, res) => {
  res.json({
    message: 'Hello from worker ' + process.pid,
    timestamp: new Date().toISOString(),
  });
});

// Graceful shutdown handling
process.on('SIGINT', gracefulShutdown);
process.on('SIGTERM', gracefulShutdown);

function gracefulShutdown() {
  console.log(`Worker ${process.pid} shutting down gracefully...`);

  // Stop accepting new connections
  server.close(() => {
    console.log(`Worker ${process.pid} closed all connections`);
    process.exit(0);
  });

  // Force exit after 10 seconds
  setTimeout(() => {
    console.error(`Worker ${process.pid} forcing shutdown`);
    process.exit(1);
  }, 10000);
}

const server = app.listen(PORT, () => {
  console.log(`Worker ${process.pid} listening on port ${PORT}`);
});

PM2 Ecosystem File

The ecosystem file (ecosystem.config.js) is the recommended way to configure PM2 applications:

jav                                                                                             

ascript// ecosystem.config.js module.exports = { apps: [ { name: 'api-server', script: './src/server.js', instances: 'max',  // Cluster mode with all CPUs exec_mode: 'cluster', // Environment variables env: { NODE_ENV: 'development', PORT: 3000, }, env_production: { NODE_ENV: 'production', PORT: 8080, }, env_staging: { NODE_ENV: 'staging', PORT: 8081, }, // Logging log_file: '/var/log/pm2/api-combined.log', out_file: '/var/log/pm2/api-out.log', error_file: '/var/log/pm2/api-error.log', log_date_format: 'YYYY-MM-DD HH:mm:ss Z', merge_logs: true, // Auto-restart settings watch: false,  // Don't watch in production max_memory_restart: '1G',  // Restart if memory exceeds 1GB restart_delay: 4000,  // Wait 4s between restarts // Crash recovery autorestart: true, max_restarts: 10, min_uptime: '10s', // Graceful shutdown kill_timeout: 5000,  // 5s to gracefully shutdown listen_timeout: 3000,  // 3s to start listening // Instance variables instance_var: 'INSTANCE_ID', }, { name: 'worker', script: './src/worker.js', instances: 2, exec_mode: 'cluster', env_production: { NODE_ENV: 'production', QUEUE_URL: 'redis://localhost:6379', }, cron_restart: '0 0 * * *',  // Restart daily at midnight }, { name: 'scheduler', script: './src/scheduler.js', instances: 1,  // Single instance for cron jobs exec_mode: 'fork', env_production: { NODE_ENV: 'production', }, }, ], // Deployment configuration deploy: { production: { user: 'deploy', host: ['server1.example.com', 'server2.example.com'], ref: 'origin/main', repo: 'git@github.com:username/repo.git', path: '/var/www/production', 'pre-deploy-local': '', 'post-deploy': 'npm ci && pm2 reload ecosystem.config.js --env production', 'pre-setup': '', }, staging: { user: 'deploy', host: 'staging.example.com', ref: 'origin/develop', repo: 'git@github.com:username/repo.git', path: '/var/www/staging', 'post-deploy': 'npm ci && pm2 reload ecosystem.config.js --env staging', }, }, };

Using the Ecosystem File

# Start all apps
pm2 start ecosystem.config.js

# Start specific app
pm2 start ecosystem.config.js --only api-server

# Start with specific environment
pm2 start ecosystem.config.js --env production

# Reload all apps (zero downtime)
pm2 reload ecosystem.config.js

# Deploy to production
pm2 deploy production setup    # First time setup
pm2 deploy production          # Deploy
pm2 deploy production revert 1 # Rollback 1 commit

Linux systemd Services

systemd is the default init system on most modern Linux distributions and provides a powerful way to manage services.

systemd Service File Structure

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js Application
Documentation=https://example.com/docs
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=nodeapp
Group=nodeapp
WorkingDirectory=/var/www/myapp

# Environment
Environment=NODE_ENV=production
Environment=PORT=3000
EnvironmentFile=/var/www/myapp/.env

# Process management
ExecStart=/usr/bin/node /var/www/myapp/dist/server.js
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID

# Restart policy
Restart=always
RestartSec=10
StartLimitInterval=60
StartLimitBurst=3

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/www/myapp/logs /var/www/myapp/uploads

# Resource limits
MemoryMax=1G
CPUQuota=80%

# Logging
StandardOutput=append:/var/log/myapp/stdout.log
StandardError=append:/var/log/myapp/stderr.log
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Managing systemd Services

# Reload systemd after changing service files
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable myapp.service

# Start service
sudo systemctl start myapp.service

# Stop service
sudo systemctl stop myapp.service

# Restart service
sudo systemctl restart myapp.service

# Check status
sudo systemctl status myapp.service

# View logs
sudo journalctl -u myapp.service
sudo journalctl -u myapp.service -f  # Follow logs
sudo journalctl -u myapp.service --since "1 hour ago"

Running PM2 Under systemd

The best practice is to run PM2 itself under systemd:

# Generate systemd startup script
pm2 startup systemd

# This outputs a command like:
# sudo env PATH=$PATH:/usr/bin pm2 startup systemd -u deploy --hp /home/deploy

# Save current PM2 process list
pm2 save

# The saved list will auto-restore on system boot

Generated systemd file for PM2:

# /etc/systemd/system/pm2-deploy.service
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=deploy
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/deploy/.nvm/versions/node/v20.10.0/bin
Environment=PM2_HOME=/home/deploy/.pm2
PIDFile=/home/deploy/.pm2/pm2.pid
Restart=on-failure

ExecStart=/home/deploy/.nvm/versions/node/v20.10.0/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/home/deploy/.nvm/versions/node/v20.10.0/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/home/deploy/.nvm/versions/node/v20.10.0/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

PM2 vs systemd: When to Use Which

Feature Comparison
==================

Feature                    PM2              systemd
──────────────────────────────────────────────────────────────
Cluster mode               Built-in         Manual (multiple units)
Zero-downtime reload       Built-in         Rolling restart
Log management             Built-in         journald
Process monitoring         Built-in + UI    journalctl + custom
Watch & auto-reload        Yes              inotify + script
Memory threshold restart   Yes              MemoryMax (OOM only)
Cron restarts              Yes              systemd timers
Deployment integration     Yes              No
Boot persistence           Needs setup      Native
Non-Node.js apps           Limited          Full support
Security hardening         Basic            Comprehensive
Resource limits            Basic            Comprehensive (cgroups)
Dependencies               Node.js          None
Learning curve             Easy             Moderate

Use PM2 When:

Running Node.js applications exclusively
Need cluster mode for multi-core utilization
Want built-in zero-downtime reloads
Prefer simple deployment workflows
Need log rotation and management
Want a web dashboard (PM2 Plus)

Use systemd When:

Running multiple types of services (not just Node.js)
Need comprehensive security hardening
Want tight resource controls (cgroups, memory limits)
Managing system-level services
Need complex service dependencies
Prefer native Linux tooling

Use Both (PM2 under systemd):

Best of both worlds
PM2 handles Node.js specifics (clustering, reloads)
systemd handles boot persistence and supervision
Most production-ready approach

Zero-Downtime Deployments

PM2 Zero-Downtime Reload

# Graceful reload - waits for new workers before killing old ones
pm2 reload ecosystem.config.js

# Reload specific app
pm2 reload api-server

How it works:

Zero-Downtime Reload Process
============================

Time  │  Old Workers    │  New Workers    │  Traffic
──────┼─────────────────┼─────────────────┼──────────────
T0    │  [1] [2] [3]    │                 │  → [1,2,3]
T1    │  [1] [2] [3]    │  [4] starting   │  → [1,2,3]
T2    │  [1] [2] [3]    │  [4] ready      │  → [1,2,3,4]
T3    │  [1] stopping   │  [4]            │  → [2,3,4]
T4    │  [2] [3]        │  [4] [5] start  │  → [2,3,4]
T5    │  [2] stopping   │  [4] [5]        │  → [3,4,5]
...   │                 │                 │
Tn    │                 │  [4] [5] [6]    │  → [4,5,6]

Graceful Shutdown Implementation

// server.js with graceful shutdown
const express = require('express');
const app = express();

// Track active connections
let connections = new Set();
let isShuttingDown = false;

const server = app.listen(3000, () => {
  console.log('Server started on port 3000');

  // Notify PM2 that we're ready
  if (process.send) {
    process.send('ready');
  }
});

// Track connections
server.on('connection', (conn) => {
  connections.add(conn);
  conn.on('close', () => connections.delete(conn));
});

// Graceful shutdown handler
async function gracefulShutdown(signal) {
  console.log(`Received ${signal}, starting graceful shutdown...`);
  isShuttingDown = true;

  // Stop accepting new connections
  server.close(async () => {
    console.log('HTTP server closed');

    // Close database connections
    await closeDatabase();

    // Close message queue connections
    await closeMessageQueue();

    console.log('All connections closed, exiting');
    process.exit(0);
  });

  // Close existing connections gracefully
  for (const conn of connections) {
    conn.end();
  }

  // Force close after timeout
  setTimeout(() => {
    console.error('Forcing shutdown after timeout');
    for (const conn of connections) {
      conn.destroy();
    }
    process.exit(1);
  }, 10000);
}

// Listen for shutdown signals
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

// Health check that respects shutdown state
app.get('/health', (req, res) => {
  if (isShuttingDown) {
    res.status(503).json({ status: 'shutting_down' });
  } else {
    res.json({ status: 'healthy' });
  }
});

PM2 Configuration for Graceful Shutdown

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'api',
    script: './server.js',
    instances: 'max',
    exec_mode: 'cluster',

    // Wait for 'ready' signal before considering app online
    wait_ready: true,
    listen_timeout: 10000,

    // Give app time to close connections
    kill_timeout: 10000,

    // Shutdown signal
    shutdown_with_message: true,
  }],
};

Log Management

PM2 Log Commands

# View logs
pm2 logs
pm2 logs api-server
pm2 logs api-server --lines 100
pm2 logs --json

# Flush logs
pm2 flush
pm2 flush api-server

# Rotate logs
pm2 install pm2-logrotate

# Configure logrotate
pm2 set pm2-logrotate:max_size 10M
pm2 set pm2-logrotate:retain 7
pm2 set pm2-logrotate:compress true
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
pm2 set pm2-logrotate:rotateModule true
pm2 set pm2-logrotate:workerInterval 30
pm2 set pm2-logrotate:rotateInterval '0 0 * * *'

systemd Journal Logging

# View all logs for a service
journalctl -u myapp.service

# Follow logs in real-time
journalctl -u myapp.service -f

# Logs since last boot
journalctl -u myapp.service -b

# Logs from specific time
journalctl -u myapp.service --since "2024-01-01 00:00:00"
journalctl -u myapp.service --since "1 hour ago"

# JSON output for parsing
journalctl -u myapp.service -o json

# Limit output
journalctl -u myapp.service -n 100

# Filter by priority
journalctl -u myapp.service -p err  # Only errors

Structured Logging for Production

// logger.js - Winston with PM2-friendly configuration
const winston = require('winston');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: {
    service: process.env.SERVICE_NAME || 'api',
    pid: process.pid,
    instanceId: process.env.INSTANCE_ID || 0,
  },
  transports: [
    // Console transport (PM2 captures this)
    new winston.transports.Console({
      format: winston.format.combine(
        winston.format.colorize(),
        winston.format.simple()
      ),
    }),
    // File transport for persistent logs
    new winston.transports.File({
      filename: '/var/log/app/error.log',
      level: 'error',
      maxsize: 10485760, // 10MB
      maxFiles: 5,
    }),
    new winston.transports.File({
      filename: '/var/log/app/combined.log',
      maxsize: 10485760,
      maxFiles: 10,
    }),
  ],
});

module.exports = logger;

Monitoring and Metrics

PM2 Built-in Monitoring

# Real-time monitoring dashboard
pm2 monit

# Process information
pm2 show api-server

# Metrics
pm2 env api-server

# Memory/CPU snapshot
pm2 prettylist

PM2 Plus (Paid Service)

# Link to PM2 Plus for advanced monitoring
pm2 plus

# Or link with specific credentials
pm2 link <secret_key> <public_key>

Custom Metrics with PM2

// Add custom metrics
const io = require('@pm2/io');

// Counter metric
const requestCounter = io.counter({
  name: 'Requests',
  id: 'app/requests',
});

// Meter metric (requests per second)
const requestMeter = io.meter({
  name: 'req/sec',
  id: 'app/requests/sec',
});

// Histogram metric
const responseTime = io.histogram({
  name: 'Response Time',
  id: 'app/response-time',
  measurement: 'mean',
});

// Gauge metric
const activeConnections = io.metric({
  name: 'Active Connections',
  id: 'app/connections',
});

// Express middleware to track metrics
app.use((req, res, next) => {
  const start = Date.now();
  requestCounter.inc();
  requestMeter.mark();

  res.on('finish', () => {
    const duration = Date.now() - start;
    responseTime.update(duration);
  });

  next();
});

// Update active connections gauge
server.on('connection', () => {
  activeConnections.set(server.connections);
});

Prometheus Metrics Export

// metrics.js - Prometheus metrics for Node.js
const client = require('prom-client');
const express = require('express');

// Create a Registry
const register = new client.Registry();

// Add default metrics (CPU, memory, etc.)
client.collectDefaultMetrics({ register });

// Custom metrics
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.3, 0.5, 0.7, 1, 3, 5, 7, 10],
});
register.registerMetric(httpRequestDuration);

const httpRequestTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
});
register.registerMetric(httpRequestTotal);

// Middleware
function metricsMiddleware(req, res, next) {
  const end = httpRequestDuration.startTimer();

  res.on('finish', () => {
    const route = req.route?.path || req.path;
    const labels = {
      method: req.method,
      route: route,
      status_code: res.statusCode,
    };
    end(labels);
    httpRequestTotal.inc(labels);
  });

  next();
}

// Metrics endpoint
function metricsHandler(req, res) {
  res.set('Content-Type', register.contentType);
  register.metrics().then(data => res.send(data));
}

module.exports = { metricsMiddleware, metricsHandler, register };

Running Multiple Applications

PM2 Multiple Apps

// ecosystem.config.js with multiple apps
module.exports = {
  apps: [
    {
      name: 'api',
      script: './services/api/server.js',
      instances: 4,
      exec_mode: 'cluster',
      env_production: {
        PORT: 3000,
        NODE_ENV: 'production',
      },
    },
    {
      name: 'admin',
      script: './services/admin/server.js',
      instances: 2,
      exec_mode: 'cluster',
      env_production: {
        PORT: 3001,
        NODE_ENV: 'production',
      },
    },
    {
      name: 'websocket',
      script: './services/websocket/server.js',
      instances: 2,
      exec_mode: 'cluster',
      env_production: {
        PORT: 3002,
        NODE_ENV: 'production',
      },
    },
    {
      name: 'worker-email',
      script: './services/workers/email.js',
      instances: 1,
      exec_mode: 'fork',
      cron_restart: '0 */6 * * *',
    },
    {
      name: 'worker-reports',
      script: './services/workers/reports.js',
      instances: 1,
      exec_mode: 'fork',
    },
  ],
};

systemd Multiple Services

# /etc/systemd/system/myapp-api.service
[Unit]
Description=MyApp API Server
After=network.target postgresql.service redis.service
Requires=postgresql.service

[Service]
Type=simple
User=nodeapp
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node services/api/server.js
Restart=always

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/myapp-worker.service
[Unit]
Description=MyApp Background Worker
After=network.target myapp-api.service redis.service
Requires=redis.service

[Service]
Type=simple
User=nodeapp
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node services/workers/main.js
Restart=always

[Install]
WantedBy=multi-user.target

Creating a target for all services:

# /etc/systemd/system/myapp.target
[Unit]
Description=MyApp All Services
Requires=myapp-api.service myapp-worker.service
After=myapp-api.service myapp-worker.service

[Install]
WantedBy=multi-user.target

# Start all myapp services
sudo systemctl start myapp.target

# Enable all to start on boot
sudo systemctl enable myapp.target

Security Best Practices

PM2 Security

// ecosystem.config.js with security settings
module.exports = {
  apps: [{
    name: 'api',
    script: './server.js',

    // Run as non-root user
    uid: 'nodeapp',
    gid: 'nodeapp',

    // Limit resources
    max_memory_restart: '500M',

    // Don't expose PM2 info
    pmx: false,
  }],
};

systemd Security Hardening

[Service]
# Run as unprivileged user
User=nodeapp
Group=nodeapp

# Filesystem restrictions
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
ReadWritePaths=/var/www/myapp/logs /var/www/myapp/uploads

# Network restrictions
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
PrivateNetwork=false

# System call filtering
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources

# Capabilities
CapabilityBoundingSet=
AmbientCapabilities=

# Memory restrictions
MemoryDenyWriteExecute=true

# Misc security
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true

Complete Production Setup

Here's a complete setup combining PM2 with systemd:

#!/bin/bash
# setup-production.sh

set -e

APP_NAME="myapp"
APP_USER="nodeapp"
APP_DIR="/var/www/$APP_NAME"
NODE_VERSION="20"

echo "=== Setting up $APP_NAME production environment ==="

# 1. Create application user
sudo useradd -r -s /bin/false $APP_USER || true

# 2. Create directories
sudo mkdir -p $APP_DIR
sudo mkdir -p /var/log/$APP_NAME
sudo chown -R $APP_USER:$APP_USER $APP_DIR
sudo chown -R $APP_USER:$APP_USER /var/log/$APP_NAME

# 3. Install Node.js via nvm (as app user)
sudo -u $APP_USER bash << 'EOF'
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
source ~/.nvm/nvm.sh
nvm install 20
nvm use 20
npm install -g pm2
EOF

# 4. Setup PM2 systemd service
sudo -u $APP_USER bash -c 'source ~/.nvm/nvm.sh && pm2 startup systemd -u nodeapp --hp /home/nodeapp'

# 5. Enable and start PM2 service
sudo systemctl enable pm2-$APP_USER
sudo systemctl start pm2-$APP_USER

echo "=== Production setup complete ==="
echo "Deploy your app to $APP_DIR and run:"
echo "sudo -u $APP_USER bash -c 'cd $APP_DIR && pm2 start ecosystem.config.js --env production'"
echo "sudo -u $APP_USER bash -c 'pm2 save'"

Conclusion

Both PM2 and systemd are excellent tools for running Node.js applications in production. The key takeaways:

PM2 excels at Node.js-specific features: clustering, zero-downtime reloads, and built-in monitoring
systemd provides robust system-level service management with comprehensive security hardening
The combination of both (PM2 under systemd) is the most production-ready approach
Always implement graceful shutdown for zero-downtime deployments
Use structured logging and metrics for observability

At Dr Dangs Lab, we run our Node.js microservices using PM2 in cluster mode, supervised by systemd, with Nginx as a reverse proxy. This stack has provided us with 99.9% uptime and seamless deployments.

Docker & Kubernetes Deployment Guide - Containerized Node.js deployments
GitHub Actions CI/CD Complete Guide - Automate your deployments
Nginx Reverse Proxy & Load Balancing Guide - Front your Node.js apps with Nginx
Apache vs Nginx Web Server Comparison - Choose the right reverse proxy