Deploy.
Survive.
Scale.

# application.properties
management.metrics.tags.version=${APP_VERSION:unknown}
management.metrics.tags.pod=${POD_NAME:unknown}

# MDC logging — add version to every log line

@Component
public class VersionMdcFilter implements Filter {
    @Value("${app.version:unknown}")
    private String version;

    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
            throws IOException, ServletException {
        MDC.put("appVersion", version);
        try { chain.doFilter(req, res); }
        finally { MDC.remove("appVersion"); }
    }
}

# Rollback to previous revision
argocd app rollback payment-service --revision 42

# Or kubectl — rollback Deployment
kubectl rollout undo deployment/payment-service -n production
kubectl rollout status deployment/payment-service -n production

Exit code 137 = killed by OS (OOM killer). Kubernetes enforces resources.limits.memory at cgroup level. Java has historically been bad at this because JVM doesn't know about container memory limits — it reads total host memory and sets heap accordingly.

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "768Mi"  # limit = 1.5x of MaxRAMPercentage allocation
    cpu: "1000m"
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=60.0  # 60% of 768Mi = ~460Mi heap
            -XX:InitialRAMPercentage=50.0
            -XX:+UseG1GC
            -XX:+HeapDumpOnOutOfMemoryError
            -XX:HeapDumpPath=/tmp/heapdump.hprof
            -Xss256k"  # Reduce thread stack size (default 512k-1MB)

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 60  # Spring Boot + Hibernate init can take 30-45s
  periodSeconds: 10
  failureThreshold: 3       # 3 failures × 10s = 30s grace before kill
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 20
  periodSeconds: 5

Periodic latency spikes at regular intervals = GC pause is the first suspect. The pattern (every 2 hours) suggests Full GC triggered by old-gen filling up.

# G1GC — recommended for API services (low pause)
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200          # Target max pause (G1 will try to meet this)
-XX:G1HeapRegionSize=16m          # Tune for large object allocation
-XX:InitiatingHeapOccupancyPercent=45  # Trigger concurrent GC earlier
-XX:+G1UseAdaptiveIHOP           # Let JVM adapt threshold
-Xlog:gc*:file=/tmp/gc.log:time,uptime:filecount=5,filesize=20m

# ZGC — for Java 15+ with <10ms pause requirement (fintech/HFT)
-XX:+UseZGC
-XX:SoftMaxHeapSize=4g            # Aggressive GC before hitting -Xmx

// PATTERN 1: Unbounded static cache
public class UserCache {
    private static final Map<String, User> cache = new HashMap<>();
    // Grows forever — use Caffeine/Guava with size limit and TTL
}

// FIX:
private final Cache<String, User> cache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .expireAfterWrite(Duration.ofMinutes(30))
    .build();

// PATTERN 2: ThreadLocal leak in thread pool
private static ThreadLocal<UserContext> ctx = new ThreadLocal<>();

// Never cleaned — pool threads reused, ThreadLocal stays
// FIX: always clean in finally block
try {
    ctx.set(new UserContext(userId));
    processRequest();
} finally {
    ctx.remove();  // CRITICAL in thread pools
}

Metaspace holds class metadata (class definitions, method bytecode). Unlike heap, Metaspace is native memory (off-heap). By default it grows until OS memory limit — can silently consume all available native memory.

Causes:

• Dynamic class generation (Groovy scripts, CGLIB proxies, ASM) creating classes that are never unloaded
• Multiple hot-deploys in app servers (Tomcat/JBoss) without ClassLoader cleanup → ClassLoader leak
• Plugins/OSGi bundles repeatedly loaded without unloading old versions

# Cap Metaspace to force GC of dead classes before OOM
-XX:MaxMetaspaceSize=256m

# Monitor: if Metaspace grows near this, you have a class leak
-verbose:class         # log every class load/unload

# JFR recording: capture class loading events
jcmd <pid> JFR.start duration=60s filename=metaspace.jfr
# Analyze in JMC: look for classes loaded but never unloaded

private final ReentrantLock lockA = new ReentrantLock();
private final ReentrantLock lockB = new ReentrantLock();

public void transfer() throws InterruptedException {
    boolean gotA = false, gotB = false;
    try {
        gotA = lockA.tryLock(100, TimeUnit.MILLISECONDS);
        gotB = lockB.tryLock(100, TimeUnit.MILLISECONDS);
        if (gotA && gotB) {
            doTransfer();
        } else {
            // Back off and retry — no deadlock possible
            throw new RetryableException("Could not acquire locks");
        }
    } finally {
        if (gotB) lockB.unlock();
        if (gotA) lockA.unlock();
    }
}

Root causes in order of likelihood:

• Pool size too small for traffic — all connections in use, requests queue up
• Connection leak — @Transactional method opens connection but exception path doesn't close it
• Slow queries holding connections for long durations — blocking pool
• DB-side: max_connections exhausted, all denied by DB server

# application.properties
spring.datasource.hikari.maximum-pool-size=20        # Default is 10 — tune per service
spring.datasource.hikari.minimum-idle=5
spring.datasource.hikari.connection-timeout=5000     # Fail fast instead of 30s wait
spring.datasource.hikari.idle-timeout=300000
spring.datasource.hikari.max-lifetime=600000         # Recycle before DB kills idle connections
spring.datasource.hikari.leak-detection-threshold=2000  # Log stack trace if conn held >2s

Optimal pool size formula (Little's Law):

@Test
void detectConnectionLeak() {
    // Enable leak detection in test config
    // Check HikariCP metrics: active connections should return to baseline
    HikariPoolMXBean pool = dataSource.getHikariPoolMXBean();
    callServiceMethod();
    assertThat(pool.getActiveConnections()).isZero(); // all returned
}

# Start JFR recording (JDK 11+)
jcmd <pid> JFR.start duration=120s filename=/tmp/recording.jfr settings=profile

# Or: always-on with rolling file (low overhead)
-XX:StartFlightRecording=disk=true,maxage=1h,maxsize=500m,\
  filename=/recordings/app.jfr,settings=default

# async-profiler — CPU flame graph
./profiler.sh -d 30 -f /tmp/flamegraph.html <pid>

GraalVM compiles Java to native binary (AOT). Startup: 50ms vs 8 seconds. Memory: 60% reduction. Great for Kubernetes where pods scale rapidly.

What breaks:

• Dynamic class loading (CGLIB proxies, runtime Groovy) — must configure reflection hints
• Many libraries not yet native-image compatible (check GraalVM reachability metadata repo)
• Build time: 3–5 minutes vs 30 seconds for JVM — slower CI
• No JIT optimization — peak throughput lower than JVM for long-running services
• Debugging native binaries is harder — no JFR, limited profiling tools

A health endpoint returning 200 only means Spring Boot's Actuator health check passed. It doesn't mean your business logic works. Common scenarios:

• Health check only pings DB SELECT 1 — but the query users actually run is deadlocked
• External dependency (payment gateway) is down — but health check doesn't include it
• Feature flag misconfigured — requests routed to disabled code path, returns 200 with empty body
• Cache serving stale data — "healthy" but responses are hours old