KubeBlogs - DevOps and Cloud content

Sign in Subscribe

We Audited a Customer’s Pull Request Security Scans. Here's What We Found.

Pull Request Security

We Audited a Customer’s Pull Request Security Scans. Here's What We Found.

Overview We audited the pull request security scans of one of our customers, a US fintech platform that sends text messages and takes payments for banks, credit unions, lenders and collections firms. Their software handles consumer account details and payments, and their own customers work in heavily regulated sectors. The

Terraform repository restructuring with Terragrunt, shared configuration, and isolated state.

How We Helped a Customer Refactor 34 Terraform Root Modules with Terragrunt Without Disrupting Production

Introduction Terraform makes it easy to provision infrastructure as code, but managing a growing codebase becomes more challenging as the number of services and environments increases. We faced this challenge while working with a client whose AWS infrastructure repository had accumulated 45 provider "aws" blocks across 34 root

Renaming 42 AWS ECS services with Terraform using ALB listener rules to switch traffic safely.

How We Renamed 42 AWS ECS Services with Terraform

Overview A simple Terraform change turned into a production risk. We needed to rename 42 AWS ECS services across four environments to follow a consistent naming convention. The problem was that ECS does not support renaming a service in place. Terraform would destroy the existing service and create a new

Kubernetes Grace Period Is Not Your Grace Period

Kubernetes Grace Period Is Not Your Grace Period

Overview Your Kubernetes grace period might not be your real grace period. terminationGracePeriodSeconds defines the Pod's termination grace-period budget, but it doesn't guarantee that your application gets that entire window. Inside the container, other components can have their own shutdown rules and timeouts. If one

Bash History Expansion: Why `!` in Passwords Can Cause 401 Errors

Bash History Expansion: Why `!` in Passwords Can Cause 401 Errors

Overview I lost most of a morning to an authentication failure that turned out to have nothing to do with authentication. The fix was obvious afterward, but the problem wasn't where you'd normally look. Hitting an API with -u user:password from an interactive terminal is

Kubernetes pod evicted under memory pressure with QoS and resource warning

Our Kubernetes Pod Kept Getting Evicted. The Real Problem Wasn't Memory.

Overview For two weeks, a log aggregation pod kept getting evicted whenever the node ran short on memory. It recovered quickly enough that, from the outside, it looked like a normal restart. What looked fine at the application level turned out to be counterintuitive. The pod wasn't being

Kubernetes Traffic Skew: Why One Pod Gets Hot While Others Sit Idle

Kubernetes Traffic Skew: Why One Pod Gets Hot While Others Sit Idle

Introduction Your Kubernetes autoscaler might be solving a problem that doesn't exist, while ignoring the one that does. One pod pinned at 90% CPU, four identical replicas idling under 5%. It's assumed to be a capacity problem, and HPA scales out. But it doesn't

Docker cache restored in CI but build still starts from scratch

We Restored the Docker Cache. The Build Still Started From Scratch.

Overview A successful restore doesn’t always mean the Docker builder is actually using that cache. CI can restore Docker cache into the job while docker build still starts from scratch with no layer reuse. We ran into exactly that kind of cache miss on one of our suites. Once