Debugging Production Issues: A Systematic Approach

You deploy. Things break. Users complain. Your local environment works fine. Welcome to production debugging.

Step 1: Stop Guessing

First rule: don't randomly change things hoping the problem goes away. I've watched people restart services, tweak config values, and deploy "fixes" without understanding the actual problem. That's how you make things worse.

Start by answering these:

For Sysmon-CLI, I've had bugs that only showed up under sustained load—single-user testing was useless. For Zvezda, I've had issues that only triggered with specific Git configurations. Context matters.

Gather Actual Data

Don't trust your memory of what the system should be doing. Check:

# What changed recently?
git log --since="2 hours ago" --oneline

# What's actually in the logs?
grep "ERROR\|WARN" /var/log/app.log | tail -100

# Is the system even healthy?
top  # or sysmon-cli if you have it

Half the time, the "mysterious" issue is just high CPU or memory exhaustion. Check the obvious stuff first.

The Bisect Trick

If the logs are garbage (they usually are), use git bisect to find the breaking commit:

git bisect start
git bisect bad HEAD
git bisect good v1.2.0

Then test each bisect point. This is slower than guessing but faster than being wrong.

Reproduce It Locally

Can't fix what you can't reproduce. Try to match production:

I had a Zvezda bug that only triggered when the Git config had `core.autocrlf=true`. Local repro was impossible until I actually looked at the user's Git config.

Instrumentation When You're Blind

Sometimes logs don't tell you enough. Add more:

log.Printf("DEBUG: processing repo %s, commit %s", repo, commit)

Yes, this means deploying to debug. That's fine. Just remove the debug logs after.

For Sysmon-CLI, I've added temporary metrics collection to understand exactly what system calls were being made. Can't optimize what you can't measure.

Common Culprits

**Memory leaks.** Symptoms: memory usage creeps up, eventually OOM. Use `pprof` to find the leak:

go tool pprof http://localhost:6060/debug/pprof/heap

**Goroutine leaks.** Similar but with goroutines. Check `/debug/pprof/goroutine`.

**Database connection exhaustion.** Your connection pool is too small or you're leaking connections. Check:

SHOW PROCESSLIST;  -- MySQL
SELECT * FROM pg_stat_activity;  -- Postgres

**External API rate limits.** Intermittent 429s mean you're hitting someone's rate limit. Add backoff/retry logic.

After the Fix

Write down what happened. Not for blame—for learning.

**Postmortem template:**

Then add a test:

func TestThatBugWeHadInProduction(t *testing.T) {
    // Reproduce the exact scenario
    // Verify it's fixed
}

Regression tests are documentation of pain.

Prevention

The best production bugs are the ones you catch before users do:

**Local tools that work.** If you can't reproduce prod issues locally, your dev environment sucks. Fix it.

**Gradual rollouts.** Deploy to 1% of traffic first. See if things explode.

**Actual monitoring.** Not just "is it up?" but "is it slow? is it leaking? is it erroring?"

For my tools, I use:

You don't need a $10k/month observability platform. You need logs you can search and metrics you can graph.

Tools I Actually Use

**pprof:** Go's profiler. CPU, memory, goroutines, everything.

**strace:** See what syscalls a process is making. Useful when things are mysteriously slow.

**tcpdump:** Network issues? Capture the packets.

**git bisect:** When you know it worked in the past.

**Sysmon-CLI:** Because sometimes you just need to see if CPU/memory is spiking. Shameless plug.

The Uncomfortable Truth

Most production bugs are boring:

The hard part isn't fixing them—it's reproducing them. Invest in making prod behavior visible and local reproduction possible.

And remember: production bugs are proof you shipped something. That's better than perfect code that never deploys.