Debugging Production Issues: A Systematic Approach

You deploy. Things break. Users complain. Your local environment works fine. Welcome to production debugging.

Step 1: Stop Guessing

First rule: don't randomly change things hoping the problem goes away. I've watched people restart services, tweak config values, and deploy "fixes" without understanding the actual problem. That's how you make things worse.

Start by answering these:

• How many people are affected? One user with weird data, or everyone?
• When did it start? Right after a deploy, or random Tuesday at 3am?
• Can we roll back? If yes, do it. Fix forward later.

For Sysmon-CLI, I've had bugs that only showed up under sustained load—single-user testing was useless. For Zvezda, I've had issues that only triggered with specific Git configurations. Context matters.

Gather Actual Data

Don't trust your memory of what the system should be doing. Check:

# What changed recently?
git log --since="2 hours ago" --oneline

# What's actually in the logs?
grep "ERROR\|WARN" /var/log/app.log | tail -100

# Is the system even healthy?
top  # or sysmon-cli if you have it

Half the time, the "mysterious" issue is just high CPU or memory exhaustion. Check the obvious stuff first.

The Bisect Trick

If the logs are garbage (they usually are), use git bisect to find the breaking commit:

git bisect start
git bisect bad HEAD
git bisect good v1.2.0

Then test each bisect point. This is slower than guessing but faster than being wrong.

Reproduce It Locally

Can't fix what you can't reproduce. Try to match production:

• Same Go version, same OS, same dependencies
• Production data (sanitized if needed)
• Same environment variables

I had a Zvezda bug that only triggered when the Git config had core.autocrlf=true. Local repro was impossible until I actually looked at the user's Git config.

Instrumentation When You're Blind

Sometimes logs don't tell you enough. Add more:

log.Printf("DEBUG: processing repo %s, commit %s", repo, commit)

Yes, this means deploying to debug. That's fine. Just remove the debug logs after.

For Sysmon-CLI, I've added temporary metrics collection to understand exactly what system calls were being made. Can't optimize what you can't measure.

Common Culprits

Memory leaks. Symptoms: memory usage creeps up, eventually OOM. Use pprof to find the leak:

go tool pprof http://localhost:6060/debug/pprof/heap

Goroutine leaks. Similar but with goroutines. Check /debug/pprof/goroutine.

Database connection exhaustion. Your connection pool is too small or you're leaking connections. Check:

SHOW PROCESSLIST;  -- MySQL
SELECT * FROM pg_stat_activity;  -- Postgres

External API rate limits. Intermittent 429s mean you're hitting someone's rate limit. Add backoff/retry logic.

After the Fix

Write down what happened. Not for blame—for learning.

Postmortem template:

• What broke and when
• What we tried (even the wrong guesses)
• What actually fixed it
• How we prevent it next time

Then add a test:

func TestThatBugWeHadInProduction(t *testing.T) {
    // Reproduce the exact scenario
    // Verify it's fixed
}

Regression tests are documentation of pain.

Prevention

The best production bugs are the ones you catch before users do:

Local tools that work. If you can't reproduce prod issues locally, your dev environment sucks. Fix it.

Gradual rollouts. Deploy to 1% of traffic first. See if things explode.

Actual monitoring. Not just "is it up?" but "is it slow? is it leaking? is it erroring?"

For my tools, I use:

• Go's builtin pprof endpoints
• Structured logs (JSON)
• Basic Prometheus metrics when needed

You don't need a $10k/month observability platform. You need logs you can search and metrics you can graph.

Tools I Actually Use

pprof: Go's profiler. CPU, memory, goroutines, everything.

strace: See what syscalls a process is making. Useful when things are mysteriously slow.

tcpdump: Network issues? Capture the packets.

git bisect: When you know it worked in the past.

Sysmon-CLI: Because sometimes you just need to see if CPU/memory is spiking. Shameless plug.

The Uncomfortable Truth

Most production bugs are boring:

• Forgot to handle an error
• Race condition in concurrent code
• Assumption about data that's wrong
• External dependency is down

The hard part isn't fixing them—it's reproducing them. Invest in making prod behavior visible and local reproduction possible.

And remember: production bugs are proof you shipped something. That's better than perfect code that never deploys.