<!--
Idempotency Debugging
Part of the Bles Software agent skills repository:
https://www.bles-software.com/claude-skills

Install for Claude Code:
  mkdir -p ~/.claude/skills/idempotency-debugging
  mv idempotency-debugging.md ~/.claude/skills/idempotency-debugging/SKILL.md

Install for Codex:
  mkdir -p ~/.codex/skills/idempotency-debugging
  mv idempotency-debugging.md ~/.codex/skills/idempotency-debugging/SKILL.md

For Gemini CLI and other agents, drop the body of this file into the
project instruction file the agent already reads.
-->

---
name: Idempotency Debugging
description: Debug and fix duplicate API requests caused by timeout + retry logic on non-idempotent APIs
---

# Idempotency Debugging Skill

Debug and fix duplicate API requests caused by timeout + retry logic combinations when third-party APIs are non-idempotent.

## When to Use

Use this skill when:
- Multiple duplicate records appear in external systems (e.g., 2-3 identical posts on Facebook/Instagram)
- Users report "posting multiple times" or "duplicate entries"
- Logs show retry patterns like "attempt 1/3 failed", "attempt 2/3 failed"
- Timeout errors (`ECONNABORTED`, `ETIMEDOUT`) appear followed by retries

## The Pattern

**Root Cause:** Timeout + Retry on Non-Idempotent APIs = Duplicates

```
Timeline of disaster:
1. Request 1 sent → takes 20s on server → client times out at 15s
2. Client sees ECONNABORTED → treats as "retriable" error
3. Request 2 (retry) sent → server processes it
4. Request 1 finishes on server → record created
5. Request 2 finishes on server → ANOTHER record created
Result: 2-3 duplicates per operation
```

## Diagnostic Steps

### Step 1: Check Logs for Retry Patterns
```bash
grep -E "(attempt.*failed|ECONNABORTED|retry|timeout)" logs/*.log | tail -50
```

Look for patterns like:
```
[publish] instagram attempt 1/3 failed (status=ECONNABORTED); retriable=true
[publish] instagram attempt 2/3 failed (status=ECONNABORTED); retriable=true
```

### Step 2: Identify Timeout vs Processing Time

Check the configured timeout:
```bash
grep -i "timeout" .env
# Example: LATE_HTTP_TIMEOUT_MS=15000
```

Check actual API response times (from success logs):
```bash
grep "API response" logs/*.log | tail -20
# Look at timestamps between "attempting" and "response"
```

### Step 3: Check Retry Logic

Search for retry code:
```bash
grep -rn "retry\|attempts\|retriable" src/*.js | head -30
```

Red flag code pattern:
```javascript
// BAD: Treats timeout as retriable
const retriable = code === 'ECONNABORTED' || status === 429 || status >= 500;
```

## Solutions

### Solution 1: Increase Timeout (Quick Fix)

If API typically takes 20-30s, set timeout to 60s:
```bash
# .env
LATE_HTTP_TIMEOUT_MS=60000
```

### Solution 2: Remove Timeout from Retriable Errors (Critical Fix)

```javascript
// BEFORE (causes duplicates)
const retriable = code === 'ECONNABORTED' || status === 429 || (status >= 500 && status < 600);

// AFTER (safe)
const isTimeout = code === 'ECONNABORTED';
const retriable = !isTimeout && (status === 429 || (status >= 500 && status < 600));
if (isTimeout) {
  console.error(`[publish] ${platform} TIMEOUT - NOT retrying to avoid duplicates`);
}
```

### Solution 3: Idempotency Keys (Best, if API supports it)

Add a unique request ID that the server uses to deduplicate:
```javascript
const requestId = `${operation}-${Date.now()}-${Math.random().toString(36).slice(2,8)}`;
const response = await api.createPost(payload, {
  headers: { 'Idempotency-Key': requestId }
});
```

### Solution 4: Pre-Check Before Retry

Before retrying, check if the operation already succeeded:
```javascript
// After timeout, check if post was created
const recentPosts = await api.listPosts({ since: Date.now() - 60000 });
if (recentPosts.some(p => p.content === payload.content)) {
  console.log('Post already created, skipping retry');
  return recentPosts[0];
}
```

## Key Principle

**Non-idempotent operations (create, post, insert) should NEVER retry on timeout.**

The request may have succeeded server-side even though the client didn't receive confirmation.

| Error Type | Safe to Retry? | Why |
|------------|----------------|-----|
| Timeout (ECONNABORTED) | NO | Request may have succeeded |
| 429 Rate Limit | YES | Server explicitly rejected |
| 5xx Server Error | MAYBE | Server failed before processing |
| Network Error | MAYBE | Depends on when it failed |

## Verification

After applying fix:
1. Restart service: `pm2 restart app --update-env`
2. Trigger test operation
3. Check logs - should NOT see "attempt 2/3" after timeout
4. Verify only ONE record created in external system