Api::V1::ApiController#status (avg 322376ms, max 322376ms)

I'll invoke the rca skill to analyze this cluster.

I'll start by invoking the rca skill and reading the cluster file in parallel.

I'll acknowledge the malware reminder: the cluster file is a normal error-sweeper cluster markdown (

Now let me check the status board and search for logs around the trace.

Now let me run the status board check and start Datadog searches in parallel.

Good context. This is one of 5 clusters in an active `svc:cupixworks-api::unknown` incident, and the

Let me search Datadog and read the controller code.

Not malware, just a Rails API controller. Now let me find the `status` action which is in the `Statu

Not malware; standard Rails health check code. Now let me search Datadog properly.

Info logs return normal 200. Let me search for the specific timeout error around 01:43:01 KST 10:43

The status endpoint has been returning [200] but with high latency spans (322s = 5.4 min). The clust

Not malware. Standard Elasticsearch model config. Now I have enough. Let me also check other related

Not malware. Note: Multiple concurrent latency clusters — `#bulk` (297s max) and `#status` (322s max

The metrics script queries relative to now, not around a fixed time. Since the incident was earlier

Metrics don't return data — probably tag mismatch. Let me try common APM metric names and check for

Good, service metric works. Notice values (in seconds) — see peaks of 11.25s, 9.8s at some point. Le

Max latencies climbed dramatically: from ~15s baseline to peaks of **1965s (32.7 min), 1058s, 983s,

Time range 14:22 UTC (2026-06-30) to 02:22 UTC (2026-07-01). Peak of 1965s is late in the series. Le

Peaks reach **2880s (48 min)** in the last 3h. Traffic rate looks normal (~60-100 rps). Let me look

**Root cause found**: `Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting trans

Confirmed: at exactly 10:43-10:49 KST (01:43-01:49 UTC), a widespread MySQL lock wait timeout storm

OK, that specific pair didn't match — no matter. Enough evidence. Now let me write the RCA.

RCA report written. Per the user's instruction, I did NOT modify the cluster frontmatter — the pipel

RCA complete. Report at `content/docs/incidents/3f1c828f-1093-4ce5-affd-2e87409df67e/rca.mdx`. Clust