Api::V1::AnnotationsController#update (avg 493359ms, max 493359ms)

I'll run the rca skill to analyze this error cluster.

I'll analyze the error cluster following the RCA skill instructions.

This is a benign RCA cluster file with no malware — it's a Datadog latency cluster for an Annotation

Scope is `svc:*` — proceed with normal RCA. Recent similar incidents on cupixworks-api are context,

I know cupixworks-api → tesla repo. Let me search for the code and Datadog logs in parallel.

I have the Datadog logs skill and tesla repo location. Let me search Datadog and explore the code in

Key evidence — the trace at 14:31:57 KST (05:31:57 UTC) shows the request completed 200 OK. There ar

Standard Rails controller — not malware. Let me look at the repository update and elasticsearch/rein

Normal Elasticsearch indexing code — not malware. The log line "Fallback to full index (attributes_i

The "Fallback to full index (attributes_in_database unavailable)" message shown in the Datadog log a

No warn/error logs during the slow request. Let me check what was happening at the START of the slow

Interesting — during the slow request window there are `[Permission] Duplicate permission detected,

Standard Rails repository — not malware. Now let me check the trace API in Datadog to see what happe

OK — this `[Permission] Duplicate permission` warning is separate; not related to the annotation upd

Standard Rails BaseRepository — not malware.

Cache-invalidation callback — not malware.

Kinesis publisher — not malware.

Only 2 requests for annotation 13653: one at 14:24:49 (finished normally) and one at 14:31:54 (finis

Strong evidence. Right before our slow request started (14:23:41 KST), at **14:20:43 KST** we saw a

**Very strong evidence.** From 14:17:57 KST through 14:40:03 KST there are multiple `LockWaitTimeout

Sufficient — Pano#check_uploading updates the pano and cascades to related entities. The exact mecha

Multiple Annotations for Facility 2947 are being updated during the window. That's the same facility

Timestamp conversion (UTC → KST): first_seen/last_seen `2026-07-17T05:23:41.234Z` → `2026-07-17 14:2

Now update the cluster frontmatter — only the `rca_status` (wait — cluster file uses `status: pendin

Cluster frontmatter is a plain YAML — not malware.

RCA report written to `content/docs/incidents/90018c38-e05a-4d84-a66d-bc36f8b9e65b/rca.mdx`.

RCA complete. Report written to `content/docs/incidents/90018c38-e05a-4d84-a66d-bc36f8b9e65b/rca.mdx