Database lock contention causing API latency spikes
Service
cupixworks-api
Occurrences
8
Status
Reported
Confidence
85%
Actionability
44%
Cluster ID
57277235-3a21-46b8-911a-57a7a713e982
First Seen
2026-07-08 03:55 UTC
Last Seen
2026-08-04 11:07 UTC
Datadog
SlackActions
Database lock contention causing API latency spikes
What Happened#
2026-07-08 12:55:32 KST에 Retool 클라이언트가 호출한 GET /api/v1/admin/teams?filter=id:87&fields=id,account_manager,customer_success_managers 요청 한 건이 총 18506ms(그중 DB 12047ms) 걸려 응답했다. 응답 자체는 HTTP 200이었고 latency 클러스터로만 잡혔다. 같은 시각(±2분) 다른 여러 엔드포인트(PanosController, ElementTracesController, SessionsController 등)도 15초 이상의 지연을 겪었다.
Quick Facts#
| Field | Value |
|---|---|
| resource_name | Api::V1::Admin::TeamsController#index |
| controller | Api::V1::Admin::TeamsController |
| action | index |
| top_frame | app/controllers/api/v1/admin/teams_controller.rb:17-27 |
| params | filter=id:87, fields=[id, account_manager, customer_success_managers] |
| user | miriam.kim@cupix.com (user.id 6942, team.id 133 admin) |
| user_agent | Retool/2.0 |
| duration | 18506.22ms |
| db | 12047.42ms (전체의 65%) |
| view | 0.1ms |
| serialization | 4ms |
| request_id | 1a4c5cbe-6f91-4587-9fa2-e491a35ac21e |
| host | ip-10-1-80-134.us-west-2.compute.internal |
| deploy | production-us-west-2-20260708t0134z0-a1160ff5-cupixworks |
| env | production, us-west-2 |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| Retool 관리자 도구 (admin domain) | 1 | 팀 조회가 ~18.5초 지연 — 사용자에게는 UI 로딩 지연으로 체감 |
| 동시 요청자 (pano upload, element traces 등) | 다수 | 같은 창(12:53-12:56 KST)에 여러 API가 15초+ 지연 (연쇄 영향) |
Timeline#
- 2026-07-08 12:53:00 KST 부근 —
PanosController#bulk,ElementTracesController#refresh등 다수의 API에서 15초+ 지연이 시작 (Datadog@duration:>15000검색) - 2026-07-08 12:55:14 KST — 같은 사용자가
filter=id:1022로 조회한 첫 요청은 정상(728ms, DB 268ms) - 2026-07-08 12:55:12 KST — 문제의
filter=id:87요청이 접수 (Datadog span first_seen) - 2026-07-08 12:55:32 KST — 문제의 요청이 200 OK로 완료(18506ms, DB 12047ms)
- 2026-07-08 12:56:14 KST — 광범위한 지연 완화 (마지막 slow log)
Scoring#
Confidence 85% · Actionability 44%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 10 code refs, 5 log queries |
| Code path explicitness | 3/3 | 10 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 5/5 | trivial |
| Affected files | 1/5 | 20 file(s) |
| Single repo | 1/3 | 5 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |