Searchable#_update_document — synchronous ES callback pool exhaustion
Service
cupixworks-api
Occurrences
5,902
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
bd35bc1d-ced5-4866-9841-d14d463894ee
First Seen
2026-06-24 15:06 UTC
Last Seen
2026-08-05 02:21 UTC
Datadog
SlackActions
Searchable#_update_document — synchronous ES callback pool exhaustion
What Happened#
2026-06-25 00:06 KST부터 09:21 KST 사이 us-west-2 production cupixworks-api의 Api::V1::PanosController#index 트레이스가 평균 31s, 최대 93s 까지 늘어졌다. 같은 시간대에 cupixworks-api 서비스 전체 평균 응답시간이 평소 ~0.1s에서 ~0.5s로 상승했고, Elasticsearch 쿼리 평균 시간도 0.05s → 0.37s로 약 7배 증가했다. 트레이스 19건, 단일 endpoint(Api::V1::PanosController#index) 단일 region(us-west-2)에서 관측된 latency 클러스터다.
Quick Facts#
| Field | Value |
|---|---|
| resource_name | Api::V1::PanosController#index |
| service | cupixworks-api |
| avg_duration | 31149 ms (cluster aggregated) |
| max_duration | 93443 ms (cluster aggregated) |
| sample_trace_id | 4175447395282887089 |
| env | production / us-west-2 |
| tenant | cupix |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| cupixworks-api consumers (Pano viewer/Review) | 19 slow traces | /api/v1/panos, /api/v1/reviews/{key}/panos 응답 지연 — 사용자 뷰어 화면에서 회전 인디케이터 장기 노출, 일부 요청은 클라이언트/ELB timeout(>60s)에 가까움 |
Timeline#
- 2026-06-25 00:06 KST — 첫 slow trace 관측 (cluster
first_seen) - 2026-06-25 01:45 KST — status-board가
svc:cupixworks-api::unknown인시던트 자동 개설 (관련 cluster 7건 누적, incident2026-06-24-svc-cupixworks-api--unknown-2) - 2026-06-25 02:45–02:59 KST — Pano/Record/Capture ES
_update_document에서 "NotFound - attributes_in_database" warn 폭증 (수백 건/분), 동시간대 ES query duration 평균 0.05s → 0.37s - 2026-06-25 02:59 KST — status-board 인시던트 자동 resolve
- 2026-06-25 09:21 KST — 마지막 slow trace 관측 (cluster
last_seen); 이후 ES 평균 latency 점진 회복 (~0.1s)
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 13 code refs, 7 log queries |
| Code path explicitness | 3/3 | 13 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 22 file(s) |
| Single repo | 1/3 | 3 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |