ConnectionPool exhaustion during ES latency spike
Service
cupixworks-worker
Occurrences
239
Status
Reported
Confidence
80%
Actionability
28%
Cluster ID
5083c6a9-5c2a-438b-b7a6-d65c8f7bebd6
First Seen
2026-05-22 06:57 UTC
Last Seen
2026-07-18 09:18 UTC
Datadog
SlackActions
ConnectionPool exhaustion during ES latency spike
What Happened#
2026-06-17 07:44:42 KST에 production cupixworks-worker (us-west-2) 의 EditingSplitWorker#perform 가 Elasticsearch client connection pool 고갈로 실패했다. 같은 4초 구간에 ES _index_document 요청 8건이 10초 timeout 으로 동시 실패하면서 size=10 ES pool 의 모든 connection 이 점유됐고, split 작업이 ES client 를 7초간 대기한 뒤 ConnectionPool::TimeoutError("Waited 7 sec, 0/10 available") 로 raise 됐다. 동시 시점에 CreateSitetrackEditingEntitiesWorker 와 Feed#_index_document 도 동일 메시지로 실패했다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | ConnectionPool::TimeoutError (가능성 — 메시지 형식이 connection_pool gem 표준) |
| exception.message | Waited 7 sec, 0/10 available |
| top_frame | app/workers/editing_split_worker.rb:12 (catch site) |
| pool source | config/initializers/elasticsearch.rb:17 (ConnectionPool::Wrapper.new(size: 10, timeout: 7)) |
| env | production / us-west-2 |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
siteinsights / editing split pipeline (EditingSplitWorker) |
1 | editing split job fail → Sidekiq retry (retry: 2) 로 자동 재시도 |
siteinsights / sitetrack assignment (CreateSitetrackEditingEntitiesWorker) |
1 | sitetrack 19563 assignment 실패 (retry: 1) |
검색 indexing (Feed#_index_document) |
9+ | bulk/partial index doc 실패 — 검색 색인 일시 누락 |
Timeline#
- 2026-06-17 07:44:32 KST 부근 — Elasticsearch cluster 응답 지연 시작 (10s
request.timeout도달 시점에서 역산) - 2026-06-17 07:44:40 KST —
Feed#_index_document첫 ES pool wait 실패:Index error - Waited 7 sec, 0/10 available - 2026-06-17 07:44:42 KST —
EditingSplitWorker#perform실패 (Editing split failed: Waited 7 sec, 0/10 available) +CreateSitetrackEditingEntitiesWorker동시 실패 (sitetrack_id 19563) - 2026-06-17 07:44:42–07:44:44 KST — ES 측 timeout 8건 (
Operation timed out after 10002 ms) 폭주 - 2026-06-17 07:44:44 KST 이후 — 더 이상 동일 에러 없음 (incident self-cleared, ES 회복)
- 2026-06-17 08:16:57 KST 이후 —
EditingSplitWorker정상 동작 (수십 건Starting editing split/Editing split finished로그 확인)
Scoring#
Confidence 80% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 12 code refs, 8 log queries |
| Code path explicitness | 3/3 | 12 file:line refs |
| Log evidence | 2/3 | section, query |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 12 file(s) |
| Single repo | 1/3 | 3 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |