StandardError - Waited 7 sec, 0/10 available

Service

cupixworks-worker

Occurrences

118

Status

Reported

Confidence

85%

Actionability

28%

Cluster ID

77863ae7-0f4a-4169-92e1-10a70a36abf7

First Seen

2026-07-01 02:47 UTC

Last Seen

2026-07-10 05:53 UTC

Datadog↗

Slack↗

Actions

StandardError - Waited 7 sec, 0/10 available

What Happened#

2026-07-01 11:47 KST부터 production cupixworks-worker (Sidekiq) 인스턴스에서 Elasticsearch 클라이언트의 ConnectionPool이 고갈되어 Waited 7 sec, 0/10 available 에러가 반복적으로 발생. Feed._index_document, EditingSplitWorker.perform, CreateSitetrackEditingEntitiesWorker.perform 등 다양한 클래스에서 ES 인덱싱이 실패했고 약 3시간 동안 49회 이상 관측됨. 원인은 신규 배포(c527a441)에 포함된 EditingSplitService의 재-split 재귀 스케줄링이 특정 editing들에서 종료 조건을 만족하지 못하고 5초 주기로 EditingSplitWorker를 계속 재-enqueue 하여 ES 커넥션을 소진시킨 것.

Quick Facts#

Field	Value
exception.class	`ConnectionPool::TimeoutError` (rescue `StandardError`로 잡혀 `.message`만 로그로 남음)
exception.message	`Waited 7 sec, 0/10 available`
top_frame	`app/models/concerns/searchable.rb:51` (`Searchable#_index_document` rescue)
runtime	Ruby / Sidekiq worker, host `ip-10-1-18-149.us-west-2.compute.internal`, pid 1511179
deploy	`production-us-west-2-20260630T0615Z0-c527a441-cupixworks` (merge `c527a441`, 2026-06-30 15:15 KST)
env	production, `us-west-2`, tenant `cupix`

Affected Teams#

Team / Domain	Error Count	Impact
SQA / siteinsights editing split pipeline	49 (본 클러스터) + 2 (sibling 클러스터 `e15e7ba7`, sitetrack_id 20474)	Editing split이 무한 재-enqueue 되어 sitetrack 처리 지연, 로그 노이즈
ES indexing (전 서비스 공용 ES client)	다수 (Feed, EditingSplitWorker, CreateSitetrackEditingEntitiesWorker)	ES 인덱싱 실패 → `BulkIndexWorker` fallback 되지만 pool 소진이 지속되면 fallback 자체도 실패 위험

Timeline#

2026-06-30 15:15 KST — c527a441 배포. EditingSplitService#split! 끝에 "post-split cap" 재확인 로직 및 EditingSplitWorker.perform_in(5.seconds, editing.id) 재-enqueue 추가.
2026-07-01 11:47 KST — 클러스터 77863ae7 최초 발생 (Feed._index_document "Index error - Waited 7 sec, 0/10 available").
2026-07-01 13:11 KST — sibling 클러스터 e15e7ba7 최초 발생 (CreateSitetrackEditingEntitiesWorker on sitetrack_id: 20474 — 하위 EditingSplitWorker 폭주에 휘말림).
2026-07-01 14:27 KST — 본 클러스터 최근 발생 (누적 49회).
2026-07-01 14:29~14:47 KST — Post-split cap still exceeded, scheduling re-split warn 로그가 3개 editing (1178504, 1177289, 1186375)에서 각각 post_split_count 6540/5880/3630으로 5~6초마다 반복 관측 — 재-split 루프 실증.

Scoring#

Confidence 85% · Actionability 28%

Confidence#

Dimension	Score	Detail
Evidence completeness	5/5	15 code refs, 6 log queries
Code path explicitness	3/3	15 file:line refs
Log evidence	3/3	section, query, pattern
RCA structure	3/3	3/3 sections
Revision history	0/3	0 revision(s)
Agent execution	3/3	success

Actionability#

Dimension	Score	Detail
Fix complexity	1/5	critical
Affected files	1/5	20 file(s)
Single repo	1/3	5 service(s)
Immediate actions	0/3	absent
Testability	0/3	no test info
External dependency	1/3	external deps found
Data migration	3/3	no migration