ES /docs

Bulk index partial failure

Service

cupixworks-api

Occurrences

Status

Dismissed

Confidence

85%

Actionability

28%

Cluster ID

6e469f55-46f5-4c9f-bf80-0620d9c6d823

First Seen

2026-06-10 04:37 UTC

Last Seen

2026-06-10 04:37 UTC

Datadog↗

Slack↗

Actions

현재 상태 dismissed 에서는 실행할 액션이 없습니다.

Bulk index partial failure

What Happened#

2026-06-10 13:37 KST, cupixworks-api (production, ap-southeast-1)에서 Capture#bulk_operation!이 Elasticsearch bulk update를 호출했지만 부분 실패가 발생했다. 실패 원인은 ES [parent] Data too large (circuit_breaking_exception, HTTP 429) — 클러스터 heap 사용량이 7.7GB로 parent breaker limit 7.5GB(8160437862 bytes)를 초과한 상태였다. 동일 시간대(13:37:10–13:37:56)에 같은 ES 클러스터를 사용하는 다수의 모델(Capture, Pano, EditingEntity, Admin::EditingRepository)에서 429 circuit-breaking 에러가 동시 다발했고, 일부 요청은 결국 Faraday timeout(10s)으로 500을 반환했다.

Quick Facts#

Field	Value
exception.class	`Elasticsearch::Transport::Transport::Errors::TooManyRequests` (logged via partial-failure handler)
exception.message	`Bulk index partial failure` (1 of 1 items failed with `circuit_breaking_exception`)
top_frame	`app/models/concerns/searchable.rb:232`
runtime	Rails / `elasticsearch-ruby` client
deploy	`production-ap-southeast-1-20260609T0227Z0-4aed8e74-cupixworks`
env	production / ap-southeast-1

Affected Teams#

Team / Domain	Error Count	Impact
Search / Indexing (Capture)	1 (this cluster)	Capture 2514 ES document update missed — index drift
Search / Indexing (Pano, EditingEntity, Admin::EditingRepository)	19+ in the same minute	동일 ES 클러스터 heap 압박으로 다수 indexing 실패
API consumers of `PUT /api/v1/jobs`, `POST /api/v1/panos`	다수	500 응답 (SYS50000) — job 진행/pano 생성 실패

Timeline#

2026-06-10 13:37:10 KST — ES parent breaker가 7.6–7.9GB 영역에서 반복적으로 trip (Pano#_update_document 등 다수 모델에서 429 시작)
2026-06-10 13:37:40 KST — Capture 2514 가 processing → done 전이. Capture#bulk_operation! 호출, ES bulk update가 partial failure (1/1 fail, status 429) 반환 → Bulk index partial failure 로깅 (이 클러스터)
2026-06-10 13:37:42 KST — 동일 request_id 컨텍스트에서 후속 Capture#_update_document 가 429 반환 (bulk 와 달리 raise 됨)
2026-06-10 13:37:52 KST — EditingEntity#_index_document 가 Faraday timeout(10002ms) → PUT /api/v1/jobs/9287 가 500 응답
2026-06-10 13:37:56 KST — request_id 8c00491c 트레이스 종료

Scoring#

Confidence 85% · Actionability 28%

Confidence#

Dimension	Score	Detail
Evidence completeness	5/5	9 code refs, 6 log queries
Code path explicitness	3/3	9 file:line refs
Log evidence	3/3	section, query, pattern
RCA structure	3/3	3/3 sections
Revision history	0/3	0 revision(s)
Agent execution	3/3	success

Actionability#

Dimension	Score	Detail
Fix complexity	1/5	critical
Affected files	1/5	15 file(s)
Single repo	1/3	3 service(s)
Immediate actions	0/3	absent
Testability	0/3	no test info
External dependency	1/3	external deps found
Data migration	3/3	no migration