ES /docs

BulkableRepository#bulk! sequential Elasticsearch indexing — resource exhaustion

Service

cupixworks-api

Occurrences

15,958

Status

Reported

Confidence

85%

Actionability

28%

Cluster ID

29a96c66-635c-45cf-9c04-2e29550f8cff

First Seen

2026-05-26 03:23 UTC

Last Seen

2026-08-05 04:28 UTC

Datadog↗

Slack↗

Actions

BulkableRepository#bulk! sequential Elasticsearch indexing — resource exhaustion

What Happened#

2026-05-26 03:24~06:16 UTC 동안 cupixworks-api의 Api::V1::PanosController#bulk 엔드포인트에서 평균 6750ms, 최대 약 16초의 응답 지연이 4개 리전(ap-southeast-2, us-west-2, eu-central-1, ap-southeast-1)에서 148건 발생했다. 모든 요청은 HTTP 200으로 성공했으나, bulk update 시 아이템당 ~100ms의 순차 처리로 인해 100개 아이템 배치에서 ~10초가 소요되는 구조적 성능 문제이다.

Quick Facts#

Field	Value
resource_name	`Api::V1::PanosController#bulk`
top_frame	`app/repositories/concerns/bulkable_repository.rb:30-39`
env	production (ap-southeast-2, us-west-2, eu-central-1, ap-southeast-1)

Affected Teams#

Team / Domain	Error Count	Impact
samsungenatest	67	Bulk pano update 응답 대기 ~10초
crcc-sama	56	Bulk pano update 응답 대기 ~10초
naylorlove	19	Bulk pano update 응답 대기 ~10초

Timeline#

2026-05-26T03:24:06Z — 최초 slow trace 감지 (ap-southeast-2)
2026-05-26T05:19:04Z — 최대 지연 관측 (18.8초, ap-southeast-2)
2026-05-26T06:16:10Z — 마지막 slow trace 기록
2026-05-26 — Error-sweeper 클러스터 생성 및 RCA 수행

Scoring#

Confidence 85% · Actionability 28%

Confidence#

Dimension	Score	Detail
Evidence completeness	5/5	9 code refs, 2 log queries
Code path explicitness	3/3	9 file:line refs
Log evidence	3/3	section, query, pattern
RCA structure	3/3	3/3 sections
Revision history	0/3	0 revision(s)
Agent execution	3/3	success

Actionability#

Dimension	Score	Detail
Fix complexity	1/5	critical
Affected files	1/5	11 file(s)
Single repo	1/3	2 service(s)
Immediate actions	0/3	absent
Testability	0/3	no test info
External dependency	1/3	external deps found
Data migration	3/3	no migration