Bulk operation failed
Service
cupixworks-worker
Occurrences
1,067
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
e00d4025-0002-4df9-8df9-8247d871680a
First Seen
2026-06-28 22:32 UTC
Last Seen
2026-07-11 03:34 UTC
Datadog
SlackActions
Bulk operation failed
What Happened#
2026-06-29 07:32 KST부터 약 45분 동안 cupixworks-worker (us-west-2)에서 BimRevision.bulk_operation이 29회 연속 실패했다. 모든 실패의 근본 원인은 동일하며, Elasticsearch bulk 인덱싱 도중 BimRevision의 as_indexed_json 직렬화 경로가 s3.me-south-1.amazonaws.com:443로 TCP 연결을 시도하다 timeout(execution expired)되었다. 동일 시간대에 별도 워커 Record.flush_geo_coordinate도 같은 me-south-1 endpoint에서 동일한 Seahorse::Client::NetworkingError로 실패하고 있어, BimRevision 코드 결함이 아니라 us-west-2 worker → me-south-1 S3 endpoint 구간의 네트워크 도달성 문제로 판단된다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | Seahorse::Client::NetworkingError |
| exception.message | Failed to open TCP connection to s3.me-south-1.amazonaws.com:443 (execution expired) |
| top_frame | app/models/concerns/searchable.rb:173 (rescued in bulk_operation) |
| originating_call | app/models/concerns/bim_revision_result.rb:48-57 (bim_revision_result_urls) |
| operation | update (Elasticsearch bulk update) |
| batch_size | 300 BimRevision IDs per call |
| deploy | production-us-west-2-20260628T2222Z0-bfdc5ebd-cupixworks |
| env | production, region us-west-2, host ip-10-1-18-149.us-west-2.compute.internal |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
BIM / Search indexing (BimRevision) |
29 (cluster) | me-south-1 tenant의 BimRevision Elasticsearch 인덱스 갱신이 지연/누락. 사용자가 검색·목록에서 최신 상태를 보지 못할 수 있음 |
Record geo-coordinate (Record.flush_geo_coordinate) |
동시 발생 (별도 cluster) | 같은 root cause로 worker 실패. capture/record 지오 좌표 동기화 지연 |
영향은 storage가 me-south-1에 있는 단일(또는 소수) tenant 범위로 추정되나, tenant ID는 로그에 포함되어 있지 않다 — uncertain, needs verification.
Timeline#
- 2026-06-29 07:32 KST — 첫
Bulk operation failed발생 (first_seen) - 2026-06-29 07:32–08:17 KST — 약 45분간 29건 연속 발생. 모두 동일 host
ip-10-1-18-149.us-west-2에서 me-south-1 endpoint로 TCP 연결 timeout - 2026-06-29 08:17 KST — 클러스터 마지막 발생 (
last_seen). 동일 시간대에 별도 cluster (Record.flush_geo_coordinate)도 같은 endpoint로 실패 중
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 22 code refs, 6 log queries |
| Code path explicitness | 3/3 | 22 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 16 file(s) |
| Single repo | 1/3 | 2 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |