Failed to save partial JSON ElementTrace 102272260
Service
cupixworks-worker
Occurrences
720,659
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
64ddc12c-2cfb-4c68-8207-762c46162ed9
First Seen
2026-05-22 20:20 UTC
Last Seen
2026-08-01 16:41 UTC
Datadog
SlackActions
Failed to save partial JSON ElementTrace 102272260
What Happened#
2026-05-22 20:20 UTC부터 약 4시간 동안 us-west-2 리전의 cupixworks-worker 호스트 ip-10-1-18-233에서 디스크 공간 부족(No space left on device)으로 인해 partial JSON 파일 저장이 전면 실패했다. ElementTrace를 포함한 4개 모델(ElementTrace, Pointcloud, Capture, Editing)에서 총 719,410건의 에러가 발생했으며, 데이터 웨어하우스로의 변경 데이터 전파가 완전히 중단되었다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | Errno::ENOSPC |
| exception.message | No space left on device @ rb_sysopen - /var/data_changes/element_traces/partial/2026-05-22/uswe2/{timestamp}-element_traces-{id}.json.gz |
| top_frame | app/models/concerns/data_ware_house/partial_json.rb:144 |
| runtime | Ruby 3.3.0, Rails 7.2.2, Sidekiq 7.3.9 |
| deploy | production-us-west-2-20260519T0920Z0-3e770a15-cupixworks |
| env | production, us-west-2 |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| Data Warehouse (ElementTrace) | ~719,000 | 변경 데이터 파일 미생성 → downstream 동기화 누락 |
| Data Warehouse (Pointcloud) | 수백 건 | Pointcloud 변경 이력 누락 |
| Data Warehouse (Capture/Editing) | 수십 건 | Capture/Editing 변경 이력 누락 |
Timeline#
- 2026-05-22 20:20:42 UTC — 최초 에러 발생 (ElementTrace 102272260, host ip-10-1-18-233)
- 2026-05-22 20:59:52 UTC — Pointcloud 모델에서도 동일 에러 확인
- 2026-05-22 23:57:59 UTC — 클러스터 내 마지막 에러 기록
- 2026-05-23 00:29:51 UTC — 날짜 변경 후 디렉토리 생성조차 실패 (
dir_s_mkdir에러) - 2026-05-26 — Error Sweeper에 의한 RCA 수행
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 13 code refs, 3 log queries |
| Code path explicitness | 3/3 | 13 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 20 file(s) |
| Single repo | 1/3 | 2 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |