Failed to save partial JSON Element 15382518
Service
cupixworks-worker
Occurrences
265,393
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
e0847088-3fe1-447f-8f9b-4807d4d8e9bb
First Seen
2026-05-22 20:20 UTC
Last Seen
2026-08-01 16:41 UTC
Datadog
SlackActions
Failed to save partial JSON Element 15382518
What Happened#
2026-05-22 20:20 UTC부터 23:57 UTC까지 약 3.5시간 동안 cupixworks-worker 서비스의 data_changes Sidekiq queue에서 partial JSON 파일 저장이 대량 실패했다. 단일 호스트(ip-10-1-18-233.us-west-2.compute.internal)의 /var/data_changes/ 파티션 디스크 용량 부족(No space left on device)이 원인이며, 264,144건의 에러가 발생했다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | Errno::ENOSPC |
| exception.message | No space left on device @ rb_sysopen - /var/data_changes/elements/partial/2026-05-22/uswe2/1779481236606406-elements-15382518.json.gz |
| top_frame | app/models/concerns/data_ware_house/partial_json.rb:144 |
| runtime | Ruby 3.3.0, Sidekiq 7.3.9, Rails 7.2.2 |
| deploy | production-us-west-2-20260519T0920Z0-3e770a15-cupixworks |
| env | production, us-west-2 |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| Data Warehouse (partial JSON pipeline) | 264,144 | 모든 모델(Element, ElementTrace, Record, Capture)의 partial JSON export 실패 — 데이터 웨어하우스 동기화 지연 |
| Elasticsearch indexing | ~수십건 | BulkPartialIndexDeadWorker 후속 실패 (document_missing_exception) |
Timeline#
- 2026-05-22T20:20:42Z — 최초 에러 발생.
Element 15382518partial JSON 저장 실패 (Errno::ENOSPC) - 2026-05-22T20:20~23:57Z — 동일 호스트에서 모든 partial JSON 저장 작업 연쇄 실패 (264,144건)
- 2026-05-22T23:41~23:42Z — 후속 영향:
BulkPartialIndexDeadWorker에서 Elasticsearchdocument_missing_exception발생 - 2026-05-22T23:57:59Z — 마지막 에러 로그 기록
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 9 code refs, 4 log queries |
| Code path explicitness | 3/3 | 9 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 20 file(s) |
| Single repo | 1/3 | 2 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |