ES /docs

Failed to save Partial JSON Capture 701882 to file /var/data_changes/captures/partial/2026-05-22/usw

Service
cupixworks-worker
Occurrences
815,168
Status
Reported
Confidence
80%
Actionability
44%
Cluster ID
169aaa8e-cc93-4de5-9988-9fcb4f8370be
First Seen
2026-04-06 09:17 UTC
Last Seen
2026-08-05 03:00 UTC
Datadog↗ Slack↗
Actions

Failed to save Partial JSON Capture 701882 to file /var/data_changes/captures/partial/2026-05-22/usw

What Happened#

2026-05-22 20:20 UTC부터 2026-05-23 00:38 UTC까지 약 4시간 동안, cupixworks-worker 서비스의 SavePartialJsonToFileWorker가 /var/data_changes/ 볼륨의 디스크 용량 부족(Errno::ENOSPC)으로 인해 partial JSON 파일 저장에 실패했다. Capture, ElementTrace, Pointcloud, Element 등 다수의 엔티티에서 총 8,272건의 에러가 발생했다.

Quick Facts#

Field Value
exception.class Errno::ENOSPC
exception.message No space left on device @ rb_sysopen - /var/data_changes/captures/partial/2026-05-22/uswe2/...json.gz
top_frame app/models/concerns/data_ware_house/partial_json.rb:144
runtime Ruby 3.3.0, Sidekiq 7.3.9
env production, us-west-2

Affected Teams#

Team / Domain Error Count Impact
Data Warehouse (data_changes pipeline) 8,272 Partial JSON export 실패 — downstream 데이터 동기화 지연

Timeline#

  1. 2026-05-22T20:20:48Z — 최초 에러 발생. rb_sysopen syscall에서 ENOSPC 에러 시작
  2. 2026-05-22T20:20~00:36 — 4시간 동안 지속적 실패. Sidekiq가 새 job을 계속 받으며 에러 누적
  3. 2026-05-23T00:36+ — 날짜 변경으로 새 디렉토리(/2026-05-23/) 생성 시도 시 dir_s_mkdir에서도 ENOSPC 발생
  4. 2026-05-23T00:38:47Z — 관측 윈도우 내 마지막 에러

Scoring#

Confidence 80% · Actionability 44%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 17 code refs, 2 log queries
Code path explicitness 3/3 17 file:line refs
Log evidence 2/3 section, query
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 5/5 trivial
Affected files 1/5 18 file(s)
Single repo 1/3 2 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 3/3 no migration