ES /docs

Errno::ENOSPC: No space left on device @ dir_s_mkdir - /var/data_changes/meta_change/element_traces/partial/2025-08-19/u

Service
cupixworks-api
Occurrences
2,107
Status
Reported
Confidence
85%
Actionability
44%
Cluster ID
38d487ae-4835-44d7-a62a-fae900ef0f4a
First Seen
2025-08-19 15:48 UTC
Last Seen
2026-08-02 01:48 UTC
Datadog↗ Slack↗
Actions

Errno::ENOSPC: No space left on device @ dir_s_mkdir - /var/data_changes/meta_change/element_traces/partial/2025-08-19/u

What Happened#

tesla 의 Data Warehouse CDC (change-data-capture) 경로가 변경 데이터를 gzip JSON 으로 /var/data_changes/<model>/partial/<date>/<region>/ 아래에 저장한다. 이 볼륨(worker 인스턴스의 전용 EBS)이 가득 차면 partition 디렉터리를 만드는 FileUtils.mkdir_p 가 Errno::ENOSPC (No space left on device) 로 실패한다. Representative Error 는 dir_s_mkdir — 즉 write 가 아니라 디렉터리 생성 단계의 ENOSPC 다. 이 정확한 증상(mkdir ENOSPC)은 이미 Airbrake #7565 로 근본 분석되어 TSLA-12956 (PR 88061) 에서 graceful degrade 로 수정되었으며, 현재 develop 에는 병합되어 있으나 master(production) 에는 아직 배포되지 않았다.

Quick Facts#

Field Value
exception.class Errno::ENOSPC
exception.message No space left on device @ dir_s_mkdir - /var/data_changes/meta_change/element_traces/partial/2025-08-19/uswe2/1755618510791938-meta_change
top_frame app/models/concerns/data_ware_house/partial_json.rb:25 (FileUtils.mkdir_p) / app/models/concerns/data_ware_house/base.rb:28
runtime Ruby on Rails (tesla), Sidekiq worker (SavePartialJsonToFileWorker)
deploy fix d4b44f30f (TSLA-12956) merged to develop, NOT in master
env production, region uswe2 (us-west-2) — worker instance role

Affected Teams#

Team / Domain Error Count Impact
cupixworks-api / cupixworks-worker (Data Warehouse CDC) 2107 (Error Tracking 누적) CDC partial JSON 일부 유실 가능 — datalake 동기화에 해당 변경분 누락. 사용자 대면 API 응답에는 영향 없음 (after_commit 비동기 경로).

Timeline#

  1. 2025-08-20 00:48 KST — 최초 발생 (first_seen 2025-08-19T15:48:29Z UTC). partition 2025-08-19/uswe2 의 meta_change mkdir 이 ENOSPC.
  2. 2026-05-23 (KST) — Airbrake #7565: /var/data_changes/elements/partial/2026-05-23 mkdir ENOSPC 광역 발생, 별도 조사 트리거.
  3. 2026-06-10 10:10 KST — TSLA-12956 (PR 88061) develop 병합 (d4b44f30f): mkdir ENOSPC/EDQUOT 를 raise 대신 로그 + metric 으로 degrade.
  4. 2026-08-02 10:48 KST — Error Tracking last_seen (2026-08-02T01:48:51Z UTC). 단, 해당 시각 전후 Datadog 로그에는 매칭 결과 0건 (아래 Log Evidence).
  5. 2026-08-04 (오늘) — RCA. develop 에 fix 존재하나 master 미배포 상태 확인.

Scoring#

Confidence 85% · Actionability 44%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 10 code refs, 8 log queries
Code path explicitness 3/3 10 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 5/5 trivial
Affected files 1/5 13 file(s)
Single repo 1/3 5 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 3/3 no migration