ES /docs

Backfill facility defaults: failed

Service
cupixworks-worker
Occurrences
2
Status
Reported
Confidence
85%
Actionability
44%
Cluster ID
e0c42c33-d3ab-4b73-85fe-07bbc7f6bb86
First Seen
2026-07-29 04:18 UTC
Last Seen
2026-07-29 04:18 UTC
Datadog↗ Slack↗
Actions

Backfill facility defaults: failed

What Happened#

2026-07-29 13:18 KST, prod us-west-2 에서 Tsla13613Backfill 라는 일회성 healing 스크립트가 실행되었다. 이 스크립트는 TSLA-13594/TSLA-13664 여파로 default_level 이 누락된 facility 를 스캔하여 facility.create_default_models! 로 자가치유(heal)를 시도한다. 405개 facility 중 55개가 healed, 215개는 이미 정상이라 skipped, 그리고 2개가 Cupix::Errors::NotFound (ARG10002, "Facility not found") 로 실패했다. 실패는 LevelFactory#create! 내부의 permission-scoped FacilityRepository.show(facility.key) 재조회에서 발생했다.

Quick Facts#

Field Value
exception.class Cupix::Errors::NotFound
exception.message {:code=>"ARG10002", :reason=>"Facility not found"}
top_frame app/repositories/base_repository.rb:355
runtime Sidekiq / Rails runner (cupixworks-worker), pid 643917
deploy production-us-west-2-20260729T0417Z0-812cb7d9-cupixworks (SHA 812cb7d9)
env production, us-west-2
script Tsla13613Backfill (backfill_tag tsla13613_facility_defaults, ad-hoc, not in tracked repo)

Affected Teams#

Team / Domain Error Count Impact
solarturbines-tst (test team) 2 Test facility 2개가 default_level 을 얻지 못하고 healing 대상에서 제외됨. 실제 고객 워크플로에는 영향 없음 (facility_size=0, name 이 sean-api-test-fac-..., api-test-no-seed-27min 인 test fixture).

Timeline#

  1. 2026-07-29 13:18:00 KST — Tsla13613Backfill 1차 실행 시작 (dry_run: false, small batch: 3 scanned, 1 healed, 2 skipped, 0 errors).
  2. 2026-07-29 13:18:01 KST — Tsla13613Backfill 2차 실행 시작 (dry_run: false, main batch).
  3. 2026-07-29 13:18:15 KST — facility 21919 (uemezf) 에 대해 Cupix::Errors::NotFound "Facility not found" 발생, Backfill facility defaults: failed 로그 기록.
  4. 2026-07-29 13:18:31 KST — facility 21947 (14gejl) 에 대해 동일 예외 발생.
  5. 2026-07-29 13:18:41 KST — 스크립트 종료 (Backfill facility defaults done, scanned=405, healed=55, skipped=215, errors=2).

Scoring#

Confidence 85% · Actionability 44%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 14 code refs, 5 log queries
Code path explicitness 3/3 14 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 20 file(s)
Single repo 1/3 2 service(s)
Immediate actions 0/3 absent
Testability 2/3 test mentioned
External dependency 3/3 no external deps
Data migration 3/3 no migration