ES /docs

SAM3 set_image() — uncorrectable GPU ECC memory error

Service
cupixworks-pano-postprocessor-instance
Occurrences
2
Status
Reported
Confidence
80%
Actionability
52%
Cluster ID
10dd9935-1b98-41fb-95b9-2f5a19cc5c98
First Seen
2026-06-04 16:08 UTC
Last Seen
2026-06-04 16:08 UTC
Datadog↗ Slack↗
Actions

SAM3 set_image() — uncorrectable GPU ECC memory error

What Happened#

2026-06-05 01:08 KST에 cupixworks-pano-postprocessor-instance 서비스에서 SAM3 face/body detection inference 수행 중 GPU 하드웨어 메모리 오류(uncorrectable ECC error)가 발생하여 해당 인스턴스의 pano 처리 작업이 실패했다. 단일 발생이며 다른 인스턴스는 정상 동작을 지속했다.

Quick Facts#

Field Value
exception.class RuntimeError
exception.message CUDA error: uncorrectable ECC error encountered
top_frame /tmp/lib/face-body-detector/sam3/model/sam3_image_processor.py:55
runtime Python 3.11, PyTorch 2.6.0, CUDA 12.8.1
env production, us-west-2

Timeline#

  1. 2026-06-05 01:06:24 KST — PanoPostprocessorService 인스턴스 초기화 (session: 893ea23004d7)
  2. 2026-06-05 01:06:28 KST — pano 다운로드 완료 (1895ms)
  3. 2026-06-05 01:07:38 KST — resize 작업에서 504 Gateway Time-out 다수 발생 (GPU 부하 징후)
  4. 2026-06-05 01:08:15 KST — infer_panos 단계 완료 기록 (elapsed: 160857ms)
  5. 2026-06-05 01:08:16 KST — CUDA uncorrectable ECC error 발생, MaskWork 프로세스 exit code 1
  6. 2026-06-05 01:08:18 KST — PanoPostprocessorService::run 실패 (end - false)
  7. 2026-06-05 01:08:19 KST — terminateService 강제 종료
  8. 2026-06-05 01:12:14 KST — 새 인스턴스 초기화, CUDA 정상 동작 재개

Scoring#

Confidence 80% · Actionability 52%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 7 code refs, 5 log queries
Code path explicitness 3/3 7 file:line refs
Log evidence 2/3 section, query
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 5/5 trivial
Affected files 1/5 18 file(s)
Single repo 1/3 5 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 3/3 no external deps
Data migration 3/3 no migration