ES /docs

BaseService::handlingMessageErrors | Error and message object - {"error":{"response":{"statusCode":5

Service
cupixworks-capture-preprocessor-agent
Occurrences
2
Status
Reported
Confidence
80%
Actionability
36%
Cluster ID
0af9cd4b-01ff-4905-ab25-2e8d92d74680
First Seen
2026-07-10 17:29 UTC
Last Seen
2026-07-10 18:00 UTC
Datadog↗ Slack↗
Actions

BaseService::handlingMessageErrors | Error and message object - {"error":{"response":{"statusCode":5

What Happened#

cupixworks-capture-preprocessor-agent 가 SQS 메시지를 처리하는 중 api-tesla.cupix.internal 로 PUT /api/v1/captures/732626/meta/mark 요청을 보냈고, 2026-07-11 02:29 KST 에 nginx 로부터 HTTP 504 Gateway Time-out 을 받아 BaseService::handlingMessageErrors 에서 error 로그가 남았다. 같은 capture(732626) 에 대한 API 요청은 그 직전 약 5분간 매 요청마다 Mysql2::Error::TimeoutError: Lock wait timeout exceeded 로 502 를 반환해 왔고, 504 는 nginx 가 upstream(Rails) 응답 지연을 끊으면서 발생한 tail 이벤트로 확인된다.

Quick Facts#

Field Value
exception.class HttpError (agent side) — 원인은 ActiveRecord::LockWaitTimeout (Rails side)
exception.message 504 Gateway Time-out / Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting transaction
top_frame (agent) packages/base/src/base-service.ts:311 (BaseService::handlingMessageErrors)
top_frame (api) app/controllers/concerns/metable_controller.rb:42-69 (MetableController#update_meta_by_key)
deploy 확인되지 않음 — 로그에 배포 SHA 없음
env production, region us-west-2, tenant cupix

Affected Teams#

Team / Domain Error Count Impact
clark-vdc (capture-preprocessor-agent) 1 SQS 메시지 처리 실패, 재시도 필요
fortis (동일 서비스, TransferManager) 2 같은 서비스에서 504 로 download 실패 — 사이드-이펙트로 추정 (관련 클러스터 cc6978c4-d452-4a79-add1-93bc84e8e0e4)
cupixworks-api (tesla) 30+ (전체 endpoint) 17:24–17:32 UTC 사이 다수 endpoint 에서 LockWaitTimeout — 시스템 전역의 MySQL row-lock 경합

Timeline#

  1. 2026-07-11 02:10 KST — TransferManager::downloadFile 에서 504 발생 (sibling cluster cc6978c4, first_seen 2026-07-10T17:10:18Z)
  2. 2026-07-11 02:11 KST — status-board 가 서비스 열화 인시던트 오픈 (2026-07-10-svc-cupixworks-capture-preprocessor-agent--unknown-1)
  3. 2026-07-11 02:23 KST — capture 732626 에서 첫 Mysql2::Error::TimeoutError 발생 (PUT /api/v1/captures/732626, 2026-07-10T17:23:11Z)
  4. 2026-07-11 02:24 KST — PUT /api/v1/captures/732626/meta/mark 첫 502 (17:24:08Z)
  5. 2026-07-11 02:25–02:28 KST — 동일 endpoint 4회 연속 502, 매번 Lock wait timeout exceeded
  6. 2026-07-11 02:29 KST — 6번째 재시도에서 nginx 가 upstream timeout 으로 504 반환. Agent 가 BaseService::handlingMessageErrors 로그 남김 (first_seen/last_seen = 2026-07-10T17:29:16.651Z)
  7. 2026-07-11 02:30–02:32 KST — 다른 endpoint (/api/v1/panos/*/check_uploading, /api/v1/jobs/*/actions/postprocessor/complete) 도 LockWaitTimeout 폭증 — 시스템 전역 이벤트

Scoring#

Confidence 80% · Actionability 36%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 9 code refs, 9 log queries
Code path explicitness 3/3 9 file:line refs
Log evidence 2/3 section, query
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 16 file(s)
Single repo 1/3 10 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 3/3 no external deps
Data migration 3/3 no migration