ES /docs

Api::V1::CapturesController#update_meta_by_key (avg 51498ms, max 53938ms)

Service
cupixworks-api
Occurrences
54
Status
Reported
Confidence
85%
Actionability
16%
Cluster ID
4b0c3041-4961-4e2e-a75b-70756210770d
First Seen
2026-07-01 01:47 UTC
Last Seen
2026-08-04 19:44 UTC
Datadog↗ Slack↗
Actions

Api::V1::CapturesController#update_meta_by_key (avg 51498ms, max 53938ms)

What Happened#

2026-07-01 10:47 KST ~ 10:54 KST 사이에 cupixworks-api 의 Api::V1::CapturesController#update_meta_by_key 요청 4건이 평균 51.5초, 최대 53.9초로 지연되었다. 지연의 원인은 endpoint 자체가 아니라 MySQL InnoDB row lock 대기이며, 동일 시간대에 다수의 pano/capture/job endpoint 가 Mysql2::Error::TimeoutError: Lock wait timeout exceeded (기본 innodb_lock_wait_timeout=50s 와 정확히 일치) 로 502 를 반환하고 있다. 이 클러스터는 status-board incident 2026-07-01-svc-cupixworks-api--unknown-1 의 일부로, 최근 7일간 유사 사고가 8회 반복 발생했다.

Quick Facts#

Field Value
resource_name Api::V1::CapturesController#update_meta_by_key
cluster_type latency
avg_duration_ms 51498
max_duration_ms 53938
observed error Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting transaction
exception.class ActiveRecord::LockWaitTimeout
entry point app/controllers/concerns/metable_controller.rb:42
env production, us-west-2
tenant cupix

Affected Teams#

Team / Domain Error Count Impact
cupixworks-api (Captures) 4 slow spans on 724756/meta/prop, 3 of which returned 502 클라이언트가 meta/prop PUT 저장에 실패, 502 재시도 필요
cupixworks-api (Panos) 다수 502 (stitched, check_uploading, check_tile_uploading, check_mask_uploading) pano 업로드 상태 갱신 지연
cupixworks-api (Jobs) 최소 1개 job(1165311) postprocessor complete 콜백 5회 재시도 후 성공 (10:43 → 10:52) 3D reconstruction / postprocessor 파이프라인 지연

Timeline#

  1. 2026-07-01 10:43 KST — status-board incident 2026-07-01-svc-cupixworks-api--unknown-1 open (첫 클러스터 3f1c828f-...).
  2. 2026-07-01 10:43:27 KST — 마지막 정상 응답: [200] PUT /api/v1/jobs/1165311/actions/postprocessor/complete.
  3. 2026-07-01 10:47:17 KST — 첫 502: PUT /api/v1/jobs/1165311/actions/postprocessor/complete LockWaitTimeout.
  4. 2026-07-01 10:47:35 KST — 본 클러스터 최초 발생 (first_seen) — capture 724756 update_meta_by_key 51초 지연.
  5. 2026-07-01 10:50:58 ~ 10:55:00 KST — Lock wait timeout 폭발적으로 확산 (pano/capture/job 여러 endpoint).
  6. 2026-07-01 10:54:09 KST — 본 클러스터 마지막 발생 (last_seen).
  7. 2026-07-01 10:55:53 KST — capture 724756 meta/prop PUT 정상 성공 (200) — 락 경합 해소 시작.
  8. 2026-07-01 10:56:07 KST — capture 724756 publish PUT 200. 이후 reconstruction 큐잉 정상 진행.
  9. 2026-07-01 11:02:16 KST — status-board incident 마지막 이벤트 (8cbb9d23-...).

Scoring#

Confidence 85% · Actionability 16%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 4 code refs, 8 log queries
Code path explicitness 3/3 4 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 15 file(s)
Single repo 1/3 5 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 0/3 migration needed