ES /docs

Api::V1::FacilitiesController#update_spacetime (avg 12681ms, max 12681ms)

Service
cupixworks-api
Occurrences
1
Status
Dismissed
Confidence
85%
Actionability
28%
Cluster ID
047684be-acbc-4477-bd06-441785bd0777
First Seen
2026-07-09 00:59 UTC
Last Seen
2026-07-09 00:59 UTC
Datadog↗ Slack↗
Actions
현재 상태 dismissed 에서는 실행할 액션이 없습니다.

Api::V1::FacilitiesController#update_spacetime (avg 12681ms, max 12681ms)

What Happened#

2026-07-09 09:59 KST에 cupixworks-api 서비스에서 Api::V1::FacilitiesController#update_spacetime 요청 1건이 12681 ms 동안 실행되었다. 정상 요청은 대부분 서브초 수준에서 완료되지만, 같은 시간대에 Mysql2::Error::TimeoutError: Lock wait timeout exceeded 가 PanosController, JobsController, CapturesController 등 여러 엔드포인트에서 다발적으로 발생했으며, 이번 요청은 InnoDB row lock 대기로 응답이 지연된 것으로 판단된다.

Quick Facts#

Field Value
exception.class n/a (성공한 요청, HTTP 200)
resource_name Api::V1::FacilitiesController#update_spacetime
avg_duration_ms 12681
max_duration_ms 12681
sample_trace_id 2565454549763153544
top_frame app/repositories/facility_repository.rb:143 (spacetime.save!)
env production (region us-west-2, tenant cupix)

Affected Teams#

Team / Domain Error Count Impact
cupixworks-api (Capture Intelligence) 1 slow trace Spacetime summary/embedding write 지연, 최종 200 반환
cupixworks-api (Panos/Jobs/Captures) 15+ 502 Lock wait timeout 같은 창에서 여러 write API가 실패

Timeline#

  1. 2026-07-09 09:32 KST — POST /api/v1/panos 첫 Lock wait timeout 발생 (Datadog error 로그).
  2. 2026-07-09 09:45~09:56 KST — PUT /api/v1/panos/*/meta/blurriness, check_tile_uploading, captures/* 등에서 502 lock timeout 반복.
  3. 2026-07-09 09:59:43 KST — PUT /api/v1/facilities/zvizoq/spacetimes/1488237 요청 시작 (trace 2565454549763153544).
  4. 2026-07-09 09:59:56 KST — 같은 요청이 HTTP 200으로 완료. 총 12.68 s 소요.
  5. 2026-07-09 10:01~10:03 KST — JobsController#update, PanosController#check_uploading 등에서 502 lock timeout 4건 추가.
  6. 2026-07-09 10:04 KST 이후 — lock timeout 로그가 사라지고 트래픽 정상화.

Scoring#

Confidence 85% · Actionability 28%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 10 code refs, 8 log queries
Code path explicitness 3/3 10 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 13 file(s)
Single repo 1/3 7 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 3/3 no migration