Api::V1::LevelsController#create (avg 141670ms, max 323409ms)

Service

cupixworks-api

Occurrences

Status

Reported

Confidence

85%

Actionability

44%

Cluster ID

12314a86-605b-468d-af66-42103857238d

First Seen

2026-08-03 05:59 UTC

Last Seen

2026-08-03 16:35 UTC

Datadog↗

Slack↗

Actions

Api::V1::LevelsController#create (avg 141670ms, max 323409ms)

What Happened#

2026-08-03 15:01–15:02 KST 사이 cupixworks-api (ap-southeast-2, env:production, tenant cupix) 에서 POST /api/v1/levels 요청 3건이 평균 141초, 최대 323초까지 지연되었다. 요청은 Mysql2::Error::TimeoutError: Lock wait timeout exceeded 로 500 응답을 냈다. 같은 시간대(14:16–15:16 KST)에 panos/jobs/captures/editings/assets 등 다른 엔드포인트에서도 동일한 Lock wait timeout 이 15건 이상 발생했다. 즉 이 latency 클러스터는 서비스 전반의 row-lock 경합 이벤트의 일부다.

Quick Facts#

Field	Value
exception.class	`Cupix::Errors::System` (wrapping `ActiveRecord::LockWaitTimeout` / `Mysql2::Error::TimeoutError`)
exception.message	`Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting transaction`
top_frame	`app/factories/base_factory.rb:124` (`self.model.save!` in `BaseFactory#create!`)
resource_name	`Api::V1::LevelsController#create`
avg_duration_ms	141670
max_duration_ms	323409
deploy	`version:production-ap-southeast-2-20260731t0617z0-5d9c7658-cupixworks`
env	`production`, `region:ap-southeast-2`
tenant	`cupix`

Affected Teams#

Team / Domain	Error Count	Impact
`endeavourgroup` (team.id 180)	2건 확인	동일 `facility_key=8je0s5` 로 `Coldroom` Level 생성 실패 (500) — 사용자 CupixConnect 데스크톱 클라이언트에서 50초 이상 대기 후 실패
기타 (`Api::V1::JobsController#update`, `PanosController#update`, `CapturesController#update`, `EditingsController#index`, `AssetsController#update`)	동시간대 15건 이상 (502/400/500)	Level 만이 아닌 서비스 전반의 lock 경합 — 동일 원인의 다른 증상

Timeline#

2026-08-03 14:16 KST — PUT /api/v1/panos/15354652/meta/ver 502 (ActiveRecord::LockWaitTimeout). Lock 경합 첫 감지.
2026-08-03 14:22 KST — POST /api/v1/captures/85824/process_output_upload_url 502.
2026-08-03 14:39 KST — PUT /api/v1/assets/kh6cuxohjn4p 502.
2026-08-03 14:51 KST — GET /api/v1/editings 502.
2026-08-03 14:58:50 KST (05:58:50Z) — 클러스터 first_seen 근처, 첫 slow Level create 시작.
2026-08-03 15:01:39 KST (06:01:39Z) — POST /api/v1/levels 500 (500ms request duration 이지만 db=50917ms — 첫 Lock wait timeout 500 응답). team=endeavourgroup, facility_key=8je0s5.
2026-08-03 15:02:31 KST (06:02:31Z) — 동일 팀, 두 번째 POST /api/v1/levels 500 (duration=50938ms, db=50917ms).
2026-08-03 15:04–15:16 KST — Jobs / Panos / Captures endpoint 계속 502.
2026-08-03 15:16:19 KST — 마지막 Lock wait timeout (PUT /api/v1/panos/15364151) 이후 로그에서 소거.

Scoring#

Confidence 85% · Actionability 44%

Confidence#

Dimension	Score	Detail
Evidence completeness	5/5	14 code refs, 4 log queries
Code path explicitness	3/3	14 file:line refs
Log evidence	3/3	section, query, pattern
RCA structure	3/3	3/3 sections
Revision history	0/3	0 revision(s)
Agent execution	3/3	success

Actionability#

Dimension	Score	Detail
Fix complexity	5/5	trivial
Affected files	1/5	18 file(s)
Single repo	1/3	4 service(s)
Immediate actions	0/3	absent
Testability	0/3	no test info
External dependency	1/3	external deps found
Data migration	3/3	no migration