failed to get captured area - error: 500 Internal Server Error
Service
cupixworks-migration-worker
Occurrences
275
Status
Reported
Confidence
80%
Actionability
16%
Cluster ID
af5f0fa7-6dda-4890-abb1-2ad1ce01d686
First Seen
2026-07-15 12:33 UTC
Last Seen
2026-07-30 08:41 UTC
Datadog
SlackActions
failed to get captured area - error: 500 Internal Server Error
What Happened#
2026-07-15 21:33 KST부터 21:58 KST 사이 cupixworks-migration-worker 서비스에서 Cupix::VoxelService.captured_area! 호출이 총 10건 실패했다. 원인은 downstream voxel Lambda(Voxel::CapturedArea)가 AWS Athena GetQueryExecution API 호출 시 ThrottlingException: Rate exceeded(max retries 3 소진)를 반환하며 500을 낸 것이다. 트리거는 21:31 KST에 실행된 cron Cupix::Cron::Facility.flush_stale_captured_size 로, stale facility 30개에 대해 연속으로 voxel-service 를 호출하면서 Athena API 를 폭주시켰다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | Cupix::Errors::System (code SYS20000) |
| exception.message | failed to get captured area - error: 500 Internal Server Error |
| top_frame | app/services/cupix/voxel_service.rb:99 |
| downstream | voxel-service Lambda (Voxel::CapturedArea) — Athena GetQueryExecution ThrottlingException |
| env | production, us-west-2, tenant=cupix |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| cupixworks-migration-worker (Sidekiq cron) | 10 | flush_stale_captured_size cron 배치에서 10개 Facility 의 captured_size 갱신이 실패. captured_size_state가 stale 그대로 남아 다음 cron 주기(4시간 뒤)에 재시도됨. 사용자 노출 화면(면적 표시)이 일시적으로 오래된 값 유지. |
Timeline#
- 2026-07-15 21:31 KST — cron 스케줄
every '31 */4 * * *'이 발화,Cupix::Cron::Facility.flush_stale_captured_size진입 (config/schedule.rb:105). - 2026-07-15 21:33 KST — 첫 실패 로그. 배치에서 처리 중이던 여러 Facility 에 대해
Cupix::VoxelService.captured_area!가RestClient::Exception(500) 로 rescue 됨. - 2026-07-15 21:34 KST 근방 — voxel-service Lambda 내부 로그에
handler_response - status_code: 500, body: "Failed to calculate captured area - An error occurred (ThrottlingException) when calling the GetQueryExecution operation (reached max retries: 3): Rate exceeded"다수 관측. - 2026-07-15 21:58 KST — 마지막 실패 로그 (Facility ID 16255). 이후 배치가 소진되며 자연 종료.
Scoring#
Confidence 80% · Actionability 16%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 17 code refs, 5 log queries |
| Code path explicitness | 3/3 | 17 file:line refs |
| Log evidence | 2/3 | section, query |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 14 file(s) |
| Single repo | 1/3 | 3 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 0/3 | migration needed |