ES /docs

VoxelService 503 — downstream throttling/concurrency limit

Service
cupixworks-worker
Occurrences
2,555
Status
Reported
Confidence
100%
Actionability
32%
Cluster ID
4f058ed2-3085-4144-b2f2-9062046d9e64
First Seen
2026-06-15 16:33 UTC
Last Seen
2026-08-05 04:52 UTC
Datadog↗ Slack↗
Actions

VoxelService 503 — downstream throttling/concurrency limit

What Happened#

2026-06-16 01:33–01:53 KST 동안 cupixworks-worker의 Cupix::Cron::Facility.flush_stale_captured_size cron이 실행한 Cupix::VoxelService.captured_area! 호출이 voxel service($CUPIX_VOXEL_SERVICE_URL/captured_area)로부터 503 Service Unavailable 을 반환받았다. Cupix::HttpClient.put 의 자동 재시도(최대 3회, exponential backoff)에도 불구하고 끝까지 실패한 facility가 10건 발생했고, 결과적으로 해당 facility들의 captured_size 갱신이 한 cron 사이클에서 누락되었다.

Quick Facts#

Field Value
exception.class Cupix::Errors::System (code SYS20000)
exception.message failed to get captured area - error: 503 Service Unavailable
top_frame app/services/cupix/voxel_service.rb:97
env production, region us-west-2, tenant cupix
upstream $CUPIX_VOXEL_SERVICE_URL/captured_area (voxel-service)

Affected Teams#

Team / Domain Error Count Impact
voxel-service (upstream) 10 Partial unavailability of /captured_area endpoint
cupixworks-worker (Sidekiq cron) 10 captured_size stale flush skipped for 10 facilities in this cycle

확인된 영향 facility ID: 13933, 17291, 17815, 17821, 17839, 17842, 17853 (그 외 일부는 cron 시작 시점 facility ID 미확인).

Timeline#

  1. 2026-06-16 01:33 KST — 첫 503 발생 (captured_area!, occurrence 시작)
  2. 2026-06-16 01:42 KST — 일부 요청은 정상 응답 (Completed - total: 10788/34759); voxel service는 완전 다운 아님, 부분 불가용
  3. 2026-06-16 01:33–01:53 KST — 약 2분 간격으로 cron이 facility 1건씩 처리, 매번 503 발생 (총 10건)
  4. 2026-06-16 01:53 KST — 마지막 503 (Facility 17853)

Scoring#

Confidence 100% · Actionability 32%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 49 code refs, 5 log queries
Code path explicitness 3/3 49 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 3/3 5 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 5/5 trivial
Affected files 1/5 28 file(s)
Single repo 1/3 3 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 0/3 migration needed