ES /docs

VoxelService captured_area API returns 503 Service Unavailable

RCA: failed to calculate captured size - 503 Service Unavailable

Error Log#

Datadog Logs

text
failed to calculate captured size for Facility ID: 10149, error: failed to get captured area - error: 503 Service Unavailable

Impact#

  • Service: cupixworks-worker
  • 발생 횟수: 93
  • 최초 발생: 2026-04-06T08:31:51.247Z
  • 최근 발생: 2026-04-07T08:36:15.381Z

Root Cause Summary#

4시간마다 실행되는 cron job Cupix::Cron::Facility.flush_stale_captured_size가 captured_size_state: stale 상태의 facility들에 대해 captured area를 재계산할 때, 외부 voxel service ($CUPIX_VOXEL_SERVICE_URL/captured_area)가 간헐적으로 503 Service Unavailable을 반환하고 있다. Cupix::VoxelService.captured_area!에서 RestClient::Exception이 발생하면 Cupix::Errors::System으로 래핑되어 상위로 전파되고, VoxelModule#calculate_captured_size의 rescue 블록에서 에러가 로깅된다. 실패한 facility는 stale 상태로 유지되어 다음 cron 주기에 재시도되지만, voxel service가 계속 503을 반환하므로 동일한 에러가 반복 누적된다. 30개 facility 배치 중 일부(약 1/3)만 503으로 실패하며 나머지는 정상 처리되므로, voxel service 측의 간헐적 과부하 또는 특정 facility의 데이터 크기에 따른 타임아웃이 원인으로 추정된다.

Technical Analysis#

Code Path#

  • Entry point (cron schedule): config/schedule.rb:105 — 4시간마다 flush_stale_captured_size 실행
ruby
# config/schedule.rb:103-106
every '31 */4 * * *' do # 00:31,04:31,08:31 ...
  runner 'Cupix::Cron::Cache.flush_user_permissions'
  runner 'Cupix::Cron::Facility.flush_stale_captured_size'
end
  • Cron handler: lib/cupix/cron/facility.rb:80-96 — stale 상태의 facility 최대 30개를 조회하여 순차적으로 calculate_captured_size 호출
ruby
# lib/cupix/cron/facility.rb:80-96
def flush_stale_captured_size
  session = Cupix::Initializer::User.tesla_internal_user.default_session
  if session.blank?
    Cupix::Logger.info('Failed to flush stale captured size: session is nil or blank', class: self.name, function: __method__, module: 'Cupix::Cron')
    return
  end

  ::Facility.stale_captured_size.limit(30).find_each do |facility|
    Cupix::Logger.info("Flushing stale captured size for Facility #{facility.id}", class: self.name, function: __method__, module: 'Cupix::Cron')

    if facility.calculate_captured_size(session: session)
      Cupix::Logger.info("Successfully flushed stale captured size for Facility #{facility.id}", class: self.name, function: __method__, module: 'Cupix::Cron')
    else
      Cupix::Logger.warn("Failed to flush stale captured size for Facility #{facility.id}", class: self.name, function: __method__, module: 'Cupix::Cron')
    end
  end
end
  • Non-bang wrapper: app/models/concerns/voxel_module.rb:19-27 — 예외를 rescue하여 에러 로그를 남기고 false 반환 (이것이 에러 로그의 출처)
ruby
# app/models/concerns/voxel_module.rb:19-27
def calculate_captured_size(session: nil)
  calculate_captured_size!(session: session)
rescue => e
  Cupix::Logger.error("failed to calculate captured size for #{self.class.name} ID: #{id}, error: #{e.message}", class: self.class.name, function: __method__, model: { id: id, type: self.class.name })

  false
else
  true
end
  • Bang method: app/models/concerns/voxel_module.rb:46-56 — Facility 인스턴스인 경우 Cupix::VoxelService.calculate_captured_size! 호출
ruby
# app/models/concerns/voxel_module.rb:46-56
def calculate_captured_size!(session: nil)
  if respond_to?(:captured_size)
    if self.instance_of?(::Facility)
      Cupix::VoxelService.calculate_captured_size!(facility: self, session: session)
    elsif self.instance_of?(::Team) || self.instance_of?(::Workspace)
      facilities.find_each do |facility|
        facility.calculate_captured_size!(session: session)
      end
    end
  end
end
  • VoxelService 호출: app/services/cupix/voxel_service.rb:14-46 — S3에 CSV 업로드 후 voxel service에 HTTP PUT 요청
ruby
# app/services/cupix/voxel_service.rb:14-46
def calculate_captured_size!(facility: nil, session: nil, **kwargs)
  # ... validation ...
  reality_captures_s3_object(facility.key).put(
    body: reality_captures_csv(facility),
    content_type: 'text/csv',
    # ...
  )

  _floorplans_count = facility.floorplans.untrashed.count
  response = captured_area!({
    facility_key: facility.key,
    group_by_record: _floorplans_count.zero?,
    'x-cupix-auth': Cupix::Auth::AccessToken.encode({ session: session })
  })

  facility.update!(captured_size: response['total'], captured_size_state: :fresh)
  # ... FootprintHistory 생성 ...
end
  • Failure point: app/services/cupix/voxel_service.rb:83-94 — RestClient.put이 503을 받으면 RestClient::ServiceUnavailable 발생, Cupix::Errors::System으로 래핑
ruby
# app/services/cupix/voxel_service.rb:83-94
response = RestClient.put("#{$CUPIX_VOXEL_SERVICE_URL}/captured_area", body.to_json, headers)
# ...
rescue RestClient::Exception => e
  Cupix::Logger.error("failed to get captured area - error: #{e.message}", class: self.name, function: __method__, facility_key: params[:facility_key], group_by_record: params[:group_by_record])

  raise Cupix::Errors::System.new(code: 'SYS20000', reason: "failed to get captured area - error: #{e.message}")

기대 동작: voxel service가 200 OK와 함께 { results: { total: N, details: [...] } }를 반환하고, facility의 captured_size_state가 :fresh로 전환됨.

실제 동작: voxel service가 간헐적으로 503 Service Unavailable을 반환하여 계산이 실패하고, facility는 stale 상태로 남아 다음 cron 주기에 재시도됨. 재시도 시에도 동일한 503이 발생하여 에러가 누적됨.

Log Evidence#

사용한 Datadog 쿼리:

text
service:cupixworks-worker status:error "failed to calculate captured size" "503 Service Unavailable"
text
service:cupixworks-worker "Flushing stale captured size"
text
service:cupixworks-worker "Successfully flushed stale captured size"
text
service:cupixworks-worker "Failed to flush stale captured size"

에러 패턴 — 동일 배치 내에서 성공/실패 혼재 (2026-04-07 17:31~17:36 KST 배치):

성공한 facility들:

text
17:31:58 KST - Successfully flushed stale captured size for Facility 1767
17:32:10 KST - Successfully flushed stale captured size for Facility 8291
17:32:55 KST - Successfully flushed stale captured size for Facility 12631
17:32:55 KST - Successfully flushed stale captured size for Facility 12834
17:32:55 KST - Successfully flushed stale captured size for Facility 12922
17:32:55 KST - Successfully flushed stale captured size for Facility 13474
17:32:55 KST - Successfully flushed stale captured size for Facility 13552
17:33:25 KST - Successfully flushed stale captured size for Facility 13760
17:33:27 KST - Successfully flushed stale captured size for Facility 13933
17:33:29 KST - Successfully flushed stale captured size for Facility 14063
17:33:31 KST - Successfully flushed stale captured size for Facility 14426
17:33:31 KST - Successfully flushed stale captured size for Facility 15093
17:34:33 KST - Successfully flushed stale captured size for Facility 15786
17:34:33 KST - Successfully flushed stale captured size for Facility 15789
17:35:07 KST - Successfully flushed stale captured size for Facility 16096
17:35:09 KST - Successfully flushed stale captured size for Facility 16141
17:35:41 KST - Successfully flushed stale captured size for Facility 16191
17:35:43 KST - Successfully flushed stale captured size for Facility 16391
17:35:43 KST - Successfully flushed stale captured size for Facility 16435
17:35:45 KST - Successfully flushed stale captured size for Facility 16530

실패한 facility들 (동일 배치):

text
17:31:45 KST - Failed to flush stale captured size for Facility 7023
17:31:50 KST - Failed to flush stale captured size for Facility 13
17:32:21 KST - Failed to flush stale captured size for Facility 12098
17:32:55 KST - Failed to flush stale captured size for Facility 12156
17:33:25 KST - Failed to flush stale captured size for Facility 13709
17:34:01 KST - Failed to flush stale captured size for Facility 15245
17:34:33 KST - Failed to flush stale captured size for Facility 15703
17:35:07 KST - Failed to flush stale captured size for Facility 15834
17:35:41 KST - Failed to flush stale captured size for Facility 16148
17:36:15 KST - Failed to flush stale captured size for Facility 16571

대응하는 voxel service 에러 로그:

json
{
  "timestamp": "2026-04-07 17:36:15 KST",
  "status": "error",
  "message": "failed to get captured area - error: 503 Service Unavailable",
  "class": "Cupix::VoxelService",
  "function": "captured_area!"
}

성공한 voxel service 응답 예시:

json
{
  "timestamp": "2026-04-07 17:35:45 KST",
  "status": "info",
  "message": "Completed - total: 9324, details: [{\"level_id\"=>68405, \"captured_area\"=>1103}, {\"level_id\"=>68369, \"captured_area\"=>8221}]",
  "class": "Cupix::VoxelService",
  "function": "captured_area!"
}

Facility 10149 타임라인:

text
2026-04-06 17:31:15 KST - [info]  Flushing stale captured size for Facility 10149
2026-04-06 17:31:51 KST - [error] failed to calculate captured size for Facility ID: 10149 - 503 Service Unavailable
2026-04-06 17:31:51 KST - [warn]  Failed to flush stale captured size for Facility 10149
2026-04-06 21:31:13 KST - [info]  Flushing stale captured size for Facility 10149
2026-04-06 21:31:15 KST - [info]  Successfully flushed stale captured size for Facility 10149
2026-04-07 13:31:45 KST - [info]  Flushing stale captured size for Facility 10149
2026-04-07 13:32:15 KST - [error] failed to calculate captured size for Facility ID: 10149 - 503 Service Unavailable
2026-04-07 13:32:15 KST - [warn]  Failed to flush stale captured size for Facility 10149
2026-04-07 17:31:45 KST - [info]  Flushing stale captured size for Facility 10149
2026-04-07 17:31:45 KST - [info]  Successfully flushed stale captured size for Facility 10149

이 타임라인은 Facility 10149가 일부 cron 주기에서 성공하고 일부에서 실패함을 보여준다. voxel service의 503이 간헐적임을 확인.

핵심 관찰:

  • 30개 배치 중 약 10개(1/3)가 매 cron 주기마다 503으로 실패
  • 실패한 facility는 stale 상태로 유지되어 다음 주기에 재시도
  • 일부 facility는 다음 주기에서 성공하지만, 새로운 facility가 실패하여 총 에러 수가 계속 누적
  • voxel service의 503 반환이 근본 원인 — worker 측 코드 결함이 아님

Fix Recommendation#

즉시 조치 (Critical)#

  • voxel service 상태 확인: $CUPIX_VOXEL_SERVICE_URL에 대한 health check 및 리소스 사용량 모니터링 필요. 503은 서버 과부하, 메모리 부족, 또는 upstream 의존성 실패를 의미하므로 voxel service 팀에 에스컬레이션.
  • app/services/cupix/voxel_service.rb:83-84: RestClient.put 호출에 timeout 옵션이 지정되어 있지 않아 RestClient 기본값(60초)이 사용됨. captured area 계산이 복잡한 facility의 경우 voxel service 쪽에서 처리 시간이 길어져 upstream load balancer가 503을 반환할 수 있음.

단기 개선 (1주 이내)#

  • Retry with backoff 추가: captured_area! 메서드에서 RestClient::ServiceUnavailable(503)에 대해 exponential backoff으로 1~2회 재시도하는 로직 추가. 현재는 재시도 없이 즉시 실패하여 다음 4시간 cron 주기까지 기다려야 함.
  • Timeout 명시: RestClient.put 호출 시 open_timeout과 read_timeout을 명시적으로 설정하여 voxel service의 응답 대기 시간을 제어.
  • 배치 처리 개선: 현재 30개 facility를 순차적으로 처리하므로 한 요청이 느려지면 전체 배치가 지연됨. 병렬 처리 또는 개별 facility에 대한 timeout을 고려.

장기 개선 (재발 방지)#

  • Voxel service 안정성: voxel service가 503을 반환하는 근본 원인 분석 필요. 특정 facility의 데이터 크기(captures/pointclouds 수)가 voxel service의 처리 용량을 초과하는지 확인.
  • Circuit breaker 패턴: voxel service 호출에 circuit breaker를 적용하여 연속 503 발생 시 일정 시간 호출을 중단하고, 불필요한 에러 로그 누적과 voxel service 부하를 줄임.
  • 실패 facility의 stale 상태 만료: 현재 실패한 facility는 영구적으로 stale 상태에 머물며 매 cron 주기마다 재시도됨. 일정 횟수 이상 실패하면 none 또는 error 상태로 전환하여 반복 재시도를 방지하는 메커니즘 필요.

Monitoring#

  • Voxel service 503 발생률 추적:
text
service:cupixworks-worker status:error "failed to get captured area" "503"
  • Cron 배치별 성공/실패 비율 추적:
text
service:cupixworks-worker "flush_stale_captured_size" (status:warn OR status:info)
  • Voxel service endpoint 응답 시간 메트릭:
text
avg:trace.rack.request.duration{resource_name:/captured_area,service:voxel-service}

Risk Assessment#

  • Risk level: medium — 기능적 데이터(captured size)가 갱신되지 않지만, 서비스 중단이나 데이터 손실은 없음. 실패한 facility는 다음 cron에서 재시도되며 일부는 성공함.
  • 예상 복잡도: standard — worker 측 retry/timeout 개선은 간단하나, 근본 원인인 voxel service 503 해결은 별도 팀 협력 필요.