Excon::Error::Forbidden: Expected(200) <=> Actual(403 Forbidden) excon.error.response :body => "<?xml version=\"1.0\" en
Service
cupixworks-api
Occurrences
158
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
0b3cfc31-4198-4bad-9673-bceb36d904cf
First Seen
2024-09-18 05:58 UTC
Last Seen
2026-08-05 01:42 UTC
Datadog
SlackActions
Excon::Error::Forbidden: Expected(200) <=> Actual(403 Forbidden) excon.error.response :body => "<?xml version="1.0" en
What Happened#
cupixworks-api (실제 앱 = tesla) 가 CarrierWave 를 통해 pointcloud 썸네일을 S3 cupix-tesla-static 버킷의 tmp/cache 경로로 PUT 업로드할 때, S3 가 403 Forbidden(SignatureDoesNotMatch 및 소수의 InvalidAccessKeyId)를 반환한다. 실패는 항상 전역 CarrierWave 설정(config/initializers/carrierwave.rb)이 사용하는 access key AKIAQBGWD5FH5URQYYOO 로 서명된 요청에서 발생하며, production 환경 다수 테넌트(walmart, gilbaneco, pclconstruction, accoes, tpc 등)에 걸쳐 결정론적으로 재현된다. 코드 결함이 아니라 전역 fog credential 이 stale/rotated/deactivated 되어 서명이 맞지 않는 credential 문제다.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | Excon::Error::Forbidden |
| exception.message | Expected(200) <=> Actual(403 Forbidden) — S3 <Code>SignatureDoesNotMatch</Code> (일부 InvalidAccessKeyId) |
| top_frame | config/initializers/carrierwave.rb:3-8 (전역 config.fog_credentials) |
| runtime | fog-aws → Excon (CarrierWave storage :fog cache/store) |
| env | production, us-west-2 |
| AWSAccessKeyId | AKIAQBGWD5FH5URQYYOO |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| cupixworks-api (tesla) — thumbnail upload | 14d 기준 365 spans (SignatureDoesNotMatch 330 + InvalidAccessKeyId 35) | pointcloud 썸네일 업로드가 조용히 실패 (사용자에게 500 노출 없음 — 아래 Impact 참조) |
Timeline#
- 2024-09-18 14:58 KST — 최초 발생 (
first_seen). Representative sample 의 StringToSign 날짜20240918와 일치. - 2024-09-18 ~ 2026-08-04 — ~23개월간 지속 재발, 149건 집계. 전역 credential
AKIAQBGWD5FH5URQYYOO로 서명한 PUT 이 계속 거부됨. - 2026-08-04 11:27 KST — 최근 발생 (
last_seen). APM span 타임스탬프2026-08-04T02:27:46Z(UTC) 와 정확히 일치 → representative 는 stale 아님.
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 30 code refs, 7 log queries |
| Code path explicitness | 3/3 | 30 file:line refs |
| Log evidence | 2/3 | section, query |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 1/3 | 1 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 21 file(s) |
| Single repo | 1/3 | 3 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |