ES /docs

AWS RDS MySQL failover — read-only mode during write

Service
cupixworks-any-compute-agent
Occurrences
8
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
64a1e5c5-d4db-4645-b253-babce3772d7d
First Seen
2026-05-23 21:15 UTC
Last Seen
2026-07-13 07:25 UTC
Datadog↗ Slack↗
Actions

AWS RDS MySQL failover — read-only mode during write

What Happened#

2026-05-23 21:15~21:48 UTC 시간대에 cupixworks-any-compute-agent 서비스에서 ECS task를 성공적으로 시작한 후 job 상태를 업데이트하려 할 때, 하위 API(api-tesla.cupix.internal)로의 PUT 요청이 MySQL read-only 에러로 실패했다. 동일 시간대에 cupixworks-api, cupixworks-worker 등 여러 서비스에서 동일한 read-only 에러가 발생하여 RDS failover 이벤트로 인한 전체 쓰기 중단임을 확인했다.

Quick Facts#

Field Value
exception.class HttpError (agent-side), Cupix::Errors::Parameter (API-side)
exception.message Mysql2::Error: The MySQL server is running with the --read-only option so it cannot execute this statement
top_frame qmJob.ts:183 → job_repository.rb:47
env production, us-west-2

Affected Teams#

Team / Domain Error Count Impact
ellisdon (cupix tenant) 2 Job 1085021, 1085024의 task_id 업데이트 실패 → ECS task 강제 중지

Timeline#

  1. 2026-05-23T21:08:02Z — cupixworks-worker에서 최초 read-only 에러 감지 (flush_geo_coordinate)
  2. 2026-05-23T21:15:26Z — cupixworks-any-compute-agent job 1085021 updateTaskIdToJob 실패 (최초 클러스터 발생)
  3. 2026-05-23T21:20:26Z — cupixworks-api에서 Eventable::Events::Update read-only 에러
  4. 2026-05-23T21:48:20Z — cupixworks-any-compute-agent job 1085024 updateTaskIdToJob 실패 (마지막 클러스터 발생)
  5. 2026-05-23T21:56:45Z — cupixworks-api에서 Clusters, References 등 다수 엔드포인트 read-only 에러 지속

Scoring#

Confidence 85% · Actionability 28%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 13 code refs, 5 log queries
Code path explicitness 3/3 13 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 9 file(s)
Single repo 1/3 4 service(s)
Immediate actions 0/3 absent
Testability 0/3 no test info
External dependency 1/3 external deps found
Data migration 3/3 no migration