ES /docs

flush_geo_coordinate - error - message: Failed to open TCP connection to s3.me-south-1.amazonaws.com

Service
cupixworks-migration-worker
Occurrences
3,016
Status
Reported
Confidence
85%
Actionability
24%
Cluster ID
ea35f48e-731d-4fad-a1ec-05ca11a431c6
First Seen
2026-07-13 04:07 UTC
Last Seen
2026-07-30 08:32 UTC
Datadog↗ Slack↗
Actions

flush_geo_coordinate - error - message: Failed to open TCP connection to s3.me-south-1.amazonaws.com

What Happened#

cupixworks-migration-worker (us-west-2 host ip-10-1-144-200) 에서 FlushRecordGeoCoordinateWorker 가 migration 후속 단계로 실행되었고, 대상 record 의 storage bucket 리전이 me-south-1 (Bahrain) 이었다. Rails Sidekiq 프로세스가 s3.me-south-1.amazonaws.com:443 으로 TCP 연결을 시도했으나 open_timeout 안에 응답을 받지 못하고 Net::OpenTimeout (Ruby 표현: execution expired) 이 발생, 43건의 flush 실패가 기록되었다.

Quick Facts#

Field Value
exception.class Net::OpenTimeout (Ruby, 메시지에서 execution expired 로 표기)
exception.message Failed to open TCP connection to s3.me-south-1.amazonaws.com:443 (execution expired)
top_frame app/models/concerns/record_geo_coordinate.rb:45 (geo_coordinate_s3_object.upload_stream)
runtime Ruby / Rails / Sidekiq, service_role: migrationworker
deploy production-us-west-2-20260713T0506Z0-4f01ffc0-cupixworks
env production, region us-west-2, host ip-10-1-144-200.us-west-2.compute.internal

Affected Teams#

Team / Domain Error Count Impact
cupix (tenant) — records with storage_option s3_bucket_region: me-south-1 43 migration 직후 record 의 geo_coordinate JSON 이 me-south-1 S3 hosting bucket 에 업로드되지 못함. geo_coordinate_url_updated_at 이 갱신되지 않아 downstream 에서 최신 pano geo_coordinate JSON 을 서빙하지 못함.

로그 tag 는 tenant:cupix 만 노출되고 record 단위 team domain 은 로그에 붙지 않음 (uncertain -- team_domain not tagged in this log stream).

Timeline#

  1. 2026-07-02 13:07 KST — 동일 오류 계열이 최초 관측 (flush_geo_coordinate - error - message: Failed to open TCP connection to 207.127.99.67:80 (execution expired), 동일 코드 경로). me-south-1 host 로 조준되는 실패는 이전부터 존재.
  2. 2026-07-13 13:07 KST — 현재 클러스터 첫 발생 (first_seen 2026-07-13T04:07:25.667Z UTC).
  3. 2026-07-13 13:07~13:30 KST — 23분 동안 43건 반복, 평균 32초 간격 (record ID 131788, 131790 등 최소 2개 이상 record 대상).
  4. 2026-07-13 13:30 KST — 클러스터 last_seen 이후에도 14:07~14:13 KST 구간에서 동일 오류 재발이 확인됨 (Datadog now-2h 재검색).

Scoring#

Confidence 85% · Actionability 24%

Confidence#

Dimension Score Detail
Evidence completeness 5/5 32 code refs, 4 log queries
Code path explicitness 3/3 32 file:line refs
Log evidence 3/3 section, query, pattern
RCA structure 3/3 3/3 sections
Revision history 0/3 0 revision(s)
Agent execution 3/3 success

Actionability#

Dimension Score Detail
Fix complexity 1/5 critical
Affected files 1/5 23 file(s)
Single repo 1/3 2 service(s)
Immediate actions 0/3 absent
Testability 2/3 test mentioned
External dependency 1/3 external deps found
Data migration 0/3 migration needed