batch_pull! error on batch arn:aws:ecs:us-west-2:002596530511:task/cupix-tesla-ece/f08cf04288c144f38
Service
cupixworks-worker
Occurrences
169
Status
Reported
Confidence
85%
Actionability
28%
Cluster ID
7c5d871d-9f84-4ae9-927f-b8d28ac3fd35
First Seen
2026-07-13 07:06 UTC
Last Seen
2026-07-13 07:34 UTC
Datadog
SlackActions
batch_pull! error on batch arn:aws:ecs:us-west-2:002596530511:task/cupix-tesla-ece/f08cf04288c144f38
What Happened#
cupixworks-worker의 AwsTask.batch_pull! cron job 이 AWS ECS DescribeTasks API 를 호출하는 과정에서 AWS Throttling(Rate exceeded) 예외가 반복적으로 발생. 2026-07-13 16:06 KST 부터 16:34 KST 까지 약 28분 동안 169건의 error 로그가 기록되었으며, 모든 로그는 동일 host (ip-10-1-17-34.ap-southeast-1.compute.internal) 의 whenever cron runner 에서 발생함.
Quick Facts#
| Field | Value |
|---|---|
| exception.class | AWS ECS Throttling (rescued as StandardError — 원본 class 미기록) |
| exception.message | Rate exceeded |
| top_frame | app/models/aws_task.rb:87 |
| runtime | Ruby (Rails whenever cron runner) |
| deploy | production-ap-southeast-1-20260713T0734Z0-69260da5-cupixworks |
| env | production, ap-southeast-1 (worker) → us-west-2 (ECS API 대상) |
Affected Teams#
| Team / Domain | Error Count | Impact |
|---|---|---|
| cupixworks-worker (AwsTask cron) | 169 | ECS task 상태 폴링 지연 — pending/blank 상태 record 가 최신화되지 못함 |
에러 자체가 사용자 트래픽을 직접 차단하지는 않지만, AwsTask.after_save 콜백을 통해 Job#run_task_stopped_callbacks 를 트리거하는 파이프라인이 지연될 수 있음.
Timeline#
- 2026-07-13 16:06 KST — 최초
batch_pull! ... Rate exceeded발생 (first_seen) - 2026-07-13 16:28–16:34 KST — 초당 여러 건 페이스로 반복 (169건 누적, last_seen 16:34 KST)
- 2026-07-13 16:58 KST — 동일 코드 경로에서 별개 오류 (
ecs:DescribeTasks not authorized,taskId length should be one of [32,36]) 관측 — 본 클러스터와 별도로 조사 필요 - Status board —
svc:cupixworks-worker::unknown인시던트2026-07-13-svc-cupixworks-worker--unknown-2에 포함, 다른 2개 cluster (8a242739...,1ad7f6fa...) 와 함께 그룹화됨
Scoring#
Confidence 85% · Actionability 28%
Confidence#
| Dimension | Score | Detail |
|---|---|---|
| Evidence completeness | 5/5 | 15 code refs, 5 log queries |
| Code path explicitness | 3/3 | 15 file:line refs |
| Log evidence | 3/3 | section, query, pattern |
| RCA structure | 3/3 | 3/3 sections |
| Revision history | 0/3 | 0 revision(s) |
| Agent execution | 3/3 | success |
Actionability#
| Dimension | Score | Detail |
|---|---|---|
| Fix complexity | 1/5 | critical |
| Affected files | 1/5 | 10 file(s) |
| Single repo | 1/3 | 3 service(s) |
| Immediate actions | 0/3 | absent |
| Testability | 0/3 | no test info |
| External dependency | 1/3 | external deps found |
| Data migration | 3/3 | no migration |