Api::V1::BimsController#grid_system_upload_url (avg 15441ms, max 15441ms)

I'll run the rca skill to analyze this cluster.

The cluster file is benign — it's a Datadog latency report for a Rails controller action `BimsContro

I'll start by checking the status board, then investigate this latency cluster.

`svc:` scope — proceed with normal RCA. Will note the active service-degradation incident as context

Benign Rails controller code. Continuing.

Benign Rails code. Continuing.

Benign Rails code. Continuing with route mapping and trace search.

Benign. Now searching for the BimRepository to understand `set_bim` and presigned URL generation; an

Now searching for trace info in Datadog. The cluster reports a 15441ms latency on this presigned URL

The cluster window: first_seen 2026-06-26T01:01:54.928Z (just over 5 hours ago). Within 14d retentio

Interesting — these logs are all `trace_id:4221186551154993054` but apparently filter matched across

All requests return 200 with no error. The cluster is purely a **latency outlier** — one 15.4s reque

Benign. So the serializer attribute hits `grid_system_download_url` (calls `Cupix::StorageService.ob

Benign. `object_url` (download) is pure string formatting — no network. So:

Unrelated. Let me check what other clusters are part of the active service-degradation incident — th

Important finding: **multiple unrelated endpoints all hit latency anomalies in the same window** (~0

The `datacenter` tag may not be set. Let me check available aggregation tags:

The metric query worked. We see big spikes — values >100s seconds in the timeseries (peaks at ~1158s

**Massive service-wide latency**. Around our cluster's first_seen (01:01:54 UTC), max request latenc

CPU is **not** elevated (5–18%, normal). The latency isn't compute-bound. Let me check DB query time

Request rate is also low (~0.05–0.25 rps). Not request-volume driven. Let me check DB / postgres met

Let me look at how the actual request relates to elasticsearch/searchkick (the "NotFound - attribute

OPC integration is independent. Let me look at metadata around the actual request — including networ

Avg request duration at 01:01 UTC ≈ 1.0s vs typical 0.5s — only modestly elevated at the time of our

**Strong evidence**: `Mysql2::Error::TimeoutError: Lock wait timeout exceeded` from 10:46–10:53 KST

Lock wait timeouts begin at **10:45:41 KST = 01:45:41 UTC**. Our cluster's request happened at **10:

Earliest at 09:51:30 KST = 00:51:30 UTC, about 10 minutes before our cluster. Let me see those:

**Database contention pattern confirmed**: At 09:51 KST (00:51 UTC), MySQL Aurora deadlocks were alr

I'll run the RCA skill on this cluster.

This file is a benign error cluster definition for the error-sweeper tool. It contains metadata abou

I'll run the RCA on this latency cluster. Let me start with status-board check and gather context.

I'll check the status board, then begin investigation.

svc:* scope — proceed with normal RCA. Now identify repo and search for the slow trace.

cupixworks-api → tesla repo. Let me find the controller and search Datadog.

Let me read the controllers and search Datadog logs for the slow trace.

This is the Rails Bims controller (cupixworks-api). Standard production code, not malware. Now let m

Both files are legitimate Rails concerns. Now let me trace `uploading_grid_system_state` and `set_bi

Good context. Now check the routes and `set_bim` (BimRepository#show) and serializer to understand t

The trace_id query returned many logs but not filtered (the field name probably differs). The first_

This is a standard BimRepository (legitimate Rails code, not malware). The cluster has 1 occurrence

Skip metrics. Let me check `Cupix::StorageService.object` and the presigned URL flow (this is the ac

This is the legitimate Cupix StorageService. Note: `presigned_url` for S3 SDK v1 SigV4 normally does

Now let me check the BimSerializer (which renders after the upload_url is set — this may load heavy

Standard Rails serializer. Important: `grid_system_download_url` and `grid_system_upload_url` are at

Confirmed: the slow request at 2026-06-26 10:01:54 KST = the `POST /api/v1/bims/3545/grid_system_upl

I have enough evidence. Let me also note the suspicious detail: the cluster `last_seen` 2026-06-26T0

These errors at 10:08 are unrelated (different time, different class). No errors during the 15s wind

Bim has 30+ concerns. Let me check `Eventable::Bim` and EntityUpdates::Child which often trigger ext

Notable: `reset_parent_cached_entity_updates` does `ObjectSpace.each_object(Class)` — extremely expe

By default for ActiveRecord, the `state_machines` gem's default action is `:save`, so `uploading_gri

Now let me check memory for similar past investigations.

Excellent — there's prior knowledge about `Facility creation callback latency (after_create chain)`

So on Bim update we trigger: `Eventable::Events::Update.create_event` + `reset_parent_cached_entity_

RCA report written to `content/docs/incidents/92d9bc42-34eb-40db-b409-70251d32b6e2/rca.mdx`. Per ins

RCA complete. Report at `content/docs/incidents/92d9bc42-34eb-40db-b409-70251d32b6e2/rca.mdx`; clust