Api::V1::RecordsController#index (avg 50745ms, max 50745ms)

I'll run the RCA skill on this cluster.

I'll start by reading the cluster file and the writing guidelines.

The cluster file is not malware - it's a benign cluster definition documenting a slow API latency in

Active `svc:` scope incident exists — proceed with normal RCA. Note this is a latency cluster, not a

Repo is `tesla`. Let me find the controller and related code.

Benign Rails controller. Now find the RecordRepository.

Benign Rails repository code. Now let me find where search delegates and check trace details in Data

Let me search Datadog for that trace and related slow request logs.

Most are quick 200 OK responses. Let me look for the specifically slow request around 19:22:47Z and

Good, request logs include `duration` field. The slow request is 50745ms. Let me search by duration

Got a clear slow request. `duration: 50733ms`, `db: 1805.93ms` — DB time is only ~1.8s, so the remai

Only 2 slow requests in the window. Let me check for any ES errors / warnings around that time, and

Pano warnings unrelated to records search. Let me check rails.request metrics and EC2 host metrics a

Interesting — there are timeouts on `ReviewRepository#captures`, just 5 minutes earlier. Let me find

Benign Elasticsearch index definition. Let me check `directly_accessible_record_ids` and `readable_f

Let me also check `readable_facility_ids` and look for ES timeouts at the same time window:

Strong signal: `Mysql2::Error::TimeoutError: Lock wait timeout exceeded` on `Pano` operations during

Same facility, same params – earlier request: 2595ms. Slow one: 50733ms. DB time roughly doubled (12

Need to create the dir. Now let me also check related db time of the second slow request:

Two slow requests: 50.7s (db 1.8s, view 0.0s, serialization 136ms) and 32.2s (db 2.2s, view ~0). In

No Elasticsearch errors logged. Let me also query the rails request metric for that latency hotspot:

Interesting — `duration: 53978.7ms` with explicit `ActiveRecord::LockWaitTimeout` (≈50s + setup). Th

Just setters. Now look at where `search` is called in BaseRepository — to confirm it delegates to ES

Now I have a clear picture. The search flow:

Clear pattern: a barrage of `PanosController#check_uploading`, `check_tile_uploading`, `mask_upload_

Now timestamp conversion:

RCA report written. Per the user's instructions, I am NOT to modify the cluster frontmatter (the pip

RCA report written to `content/docs/incidents/cfe24462-0164-4097-acea-c69855c91566/rca.mdx`. Cluster