Searchable#_update_document — synchronous ES callback pool exhaustion

I'll run the rca skill on this cluster.

The cluster file is a benign error-tracking metadata file (YAML frontmatter + markdown describing a

I'll follow the RCA skill. Let me start by checking the status board and gathering context.

The skill expects a cluster ID.

Recent svc-scope incident — proceed with normal RCA. Now reading writing guidelines and exploring re

Benign documentation file. Continuing.

tesla repo. Now let me find the PanosController#index code and search Datadog logs in parallel.

Benign Rails controller code. Now let me find PanoRepository.search.

Benign Rails code. Continue reading.

Benign Rails repository code. Let me look at the key parts: default_joins and permission_joins and t

Benign Rails query code with many LEFT JOINs across permissions tables. This is a complex query with

This cluster is APM/latency, not log errors. Let me search logs around this resource and trace.

Lots of normal traffic. Now let me search with duration filter to find the slow ones, and try to fin

Logs use ms `@duration`. Let me find slow ones (>5000 ms).

The output starts with text. Let me strip leading lines.

Excellent evidence. The slow requests cluster around team `gad` paginating through 460 results with

Critical finding. The 68s request asked for `resources`, `original_upload_url`, `upload_url`, `mask_

Benign serializer code. Now let me check what `resources`, `tile_download_urls`, and `original_uploa

Critical finding: `upload_url` calls `self.uploading unless self.uploading?` — that's a state machin

`pano.resource` calls `resources.where(kind: nil).first` — N+1 risk. The repository preloads `:resou

Now check `view: 0.11` field in logs — it's tiny. So the slow part is `db` (5.8s) + Ruby/AWS SDK tim

Let me check Sidekiq concurrent activity / DB metrics around that time:

I have enough evidence. Let me now write the RCA report.

Computing KST timestamps:

RCA report written to `content/docs/incidents/bd35bc1d-ced5-4866-9841-d14d463894ee/rca.mdx`.

I'll run the RCA skill on this cluster.

I'll start by checking the status-board and gathering context.

This is a latency cluster (not an error), so the cluster file has `## Representative Span` instead o

Status board shows this cluster was part of a recent resolved "cupixworks-api service degraded" inci

Let me find the controller in tesla repo and check Datadog APM traces.

Not malware - standard Rails controller. Let me look at the search repository code.

Not malware - it's a Rails ActiveRecord repository. Let me find the `search` method.

Both files are legitimate Rails code. Now let me look at metrics around the incident window and chec

These errors are from external integrations (OPC, BIM360) — unrelated to the latency. Let me search

Not malware - this is standard Rails permission joins for ACL filtering. The complexity here is sign

Confirmed — uses Elasticsearch (`::Pano.search`). Let me check related metrics (DB/ES) during the sl

Let me check what was happening during the cluster window at 16:45-17:59 UTC on 2026-06-24 — the tim

There's a heavy burst of "NotFound - attributes_in_database" warnings around 03:29 UTC on 2026-06-25

ES query duration shows a clear spike — going from ~0.05s baseline up to 0.366s, then settling to ~0

Now I understand: `NotFound - attributes_in_database` warn means a Pano model invoked `_update_docum

Overall API latency is elevated too (avg 0.3-0.6s typical, peaks higher). The latency cluster is lik

Not malware. Now I have enough evidence. Let me write the RCA report.

Now I'll write the RCA report.

Not malware, just a docs file. Now I can write.

RCA report written. Note: per the user's instruction I did NOT modify the cluster frontmatter — the

RCA report written to `content/docs/incidents/bd35bc1d-ced5-4866-9841-d14d463894ee/rca.mdx` and stru