Searchable concern synchronous Elasticsearch indexing timeout

I'll run the rca skill to analyze this cluster.

I'll start by reading the cluster file and checking the status board.

The cluster file is not malware — it's a benign latency cluster from the error-sweeper collector tra

`svc:*` scope — proceed with normal RCA. Note context: cupixworks-api service degraded.

Standard Rails controller, not malware. The `bulk` action is in `BulkableController`. Let me find it

Standard bulk controller, not malware. Let me find the `bulk` method in PanoRepository / Factory.

Standard CRUD bulk repository, not malware. Let me look at the factory bulk method and pano_factory.

Standard factory code, not malware. Let me check pano-specific factory and search Datadog for actual

Now let me search Datadog for traces.

This is a latency cluster (APM trace data), not log errors. Let me search for related logs around th

Lots of bulk PUTs succeeding. Let me check the actual slow/error ones and look at APM trace search.

Errors during the window are unrelated. Let me try a different approach — try the bulk action specif

All 200 OK responses. So `bulk` is succeeding — just slow. Let me check duration info from these log

Strong signal:

PanoRepository doesn't have siteinsights producer. Let me check what's happening in the per-pano upd

Standard code. So in the bulk update path: 100 items × per-item update. Each `_repository.update(_it

Pano includes many concerns — `EntityIndexable`, `Searchable`, `Eventable`, `Publishable`, `Revision

So per item: `Eventable::Events::Update.create_event` runs. Plus `Searchable::Pano` does Elasticsear

Found it. **Per-pano update triggers `after_commit on: [:update]` → `_update_document` → synchronous

No ES errors. ES writes are succeeding but slow due to N×1 pattern. Let me verify there is no batch

No recent changes in those files. This is a pre-existing N+1 indexing pattern, not a regression. Let

Now let me check writing guidelines briefly and compute KST times:

I'll run the RCA skill on the specified cluster.

This file is a benign error cluster definition (YAML frontmatter + markdown) used by the error-sweep

I'll check the status board first, then proceed with the RCA skill.

This is a `svc:*` scope active incident — context only, proceed with normal RCA. Now invoke the rca

I'll now conduct the RCA. Let me start by gathering evidence in parallel.

cupixworks-api → tesla repository. Let me search for the controller code and Datadog logs in paralle

Not malware — standard Rails controller. The `bulk` action is provided by `BulkableController` conce

Not malware. Standard bulk controller. Let me look at the repository/factory `bulk` methods.

Not malware. Let me look at the pano_repository for any pano-specific bulk overrides.

Not malware. Let me check `pano_repository.rb` and `pano_factory.rb` for pano-specific bulk hooks (e

Not malware — normal application code. Let me check pano-specific BulkableRepository/Factory hooks a

Not malware. Let me search for trace-level info — duration distribution and traces around bulk_actio

Excellent — I have rich evidence. Each request has `bulk.items_count` and `duration` (ms) and `db` (

The script outputs the header into stdout too. Let me use a different filter that worked earlier. Th

Strong evidence: All slow requests are `action: update` with `items_count: 100` (or close to it). `d

So each iteration: per-item permission check → set_parameters → save! → triggers callbacks (ES index

Not malware. Pano has many concerns with after_commit hooks (`Searchable`, `EntityIndexable`, `Event

Not malware. **Critical finding**: `EntityIndexable` performs a **synchronous Elasticsearch index ca

Not malware. Let me find the `Searchable` parent module to see if it does sync indexing on save.

Confirmed: each `save!` triggers **two synchronous Elasticsearch index calls per pano** (one from `S

I have enough evidence. The Eventable details are not crucial — the dominant non-DB cost is clearly

All 50 sampled requests in 90 minutes are `update` action with avg duration 7.1s, avg items 75. Let

Now writing the RCA report. Timestamps: first_seen `2026-06-24T04:53:32.928Z` → `2026-06-24 13:53 KS

Not malware — it's an RCA report file. Now writing the full content.

RCA report written to `/home/ec2-user/error-sweeper/content/docs/incidents/42e28124-cfc8-4f3e-89af-2

I need to fetch and call the StructuredOutput tool.

RCA complete. Report at `/home/ec2-user/error-sweeper/content/docs/incidents/42e28124-cfc8-4f3e-89af