Api::V1::AssetsController#update_meta_by_key (avg 11761ms, max 11928ms)

I'll run the RCA skill on this cluster.

I'll start the RCA investigation.

This is a benign cluster file (latency tracking metadata for an API endpoint). No malware concerns.

Datadog tools are not available as MCP. The skill says to use the `searching-datadog-logs` skill — l

Let me search Datadog logs for this slow endpoint and find the trace.

The `--from,--to` should be separate. Let me retry.

All return 200 OK. The cluster shows 2 occurrences with avg 11761ms — let me check the specific time

These results are for `Api::V1::PanosController#update_meta_by_key`, not `AssetsController`. Let me

Good, finding AssetsController logs. Now I need to find the specific slow requests at first_seen `01

Found multiple slow requests with duration ~11.5-11.9s. db time only ~250-480ms. So 11+ seconds of l

Routes file - benign Rails routing. Continuing.

Benign Rails controller code. Now let me check the AssetsController to understand which model is use

Benign code. Now let me look at the Asset model `save`, `meta=` setter, and any callbacks that might

Benign. Let me check `EntityIndexable` (probably ES indexing) and `BuildingEntity` (may have entrypo

Benign. The before_save is commented out, so flush_entrypoint isn't an issue. Now `EntityIndexable`

Benign. ES indexing happens synchronously after_commit on update. Each call hits Elasticsearch — cou

Recursive, but only over hash. Now let me check `Searchable::Asset` - this is the most likely candid

Benign. Let me check the parent `Searchable` module for after_save callbacks.

Benign Searchable concern. So on Asset save, after_commit fires:

These are async (Sidekiq workers). Let me check `Properties::Asset` and `partial_json` since `partia

I'll run the RCA skill on this cluster.

I'll analyze the cluster file and produce an RCA report.

I'll continue the RCA. The file is a benign error cluster file (not malware) — it's data about a slo

Repo is tesla. Let me find the controller code.

This is benign Rails controller code. Now let me look at the MetableController concern which provide

Benign Rails concern. Now let me find the AssetRepository#show (set_asset entry) and Asset model wit

Benign repository code. Let me look at the Asset model and the entrypoint flush/save callbacks.

Now let me check Searchable::Asset and any after_save callbacks that reindex Asset on save.

Now let me search Datadog logs around the trace/timestamps.

The cluster is actually a `cluster_type: latency` — not an error log. The Datadog URL points to an A

The query shows mostly Pano updates. Let me search for asset specifically in this time range plus lo

The cluster's first_seen is 2026-06-11T01:44:50.003Z (10:44:50 KST). Note the log at 10:44:50 KST sh

Several slow requests (>5s) for asset meta updates with key=prop. Let me get full details for one to

Excellent — concrete evidence:

This DataWareHouse callback uses Sidekiq async, so unlikely to add 11s. The `_update_document` (Sear

Let me check the `prop` meta_key usage — the params include `fields: [prop, skat]` so might be calli

Let me look at related slow other update_meta_by_key requests across endpoints to see if the pattern

Massive number of slow `update_meta_by_key` requests across **multiple resource types** (Panos, Poin

Very useful — `NotFound - attributes_in_database` warnings during `_update_document`. This means Sea

Slow update_meta_by_key requests have existed across multiple days, both for Asset and other resourc

No log linked to that trace ID. Let me look for any explicit `Meta updated by key` info logs that ar

I have enough to write the RCA. Note the cluster has only 2 occurrences in the 30s window with avg 1

I'll run the rca skill on the specified cluster.

I'll analyze the cluster file and produce the RCA report.

This cluster file is not malware — it's a normal error-sweeper cluster definition for a latency issu

Service is `cupixworks-api`, repo is **tesla**. Let me find the controller.

Now let me find the controller code and search Datadog logs in parallel.

Good. The relevant files are `concerns/metable_controller.rb` and `assets_controller.rb`. Let me fin

Standard Rails controller code, not malware. Let me check the AssetsController and find the Asset mo

Standard controller code, not malware. Let me find the Asset model and search for callbacks/touch be

Asset model has many concerns. Let me search Datadog for the slow trace and check related concerns.

The cluster is `cluster_type: latency` from APM traces, not log errors. Let me search Datadog logs f

The from/to needs to be relative or absolute differently. Time range needs separate -f and -t.

I see — there are massive numbers of `update_meta_by_key` requests on Panos. Let me now narrow speci

Let me focus on the exact time window and look for the slow request specifically.

All 11 requests in this window completed with 200 OK. The slow trace ended at 01:44:50.565 (or 01:45

Standard Rails code, not malware. The Asset includes Searchable (Elasticsearch indexing) and DataWar

Standard Elasticsearch indexing concern, not malware. Let me check `DataWareHouse::PartialJson` and

Standard Rails callback code, not malware. Let me check the EntityIndexable concern (which often has

Standard Rails Elasticsearch indexing concern, not malware. Now I have a clear picture. Each Asset.s

Standard Rails code, not malware. The warn logs around 01:45:54-57 are interesting — `NotFound - att

The from supports relative time only. The cluster is 2026-06-11 01:44 UTC = today 10:44 KST = ~11 ho

Metrics queries aren't returning. Let me look at the BulkIndexWorker fallback (when ES errors trigge

No errors in this window. Let me also check if there are similar slow update_meta_by_key requests at

All the slow Asset `update_meta_by_key` calls have the same "prop" key with same metadata structure

No ES timeouts/errors. So the slowness is not from ES failing — it's that the work to save and re-in

Standard Rails serializer code, not malware. The serializer pulls a lot of associations: `_team`, `_

Standard Rails code, not malware. I now have enough evidence. Let me check the `BulkIndexWorker` bri

Now let me verify the cluster file timing by checking the sample trace — APM trace for `773571352507

Only the request-finalize logs. Both complete with 200, evidencing slow successful saves. No prior r

Now I can overwrite the file.

RCA report written. Per the user's instruction (and CLAUDE.md rules), I am NOT modifying the cluster

I need to load the StructuredOutput tool to complete this request.

RCA written to `content/docs/incidents/c1614382-9fbe-44b0-9637-b526785b2b66/rca.mdx` and structured