Api::V1::PointcloudsController#create_resource (avg 20803ms, max 20803ms)

I'll run the rca skill on this error cluster.

I'll analyze this error cluster. Let me start by reading the cluster file and checking status.

This is a legitimate error cluster file for RCA analysis, not malware. It's an APM latency cluster f

`svc:` scope — proceed with normal RCA. Related cluster `1a719bfe-5c3f-4428-9ba8-d0af2c7c4d37` fired

`cupixworks-api` maps to tesla. Let me find the controller and search Datadog.

Not malware — standard Rails controller. `create_resource` isn't defined here directly, so it's from

Not malware — standard Rails concern. Now let me examine `resource.save` — Resource model presumably

Let me search for logs around the trace.

Single info log with successful 200. Let me get more details of this log.

Excellent data. Key findings:

Two slow calls in 24h — both from same tenant?

Different tenants (walmart 1124 and fgip-pkg1 1239). Now let me look at the Resource model to unders

Not malware — Rails model. Many concerns. Let me look at what runs on create/save that could be slow

Not malware. Important: `after_commit :_entity_index_document, on: [:create]` performs a **synchrono

Many slow requests during that window — service-wide slowness. Let me check DB metrics/broader slown

Widespread service latency during the window — many controllers, mostly PanosController. But DB time

No ES errors. Let me check DB metric during window and check if there's baseline slowness in the DB.

Average duration is ~0.3-1.2s during period. There's a spike to ~1.1s. Let me check p99 or max:

Max is consistently 15-45 seconds throughout the period — this is chronic tail latency, not an acute

Let me check `Statable::Resource` and `Storagable::Resource` briefly since they can add before_save

So the only lifecycle callback on `resource.save` for create in the concerns is `EntityIndexable#_en

Not malware. This is significant — `permission_joins` builds a very complex query with 8 LEFT JOIN s

I'll run the rca skill to analyze this cluster.

I'll start by reading the cluster file and checking status board.

This is a benign cluster file for RCA analysis, not malware. Proceeding with the RCA.

This is an `svc:` scope, so I proceed with normal RCA. Sibling cluster `1a719bfe-5c3f-4428-9ba8-d0af

cupixworks-api → tesla repo. Let me find PointcloudsController.

Standard Rails controller code, not malware. Let me find `create_resource`.

Understood. Standard concern, not malware. Note: `create_resource` involves `@model.resources.new` +

This is a latency cluster (trace-based), not an error log cluster. Let me search Datadog for related

Good — endpoint is normally fast. Let me search around the incident timestamp specifically (2026-07-

All returned 200. The slow one at 03:34:37 UTC = 12:34:37 KST would have been the request that start

These NotFound warnings occur after the incident. Let me search around the exact time window (03:34:

High volume of Elasticsearch `_update_document` NotFound warnings during that window. This suggests

Let me look at the Resource model save callbacks (which run in the request path) to identify slow pa

Standard AR model, not malware. Numerous concerns included. `EntityIndexable` likely handles Elastic

Standard indexing code, not malware. This is a synchronous ES call on save. Now let me check Storaga

Understood. This isn't malware. `create_resource` runs `resource.save` — during which `after_commit`

Let me check the Datadog metrics for postgres/api latency during that window to see if there was a g

Only relative time. Let me check for any DB deadlock / connection issues in logs during that window.

So pointcloud/1203856 create_resource completed at 12:34:29 KST (03:34:29 UTC). The 20.8s span with

Interesting — CapturesController#create_resource returned 400 (Duplicate kind), not the pointcloud o

The exact log entry for the slow request may not have made it to the info-level completion log if it

No create_resource logs in the sibling's window either. Let me look for what's happening in that bro

Understood. Not malware. Confirmed: `Searchable#_update_document` runs synchronously in `after_commi

Average ES query duration ~200ms consistently (no spike). So there's no clear ES slowdown. Let me lo

So we can see in the 30-second window, there's one create_resource at 12:34:29 (for 1203856) that re

Let me check for reindex activity in logs during the incident window:

No active reindex. Let me look at overall API traffic during the window:

Concurrent bulk pano creation was happening at ~50 requests/second. Very high volume — matches the m

That's for `create`, not `create_resource`. `create_resource` doesn't call factory. I have enough ev

I'll run the rca skill to analyze this error cluster.

I'll start by checking the status board, then reading the cluster file and any relevant memory.

The cluster file is a benign error-tracking artifact — not malware. Continuing with the RCA.

Memory file is benign notes. Now let me check the sibling cluster and check for a services memory fi

Benign notes. Now let me find the repo and code path.

Benign controller. `create_resource` isn't defined here — it must be in a concern. Let me find it.

Benign. Now let's search Datadog for the trace and related latency logs.

The cluster is a **latency** cluster (`cluster_type: latency`), not an error. The Datadog link point

I need to search specifically for the POST create_resource request around 03:34:37Z. The event was a

Interesting: 03:34:37 UTC = 12:34:37 KST. The slow request is likely `pointclouds/1203858/resources`

Let me get the exact request record (with duration/trace_id) — check the raw record for the 12:34:59

Key finding: The slow request completed at `03:34:59.481Z`, `duration: 20707.38ms`, `db: 12783.07ms`

Good context. Pointcloud 1203858 was created at 12:34:32, transitioned to `queued` at 12:34:36, then

Benign. The 12.8s db time on a simple `Resource#save` (with `find_by_kind` upfront) is suspicious. T

Let me check other relevant callbacks: WorkspaceEntity::Resource, EntityIndexable, and Cachable at R

Benign. This after_commit does an ES index but does NOT count in db time. The db=12.8s is Postgres.

Let me look at `Cachable::Pointcloud` and any `Cachable` module (touches parent, invalidates review

Let me check RDS/DB metrics around 03:34:37Z to see if postgres was slow overall (common cause of la

Metrics tool only supports relative time ranges (24h max). Too far back (event was ~5-6 hrs ago). Le

Lots of slow requests in this window — panos bulk, element_traces bulk. This is systemic, not specif

Very interesting: capture 728538 has multiple slow updates (13.2s, 12.8s, 9.9s) with **low db time (

Important context: the same user is uploading many pointcloud thumbnails for capture 728538, and hit

No PG errors. Split-lock chatter is a separate feature. The db=12.8s on a simple `resource.save` is

So user 50322 (walmart team) creates ~5 pointclouds within seconds (1203856-1203860+), all doing thu

Very strong evidence: **DB was contended during this window**:

Excellent — strong confirmation. At 03:31-32 UTC, similar pattern of high-db latency across many end

An RCA report already exists from a prior run. It's an RCA doc (benign, not malware). The instructio

No `def show` in pointcloud_repository — let me check base repo and confirm what `show` does.

Confirmed: `BaseRepository.show` DOES call `permission_joins(default_joins(current_class), current_u

Confirmed — permission_joins builds ~8 LEFT JOIN subqueries even for a single-row `show` fetch. This

I'll invoke the rca skill to analyze this cluster.

I'll analyze this error cluster following the RCA skill process.

Note: This is a benign latency RCA cluster file, not malware. Proceeding with analysis.

Note: svc:* scope — proceed with normal RCA flow. There's a companion cluster `1a719bfe-5c3f-4428-9b

cupixworks-api → tesla repo. Let me find the controller.

Not malware — standard Rails controller. `create_resource` isn't defined directly. Let me trace it v

Not malware — standard Rails concern. Now let me look at Pointcloud resource save flow (likely S3-re

This is a **latency** cluster (trace, not error log). Let me query APM traces and any related error

Good — 200 responses. Let me search for the specific trace and check what's slow.

The trace ended at 03:34:37.040Z with 20803ms duration → started around 03:34:16.237Z. Not visible a

Let me look for the *specific* slow request. The trace's last_seen is 03:34:37.040Z with 20803ms dur

Now let me look for what request took 20 seconds. The trace ended at 03:34:37.040Z, so it started at

I notice the completion at 03:34:29.432 for pointcloud/1203856. If that request took 20803ms, it wou

Companion cluster file no longer exists — it may have been processed. Let me check current DB metric

Not malware. Many concerns/callbacks — save can be expensive. Let me check APM DB/S3 metrics around

Metric works — resource_name filter probably not indexed. Let me check DB slow queries or general sl

Let me check if there are other slow trace signals — the burst of `NotFound - attributes_in_database

That's a general background pattern (frequent throughout). Not tied to this specific slow request. L

Not malware. Now let me check the Cachable/EntityIndexable/Revisionable concerns — these may hit Ela

Not malware. Uses `after_commit` for ES indexing. Let me check what happens in Revisionable/Cachable

Not malware. So `create_resource` for PointcloudsController uses the shared MultipleResourcableContr

I have enough evidence to write the report. Let me update todos and write.

KST conversion: 2026-07-08T03:34:37.040Z → 2026-07-08 12:34 KST.

I'll run the RCA skill on this cluster.

I'll analyze this cluster following the RCA skill process.

This is a benign latency cluster file used by the error-sweeper pipeline — not malware. I'll continu

`svc:` scope — proceed with normal RCA. Note the related cluster 1a719bfe within same incident.

cupixworks-api → tesla repo.

Standard Rails controller - not malware. Now let's find `create_resource` action, which is in a conc

Normal Rails code. Now let's search Datadog for logs of this trace.

The cluster indicates trace 307177218057975194 for `create_resource` with 20803ms duration at 2026-0

The `create_resource` returned 200 at 12:34:59 KST (03:34:59 UTC) — the request completed successful

Normal Rails code. Let me look at the sibling cluster (1a719bfe) - the same incident - to understand

These are unrelated. Let me look at the cluster type — this is a `latency` cluster (avg 20803ms), no

Let me look at broader context around the 03:34:37 spike. First check other latency signals — DB, co

Unrelated. Let me check the metric for request duration to see if this is an isolated slow request.

Average API is ~0.5s. The 20.8s duration is exceptionally high. Only 1 occurrence for this cluster.

Now I have enough. This is a single, isolated slow request. The controller code path is straightforw

Normal ES indexing code. Let me look at the Storagable::Resource — resource creation may trigger AWS

Standard Rails/S3 code, not malware. `create_resource` just calls `resource.save`. No obvious slow p

Enough evidence. Let me write the RCA now. Key facts:

Timestamps: 2026-07-08T03:34:37Z → 2026-07-08 12:34:37 KST.

It's a stub RCA file (not malware). I'll overwrite it now.

Now update the cluster frontmatter's `rca_status`. But looking at the cluster file, there is no `rca

RCA report written to `content/docs/incidents/50f67fca-f831-47cd-bf85-d5937de59eaa/rca.mdx`.

RCA complete. Report written to `content/docs/incidents/50f67fca-f831-47cd-bf85-d5937de59eaa/rca.mdx