Status

Auto-detected from errors/ clusters · Updated just now

Open threshold

≥ 2 clusters within 10 min · same scope

Auto-resolve

30 min with no new cluster on the scope

Grouping

External dependency → internal service fallback

Monitored components

External dependencies all operational · 3

ESelasticsearch

Operational

99.20%

S3s3

Operational

88.65%

S3s3-us-west-2

Operational

100.00%

Internal services 4 affected · 23

CUcupixworks-any-thumbnail-agent::logging_issue

Degraded

98.15%

CUcupixworks-api::external_dependency

Degraded

39.08%

CUcupixworks-api::resource_exhaustion

Degraded

65.00%

CUcupixworks-capture-refinement-arm-instance::native_crash

Degraded

98.73%

CUcupixworks-any-bimrevision-agent::configuration_error

Operational

100.00%

CUcupixworks-any-bimrevision-agent::external_dependency

Operational

94.07%

CUcupixworks-any-bimrevision-agent::logging_issue

Operational

94.08%

CUcupixworks-any-bimrevision-agent::missing_validation

Operational

91.25%

CUcupixworks-any-room-agent::unknown

Operational

95.52%

CUcupixworks-api::configuration_error

Operational

100.00%

CUcupixworks-api::race_condition

Operational

99.83%

CUcupixworks-api::timeout

Operational

100.00%

CUcupixworks-api::unknown

Operational

93.14%

CUcupixworks-capture-intelligence-agent::unknown

Operational

100.00%

CUcupixworks-capture-postprocessor-agent::missing_null_check

Operational

58.27%

CUcupixworks-capture-postprocessor-agent::missing_validation

Operational

100.00%

CUcupixworks-pano-postprocessor-instance::timeout

Operational

100.00%

CUcupixworks-pix-genie-preprocessor-instance::timeout

Operational

100.00%

CUcupixworks-worker::data_inconsistency

Operational

99.46%

CUcupixworks-worker::external_dependency

Operational

100.00%

CUcupixworks-worker::missing_null_check

Operational

100.00%

CUcupixworks-worker::race_condition

Operational

98.94%

CUcupixworks-worker::timeout

Operational

99.00%

24h agouptime per ~16 min bucketnow

Active incidents

cupixworks-api — external_dependency servicecupixworks-api Open

Started 20:38 (14h 37m ago) · ongoing 14h 37m · 8 clusters · last 1h 30m ago

08-06 20:38 8 clusters now

Related clusters → incident docs

f3667392-99ff-475a-8000-ec9ca6cc046320:38↗29acb2e1-8e37-4d54-93dc-9411a86f30b120:47↗277b768e-3b8e-46af-a570-0292385e4cfa21:06↗148967aa-11e8-4150-a2b3-91be0cd6c00921:06↗8d269564-bc1a-4e96-9479-c1a601f57cd623:00↗ca98b94f-ab34-40d2-9898-6e8c4bf965b907:21↗fe704c88-7ed9-44b1-ab09-abbc6617dc0309:46↗2e873045-935b-460e-93ec-d7cf5e55935009:46↗

Cause

cupixworks-api

Auto-resolve countdown 0m left

last cluster 90m ago · resolves 10:16 if quiet

{ "id": "2026-08-06-svc-cupixworks-api--external-dependency-1", "title": "cupixworks-api — external_dependency", "scope": { "type": "service", "name": "cupixworks-api" }, "status": "open", "started_at": "2026-08-06T11:38:43.372Z", "resolved_at": null, "cause_services": ["cupixworks-api"], "root_cause_types": ["external_dependency"], "affected_services": ["cupixworks-api"], "cluster_ids": [ "f3667392-99ff-475a-8000-ec9ca6cc0463", "29acb2e1-8e37-4d54-93dc-9411a86f30b1", "277b768e-3b8e-46af-a570-0292385e4cfa", "148967aa-11e8-4150-a2b3-91be0cd6c009", "8d269564-bc1a-4e96-9479-c1a601f57cd6", "ca98b94f-ab34-40d2-9898-6e8c4bf965b9", "fe704c88-7ed9-44b1-ab09-abbc6617dc03", "2e873045-935b-460e-93ec-d7cf5e559350" ] }

cupixworks-any-thumbnail-agent — logging_issue servicecupixworks-any-thumbnail-agent Open

Started 10:49 (27m ago) · ongoing 27m · 2 clusters · last 27m ago

10:49 2 clusters now

Related clusters → incident docs

33bd942e-bd81-4081-ac35-a4a79b980b8510:49↗483f4519-0669-4445-95c8-3a5aed6e400b10:49↗

Cause

cupixworks-any-thumbnail-agent

Auto-resolve countdown 3m left

last cluster 27m ago · resolves 11:19 if quiet

{ "id": "2026-08-07-svc-cupixworks-any-thumbnail-agent--logging-issue-2", "title": "cupixworks-any-thumbnail-agent — logging_issue", "scope": { "type": "service", "name": "cupixworks-any-thumbnail-agent" }, "status": "open", "started_at": "2026-08-07T01:49:19.096Z", "resolved_at": null, "cause_services": ["cupixworks-any-thumbnail-agent"], "root_cause_types": ["logging_issue"], "affected_services": ["cupixworks-any-thumbnail-agent"], "cluster_ids": [ "33bd942e-bd81-4081-ac35-a4a79b980b85", "483f4519-0669-4445-95c8-3a5aed6e400b" ] }

cupixworks-api — resource_exhaustion servicecupixworks-api Open

Started 10:58 (18m ago) · ongoing 18m · 2 clusters · last 9m ago

10:58 2 clusters now

Related clusters → incident docs

e9b399a2-43a9-48c0-a9d6-5500cb0b5f3110:58↗604f3732-17d4-4700-b62e-18bfe485e19511:06↗

Cause

cupixworks-api

Auto-resolve countdown 21m left

last cluster 9m ago · resolves 11:36 if quiet

{ "id": "2026-08-07-svc-cupixworks-api--resource-exhaustion-1", "title": "cupixworks-api — resource_exhaustion", "scope": { "type": "service", "name": "cupixworks-api" }, "status": "open", "started_at": "2026-08-07T01:58:13.793Z", "resolved_at": null, "cause_services": ["cupixworks-api"], "root_cause_types": ["resource_exhaustion"], "affected_services": ["cupixworks-api"], "cluster_ids": [ "e9b399a2-43a9-48c0-a9d6-5500cb0b5f31", "604f3732-17d4-4700-b62e-18bfe485e195" ] }

cupixworks-capture-refinement-arm-instance — native_crash servicecupixworks-capture-refinement-arm-instance Open

Started 11:11 (4m ago) · ongoing 4m · 2 clusters · last 4m ago

11:11 2 clusters now

Related clusters → incident docs

c6e3e4c1-8aa3-4602-a196-10384c02a92411:11↗40fdb259-470d-445a-9efd-670eb24a3ca011:11↗

Cause

cupixworks-capture-refinement-arm-instance

Auto-resolve countdown 26m left

last cluster 4m ago · resolves 11:41 if quiet

{ "id": "2026-08-07-svc-cupixworks-capture-refinement-arm-instance--native-crash-1", "title": "cupixworks-capture-refinement-arm-instance — native_crash", "scope": { "type": "service", "name": "cupixworks-capture-refinement-arm-instance" }, "status": "open", "started_at": "2026-08-07T02:11:57.003Z", "resolved_at": null, "cause_services": ["cupixworks-capture-refinement-arm-instance"], "root_cause_types": ["native_crash"], "affected_services": ["cupixworks-capture-refinement-arm-instance"], "cluster_ids": [ "c6e3e4c1-8aa3-4602-a196-10384c02a924", "40fdb259-470d-445a-9efd-670eb24a3ca0" ] }

Recently resolved

128

cupixworks-any-thumbnail-agent — logging_issue servicecupixworks-any-thumbnail-agent Resolved

10:09 → 10:09 · resolved 1h 7m ago · cause cupixworks-any-thumbnail-agent