Omi 24h Health Report — 2026-07-26 08:47 UTC

Omi 24h Health Report

Window: 2026-07-25T08:39:49 → 2026-07-26T08:39:49 UTC | Generated: 2026-07-26 08:47 UTC | Signal filter: active (sora v1)

1. CTO Triage

SeverityIssueServiceBlast RadiusOwnerNext Action
CRITICALdesktop-backend 5xx at 90s timeoutdesktop-backend60 5xx in 15m, p99=90sBackend teamInvestigate 90s timeout — likely Cloud Run max timeout hit
CRITICALbackend-integration truncated responsesbackend-integration408 truncated in 24hBackend teamCheck long-running conversation processing timeouts
CRITICALWS auth token expiry stormbackend-listen627 auth failures in 24hMobile + BackendToken refresh timing — expired tokens causing reconnect loops
WARNINGStale conversation reconciliation failuresbackend-sync, backend-integration196 failures in 24hBackend teamFirestore query timeouts in reconcile_stale_processing
WARNING5xx spike 13x baselineCloud Run aggregate66 errors/15m vs 5.2 baselineOpsMonitor — correlate with desktop-backend timeout pattern

2. Service Health Matrix

desktop-backend
CRITICAL
backend-integration
CRITICAL
backend-sync
DEGRADED
pusher
DEGRADED
diarizer
HEALTHY
parakeet
HEALTHY
vad
HEALTHY
backend-listen
HEALTHY
agent-proxy
HEALTHY

3. Mobile Crashes (24h)

iOS Crashes
3,676
153/hr
Android Crashes
563
23/hr
Platform/DeviceVersionPackageCount
uid=<HEX> platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
uid=<HEX> platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout

4. Per-Status Latency

WARNING desktop-backend: 5xx p99 90226ms vs 2xx p99 20309ms (4.4x)
ServiceStatusCountP50 (ms)P95 (ms)P99 (ms)
backend20040,767457282,773
backend40058518811,066
backend-integration2001,362111381,281
backend-integration40094223579
backend-sync20028277614,13829,049
backend-sync40062131722
desktop-backend2002,4463795,34420,309
desktop-backend40045435157
desktop-backend5006090,01790,11690,226

5. GPU Capacity

Total GPUs
7
Allocated
6
Remaining
1
Status
WARNING
NodeGPUVRAM %Bar
diarizer-qt5h022.6%
vad-71u5023.5%
nllb-g5f605.1%
parakeet-kwxz016.1%
parakeet-9lzh016.3%
vad-brjg010.1%
vad-5jr200%

6. Load Balancer Traffic

ServiceTotal Reqs5xxP50 (ms)P99 (ms)Top Paths
backend-listen5,00001442,141/v1/action-items: 1825 reqs
/v1/health: 1679 reqs
/v1/candidates/control: 559 reqs
pusher144040,676471,910/v1/trigger/listen: 144 reqs
parakeet2801,017278,052/v2/transcribe: 24 reqs
/v1/transcribe: 2 reqs
/v3/stream: 2 reqs
vad10105081,916/v1/vad: 101 reqs
diarizer1,19003566/v2/embedding: 1190 reqs

Status Code Distribution

ServiceStatus Codes
backend-listen0: 3 | 101: 6 | 200: 4813 | 401: 3 | 403: 76 | 404: 12 | 405: 9 | 429: 78
pusher101: 144
parakeet101: 2 | 200: 26
vad200: 101
diarizer200: 1190

7. Slowest Endpoints

ServicePathP99 (ms)Requests
pusher/v1/trigger/listen471,910144
parakeet/v2/transcribe278,05224
parakeet/v1/transcribe278,0522
parakeet/v3/stream278,0522
backend-listen/v1/action-items2,1411,825
backend-listen/v1/health2,1411,679
backend-listen/v1/candidates/control2,141559
backend-listen/v1/staged-tasks/promote2,141279
backend-listen/v1/conversations2,141245
vad/v1/vad1,916101

8. Abnormal Log Patterns (20 patterns, full trace logs)

Click to expand/collapse log entries. CRITICAL patterns auto-expanded.

CRITICAL backend-listen — 627x
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, <HEX> < <HEX>
#1 | 2026-07-26T08:45:09 | backend-listen | pod: prod-omi-backend-listen-74b88657b-p2bc5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1784909009 < 1785055509
#2 | 2026-07-26T08:45:07 | backend-listen | pod: prod-omi-backend-listen-74b88657b-8rgrl
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055507
#3 | 2026-07-26T08:45:05 | backend-listen | pod: prod-omi-backend-listen-74b88657b-jrjnr
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1784909009 < 1785055505
#4 | 2026-07-26T08:45:04 | backend-listen | pod: prod-omi-backend-listen-74b88657b-tzxql
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055504
#5 | 2026-07-26T08:45:02 | backend-listen | pod: prod-omi-backend-listen-74b88657b-jl7q4
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1784909009 < 1785055502
#6 | 2026-07-26T08:45:01 | backend-listen | pod: prod-omi-backend-listen-74b88657b-czfzj
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055501
#7 | 2026-07-26T08:44:58 | backend-listen | pod: prod-omi-backend-listen-74b88657b-x6ff5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055498
#8 | 2026-07-26T08:44:54 | backend-listen | pod: prod-omi-backend-listen-74b88657b-cpgxp
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055494
#9 | 2026-07-26T08:44:51 | backend-listen | pod: prod-omi-backend-listen-74b88657b-rtcds
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055491
#10 | 2026-07-26T08:44:48 | backend-listen | pod: prod-omi-backend-listen-74b88657b-xgmnr
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785050054 < 1785055488
CRITICAL backend-integration — 408x
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#1 | 2026-07-26 08:44:37 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#2 | 2026-07-26 08:44:37 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#3 | 2026-07-26 08:44:36 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#4 | 2026-07-26 08:44:36 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#5 | 2026-07-26 08:44:27 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#6 | 2026-07-26 08:44:27 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#7 | 2026-07-26 08:44:23 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#8 | 2026-07-26 08:44:23 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#9 | 2026-07-26 08:44:21 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#10 | 2026-07-26 08:44:21 | backend-integration
Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
CRITICAL backend-integration — 127x
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#1 | 2026-07-26 08:41:11 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#2 | 2026-07-26 08:41:09 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#3 | 2026-07-26 08:41:07 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#4 | 2026-07-26 08:41:05 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#5 | 2026-07-26 08:41:05 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#6 | 2026-07-26 08:41:04 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#7 | 2026-07-26 08:41:03 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#8 | 2026-07-26 08:41:03 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#9 | 2026-07-26 08:41:03 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#10 | 2026-07-26 08:41:03 | backend-integration
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
CRITICAL backend-integration — 125x
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates
#1 | 2026-07-26 08:41:11 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#2 | 2026-07-26 08:41:09 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#3 | 2026-07-26 08:41:07 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#4 | 2026-07-26 08:41:05 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#5 | 2026-07-26 08:41:05 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#6 | 2026-07-26 08:41:04 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#7 | 2026-07-26 08:41:03 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#8 | 2026-07-26 08:41:03 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#9 | 2026-07-26 08:41:03 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#10 | 2026-07-26 08:41:03 | backend-integration
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates page = list(query.stream()) ^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
WARNING backend-sync — 72x
Traceback (most recent call last): File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations sweep = jobs_db.get_stale_processing_orphan_candidates
WARNING backend-sync — 69x
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
WARNING pusher — 61x
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid CInQQMaPJvZfaPJTml1djJKxseL2
WARNING backend-listen — 24x
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id <HEX> not found.
INFO backend-listen — 14x
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
INFO crashlytics — 13x
uid=<HEX> platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
INFO pusher — 12x
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
INFO pusher — 9x
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid TKfwEt2j5BWjeTsv8es4evli5CN2
INFO crashlytics — 8x
uid=<HEX> platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
INFO pusher — 7x
ERROR:utils.app_integrations:Plugin integration error:
INFO backend-sync — 6x
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
INFO backend-sync — 6x
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
INFO backend-listen — 5x
ERROR:routers.listen.runtime:Listen event delivery failed type=RuntimeError
INFO pusher — 5x
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid d7KLOkzMs6YeXKs0D1GZSfUWFrl1
INFO backend-sync — 5x
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=MVNZQHN42xeKJsXjfm2F5ZpK1sx1 conversation=<UUID> failure=processing_failed
INFO pusher — 5x
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid pm9k43YCzUSwXspjEvbve1aSkEf1

9. Detected Anomalies

#SeverityServiceDetails
1CRITICALvad{"time": "2026-07-26T00:40:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1722ms \u2192 20120ms (11.7x)", "severity": "CRITICAL"}
2CRITICALvad{"time": "2026-07-26T04:40:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1954ms \u2192 20300ms (10.4x)", "severity": "CRITICAL"}
3WARNINGvad{"time": "2026-07-25T22:00:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1209ms \u2192 5228ms (4.3x)", "severity": "WARNING"}
4WARNINGvad{"time": "2026-07-25T09:20:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1400ms \u2192 4200ms (3.0x)", "severity": "WARNING"}
5WARNINGparakeet{"time": "2026-07-25T12:00:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 6789ms \u2192 14200ms (2.1x)", "severity": "WARNING"}
6WARNINGparakeet{"time": "2026-07-25T14:00:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 3500ms \u2192 20981ms (6.0x)", "severity": "WARNING"}
7WARNINGparakeet{"time": "2026-07-25T15:40:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 8207ms \u2192 35888ms (4.4x)", "severity": "WARNING"}
8WARNINGvad{"time": "2026-07-25T16:20:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 2706ms \u2192 8646ms (3.2x)", "severity": "WARNING"}
9WARNINGvad{"time": "2026-07-25T17:20:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1500ms \u2192 3000ms (2.0x)", "severity": "WARNING"}
10WARNINGvad{"time": "2026-07-25T17:40:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 3000ms \u2192 8456ms (2.8x)", "severity": "WARNING"}
11WARNINGvad{"time": "2026-07-25T08:40:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 3400ms \u2192 6800ms (2.0x)", "severity": "WARNING"}
12WARNINGparakeet{"time": "2026-07-25T23:00:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 11200ms \u2192 22848ms (2.0x)", "severity": "WARNING"}
13WARNINGvad{"time": "2026-07-25T23:00:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1200ms \u2192 2641ms (2.2x)", "severity": "WARNING"}
14WARNINGparakeet{"time": "2026-07-25T10:00:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 20400ms \u2192 42361ms (2.1x)", "severity": "WARNING"}
15WARNINGparakeet{"time": "2026-07-26T01:00:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 14344ms \u2192 31768ms (2.2x)", "severity": "WARNING"}
16WARNINGvad{"time": "2026-07-26T03:40:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 1951ms \u2192 6645ms (3.4x)", "severity": "WARNING"}
17WARNINGparakeet{"time": "2026-07-26T04:20:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 10100ms \u2192 39700ms (3.9x)", "severity": "WARNING"}
18WARNINGparakeet{"time": "2026-07-25T09:20:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 3600ms \u2192 11800ms (3.3x)", "severity": "WARNING"}
19WARNINGparakeet{"time": "2026-07-26T06:20:00Z", "service": "parakeet", "type": "latency_spike", "summary": "P99 latency: 6658ms \u2192 40700ms (6.1x)", "severity": "WARNING"}
20WARNINGvad{"time": "2026-07-26T06:20:00Z", "service": "vad", "type": "latency_spike", "summary": "P99 latency: 985ms \u2192 9800ms (10.0x)", "severity": "WARNING"}

10. Infrastructure Health

CheckStatusDetails
autoscalerWARNINGFailedScheduling: 1134
cloudrun_5xxWARNINGdesktop-backend: 5915 total, 295 peak/hr; backend-sync: 1473 total, 401 peak/hr; ext-firestore-typesense-conversations-indexonwrite: 533 total, 311 peak/hr
deploy_workflowsWARNINGBuild Desktop Release Candidate: success (main, 2026-07-26T08:20); Build Desktop Release Candidate: success (main, 2026-07-26T07:25); Deploy Backend to Cloud RUN: success (main, 2026-07-26T07:04)
grafana_alertsCRITICALFIRING: Backend-integration max concurrent requests are high; FIRING: Backend-listen request per pod is high; FIRING: LLM Gateway - fallback rate elevated (ticket)
hpa_scalingOKno rescales
kubectl_contextOK
node_maintenanceOKno events
pod_lifecycleCRITICALUnhealthy: 1803; Killing: 324; Started: 323

11. Grafana Alerts

AlertStateDetails
Backend-integration max concurrent requests are high
Backend-listen request per pod is high
LLM Gateway - fallback rate elevated (ticket)
Cloud Armor denied requests per second is high (pending)
Desktop-backend - 5xx P99 latency indicates timeout cascade
Backend-listen - 4XX error rate is high (pending)
VAD request latency is high (pending)
Backend - 5xx P99 latency is critically high
Backend - 5xx P99 latency is critically high

12. Signal Detection

SignalCurrentBaselineRatioStatus
5xx Rate665.212.7xALERT
Anthropic Errors000.0xOK
Openai Errors01.40.0xOK
Deepgram Errors819.50.4xOK
LB backend-listen 5xx%000.0xOK
LB vad 5xx%00.010.0xOK
LB pusher 5xx%000.0xOK
LB diarizer 5xx%000.0xOK
LB parakeet 5xx%00.010.0xOK
GPU Capacity170.0xWARNING
Cost Anomalies000.0xOK

Active Alerts

HIGH 5xx spike — 66 errors in 15m (13x above baseline)
WARNING desktop-backend: 5xx p99 90226ms vs 2xx p99 20309ms (4.4x)
WARNING GPU capacity low: 6/7 allocated, 1 remaining

13. Detection Gap Coverage

GapStatusDetails
Per-status latencyCOVERED14 service×status rows, 1 alerts
Route tier classificationCOVEREDUser-facing services prioritized in triage
GPU capacity monitoringCOVERED7 GPUs, 6 allocated, VRAM per-node tracked
Signal noise filterOKsora v1 — 9 rules, ~62-79% noise reduction on backend-listen
Full trace logsCOVERED20/20 patterns with 186 total log entries

14. Time Series Data

Report metadata: Window 2026-07-25T08:39:49 → 2026-07-26T08:39:49 UTC | Generated 2026-07-26 08:47 UTC
Sources: beast omi ops health-report (24h, drill-down), signal detect (per-status, gpu), health (fresh)
Log sources: BQ cloud_run_logs (stderr, stdout, varlog), gcloud logging read (k8s_container), Firebase Crashlytics
Signal filter: active (sora v1, 9 rules, sha256 fix deployed) | Patterns: 20/20 covered | Total log entries: 186
- mon, by AI for @beastoin