Omi 24h Health Report — Full Trace Edition
Generated: 2026-07-26T06:28:48.305151637Z | Window: 24h | Each log pattern includes up to 10 full entries for root cause tracing
CTO Triage Table
| P |
Signal |
Tier |
Impact |
Code Entry Point |
| P0 |
GPU oversubscribed — 7 on 5 available |
critical |
Transcription/diarization degraded |
GKE autoscaler, parakeet-pool HPA |
| P1 |
Desktop-backend 5xx P99 90s timeout cascade |
user_facing |
Desktop users frozen for 90s |
Desktop-backend upstream call chain |
| P1 |
iOS crashes 4,080 (170/hr) |
critical |
6.6x Android rate |
Crashlytics → http_pool_manager.dart |
| P2 |
WebSocket auth: 587x Token expired + cert errors |
user_facing |
Disconnected from transcription |
utils/other/endpoints.py:WebSocket auth |
| P2 |
Conversation finalization: 104x stale reconciliation |
user_facing |
Orphaned conversations |
services/conversation_finalization.py:135 |
| P2 |
Sync pipeline: 30x upstream_error retrying |
user_facing |
Memory sync delays |
utils/sync/pipeline.py:sync_transcription_job |
Service Health Matrix
| Service |
Status |
Issues |
| desktop-backend |
CRITICAL |
No issues |
| backend-integration |
CRITICAL |
No issues |
| pusher |
DEGRADED |
No issues |
| backend-sync |
DEGRADED |
No issues |
| parakeet |
HEALTHY |
No issues |
| agent-proxy |
HEALTHY |
No issues |
| diarizer |
HEALTHY |
No issues |
| vad |
HEALTHY |
No issues |
| backend-listen |
HEALTHY |
No issues |
Mobile Crashes (24h)
| Platform |
Total |
Per Hour |
| iOS |
4,080 |
170/hr |
| Android |
618 |
26/hr |
Per-Status Latency (Gap 1)
| Service |
Status |
Count |
P50 |
P99 |
Analysis |
| desktop-backend | 2xx | 2,524 | 336ms | 17.7s | Normal |
| 5xx | 50 | 90.0s | 90.1s | Gateway timeout — 100% are 90s waits (5.1x vs 2xx) |
| backend | 2xx | 42,786 | 39ms | 2.9s | Normal |
| backend-sync | 2xx | 848 | 1.2s | 54.2s | Normal |
| backend-integration | 2xx | 1,308 | 10ms | 799ms | Normal |
GPU Capacity (Gap 3)
| Metric | Value | Status |
|---|
| GPU Allocation | 7 requested / 5 available (-2) | CRITICAL |
| FailedScheduling (24h) | 1,168 | CRITICAL |
| VRAM (max) | 23.5% | OK |
| Node (pool) | VRAM % |
|---|
| vad-pool-v2-47a302d1-71u5 | 23.5% |
| diarizer-pool-037ee134-qt5h | 22.6% |
| parakeet-pool-c1f29adf-9lzh | 16.3% |
| parakeet-pool-c1f29adf-kwxz | 16.1% |
| nllb-translation-poo-bf276826-g5f6 | 5.1% |
Top Error Endpoints
| Service | Path | Count | Tier |
|---|
| backend-integration | /v1/mcp/sse | 34,121 | internal |
| backend-integration | /v1/mcp/conversations?limit=3 | 29,413 | internal |
| backend-sync | /v2/sync-local-files | 26,500 | user_facing |
| pusher | https://audio.soundbrain.cc/webhook | 22,719 | known_noise |
| pusher | https://omi-assist-api-prod.dakom.workers.dev/omi | 19,747 | known_noise |
| pusher | https://omi.supermemory.ai/webhook | 10,390 | known_noise |
| backend-sync | /v2/audio-merge-jobs/run | 6,707 | user_facing |
| backend-sync | /v2/sync-jobs/run | 3,532 | user_facing |
| backend-sync | /v2/sync-local-files | 2,426 | user_facing |
| backend-sync | /v1/conversation-finalization-jobs/run | 2,092 | user_facing |
Slowest Endpoints
| Service | Path | Avg Latency | Count |
|---|
| pusher | /v1/trigger/listen | 17.7s | 47,482 |
| parakeet | /v2/transcribe | 3.6s | 90,840 |
| parakeet | /v3/stream | 12.0s | 3,856 |
| parakeet | /v1/transcribe | 298ms | 1,521 |
Abnormal Log Patterns — Full Trace (30 patterns, up to 10 logs each)
Each pattern shows up to 10 raw log entries with timestamps, pod names, and full text for code tracing. Click "Show logs" to expand.
#1 [CRITICAL]
backend-listen
— 587x in 24h
utils.other.endpoints
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, <HEX> < <HEX>
Show 10 logs ▼
[1] 2026-07-26T06:49:39
pod:listen-74b88657b-l85sg
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048579
[2] 2026-07-26T06:49:36
pod:listen-74b88657b-l85sg
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048576
[3] 2026-07-26T06:49:33
pod:listen-74b88657b-p2bc5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048573
[4] 2026-07-26T06:49:30
pod:listen-74b88657b-wpkzh
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048570
[5] 2026-07-26T06:49:26
pod:listen-74b88657b-jl7q4
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048566
[6] 2026-07-26T06:49:23
pod:listen-74b88657b-p2bc5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048563
[7] 2026-07-26T06:49:20
pod:listen-74b88657b-pcbh6
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048560
[8] 2026-07-26T06:49:17
pod:listen-74b88657b-x6ff5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048557
[9] 2026-07-26T06:49:13
pod:listen-74b88657b-x6ff5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1785040759 < 1785048553
[10] 2026-07-26T06:49:11
pod:listen-74b88657b-wpkzh
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Token expired, 1784983012 < 1785048551
#2 [CRITICAL]
backend-integration
— 341x in 24h
Truncated response body. Usually implies that the request timed out or the application exited before the response was fi
Show 3 logs ▼
[1]
[09:28:44] Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
[2]
[09:28:41] Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
[3]
[09:28:40] Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.
#3 [CRITICAL]
backend-integration
— 104x in 24h
services.conversation_finalization
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
Show 10 logs ▼
[1] 2026-07-26 06:44:31
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[2] 2026-07-26 06:44:29
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[3] 2026-07-26 06:44:27
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[4] 2026-07-26 06:44:27
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[5] 2026-07-26 06:44:25
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[6] 2026-07-26 06:44:24
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[7] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[8] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[9] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[10] 2026-07-26 06:44:19
pod:backend-integration
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#4 [CRITICAL]
backend-integration
— 104x in 24h
/app/services/conversation_finalization.py:135
Traceback (most recent call last):
Show 10 logs ▼
[1] 2026-07-26 06:44:31
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[2] 2026-07-26 06:44:29
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[3] 2026-07-26 06:44:27
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[4] 2026-07-26 06:44:27
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[5] 2026-07-26 06:44:25
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[6] 2026-07-26 06:44:24
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[7] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[8] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[9] 2026-07-26 06:44:22
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[10] 2026-07-26 06:44:19
pod:backend-integration
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#5 [WARNING]
backend-sync
— 83x in 24h
/app/services/conversation_finalization.py:135
Traceback (most recent call last):
Show 10 logs ▼
[1] 2026-07-26 06:46:14
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[2] 2026-07-26 06:46:03
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[3] 2026-07-26 06:45:55
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[4] 2026-07-26 06:45:55
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[5] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[6] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[7] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[8] 2026-07-26 06:44:46
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[9] 2026-07-26 06:44:41
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
[10] 2026-07-26 06:44:40
pod:backend-sync
commit:8fba2060
Traceback (most recent call last):
File "/app/services/conversation_finalization.py", line 135, in reconcile_stale_processing_conversations
sweep = jobs_db.get_stale_processing_orphan_candidates(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/database/conversation_finalization_jobs.py", line 1055, in get_stale_processing_orphan_candidates
page = list(query.stream())
^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/google/cloud/fir
#6 [WARNING]
backend-sync
— 83x in 24h
services.conversation_finalization
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
Show 10 logs ▼
[1] 2026-07-26 06:46:14
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[2] 2026-07-26 06:46:03
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[3] 2026-07-26 06:45:55
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[4] 2026-07-26 06:45:55
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[5] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[6] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[7] 2026-07-26 06:45:00
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[8] 2026-07-26 06:44:46
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[9] 2026-07-26 06:44:41
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
[10] 2026-07-26 06:44:40
pod:backend-sync
commit:8fba2060
ERROR:services.conversation_finalization:stale processing conversation reconciliation query failed
#7 [WARNING]
pusher
— 40x in 24h
utils.app_integrations
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9
Show 10 logs ▼
[1] 2026-07-26T06:50:18
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[2] 2026-07-26T06:35:54
pod:pusher-58b94bc575-vcc5z
ERROR:utils.app_integrations:App 01KCJ0ZQ0HV4CDGDGH1Y6TCXA1 is not proactive_notification or data invalid PeIR6QesVnM9m9EjSQF9PI25H0r1
[3] 2026-07-26T06:30:12
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[4] 2026-07-26T06:30:07
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[5] 2026-07-26T06:27:36
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
[6] 2026-07-26T06:27:30
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
[7] 2026-07-26T06:27:30
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
[8] 2026-07-26T06:27:27
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
[9] 2026-07-26T06:27:25
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
[10] 2026-07-26T06:27:22
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
#8 [WARNING]
backend-sync
— 30x in 24h
utils.sync.pipeline
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=upstream_error status=retrying provider=unknown model=unk
Show 10 logs ▼
[1] 2026-07-26 06:39:06
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[2] 2026-07-26 06:39:04
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[3] 2026-07-26 06:39:01
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[4] 2026-07-26 06:38:57
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[5] 2026-07-26 06:37:46
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[6] 2026-07-26 06:30:20
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[7] 2026-07-26 06:30:18
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[8] 2026-07-26 06:30:16
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[9] 2026-07-26 06:28:47
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[10] 2026-07-26 06:28:39
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
#9 [INFO]
backend-listen
— 19x in 24h
deepgram.clients.common.v1.abstract_sync_websocket
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: server
Show 10 logs ▼
[1] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out
[2] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
WebSocketException in AbstractSyncWebSocketClient.start: timed out
[3] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[4] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[5] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[6] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[7] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[8] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[9] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[10] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
#10 [INFO]
backend-listen
— 19x in 24h
utils.stt.streaming
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
Show 10 logs ▼
[1] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[2] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[3] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[4] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[5] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[6] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[7] 2026-07-25T09:33:51
pod:listen-684bc45446-kx2f9
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[8] 2026-07-25T09:33:45
pod:listen-684bc45446-kx2f9
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[9] 2026-07-25T09:33:35
pod:listen-684bc45446-f8cjb
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
[10] 2026-07-25T09:33:28
pod:listen-684bc45446-f8cjb
ERROR:utils.stt.streaming:Deepgram connection start() returned False — connection not established
#11 [INFO]
backend-listen
— 19x in 24h
deepgram.clients.common.v1.abstract_sync_websocket
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: server rejec
Show 10 logs ▼
[1] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: timed out
[2] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
WebSocketException in ListenWebSocketClient.start: timed out
[3] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: timed out during handshake
[4] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
WebSocketException in ListenWebSocketClient.start: timed out during handshake
[5] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: timed out during handshake
[6] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
WebSocketException in ListenWebSocketClient.start: timed out during handshake
[7] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: timed out during handshake
[8] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
WebSocketException in ListenWebSocketClient.start: timed out during handshake
[9] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: timed out during handshake
[10] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
WebSocketException in ListenWebSocketClient.start: timed out during handshake
#12 [INFO]
backend-listen
— 19x in 24h
WebSocketException in ListenWebSocketClient.start
WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
Show 10 logs ▼
[1] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[2] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[3] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[4] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
WebSocketException in AbstractSyncWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[5] 2026-07-25T09:33:51
pod:listen-684bc45446-kx2f9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[6] 2026-07-25T09:33:51
pod:listen-684bc45446-kx2f9
WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[7] 2026-07-25T09:33:51
pod:listen-684bc45446-kx2f9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[8] 2026-07-25T09:33:51
pod:listen-684bc45446-kx2f9
WebSocketException in AbstractSyncWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[9] 2026-07-25T09:33:45
pod:listen-684bc45446-kx2f9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
[10] 2026-07-25T09:33:45
pod:listen-684bc45446-kx2f9
WebSocketException in ListenWebSocketClient.start: server rejected WebSocket connection: HTTP 401
#13 [INFO]
backend-listen
— 19x in 24h
WebSocketException in AbstractSyncWebSocketClient.start
WebSocketException in AbstractSyncWebSocketClient.start: server rejected WebSocket connection: HTTP 401
Show 10 logs ▼
[1] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out
[2] 2026-07-25T18:43:58
pod:listen-74b88657b-fww5p
WebSocketException in AbstractSyncWebSocketClient.start: timed out
[3] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[4] 2026-07-25T18:12:10
pod:listen-74b88657b-hwkt9
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[5] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[6] 2026-07-25T16:21:48
pod:listen-74b88657b-5rtk5
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[7] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[8] 2026-07-25T16:21:48
pod:listen-74b88657b-ssqmn
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[9] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
ERROR:deepgram.clients.common.v1.abstract_sync_websocket:WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
[10] 2026-07-25T10:54:12
pod:listen-684bc45446-x9xbp
WebSocketException in AbstractSyncWebSocketClient.start: timed out during handshake
#14 [INFO]
backend-sync
— 17x in 24h
utils.sync.pipeline
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknow
Show 10 logs ▼
[1] 2026-07-26 06:39:06
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[2] 2026-07-26 06:39:04
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[3] 2026-07-26 06:39:01
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[4] 2026-07-26 06:38:57
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[5] 2026-07-26 06:37:46
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[6] 2026-07-26 06:30:20
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[7] 2026-07-26 06:30:18
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[8] 2026-07-26 06:30:16
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[9] 2026-07-26 06:28:47
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[10] 2026-07-26 06:28:39
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
#15 [INFO]
backend-listen
— 17x in 24h
utils.stt.streaming
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
Show 10 logs ▼
[1] 2026-07-26T02:21:40
pod:listen-74b88657b-jl7q4
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[2] 2026-07-26T02:20:35
pod:listen-74b88657b-ctv8l
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[3] 2026-07-26T02:19:59
pod:listen-74b88657b-l85sg
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[4] 2026-07-26T02:19:38
pod:listen-74b88657b-wpkzh
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[5] 2026-07-26T02:19:18
pod:listen-74b88657b-9gpwm
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[6] 2026-07-26T02:18:58
pod:listen-74b88657b-g8m75
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[7] 2026-07-26T02:18:38
pod:listen-74b88657b-l85sg
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[8] 2026-07-26T02:18:21
pod:listen-74b88657b-9p669
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[9] 2026-07-26T02:18:02
pod:listen-74b88657b-9p669
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
[10] 2026-07-26T02:17:44
pod:listen-74b88657b-9p669
ERROR:utils.stt.streaming:Parakeet WS recv error: no close frame received or sent
#16 [INFO]
crashlytics
— 16x in 24h
uid=<HEX> platform=ios device=iPhone17,2 version=1.0.544 package
uid=<HEX> platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManag
Show 3 logs ▼
[1]
[09:28:43] uid=4F9AD7146B36C052321F65A8C57F0F88CA74720C25B0C4545EB71C3A26BB3FD4 platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[2]
[09:27:28] uid=4F9AD7146B36C052321F65A8C57F0F88CA74720C25B0C4545EB71C3A26BB3FD4 platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[3]
[12:26:45] uid=4F9AD7146B36C052321F65A8C57F0F88CA74720C25B0C4545EB71C3A26BB3FD4 platform=ios device=iPhone17,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
#17 [INFO]
crashlytics
— 13x in 24h
uid=<HEX> platform=ios device=iPhone18,2 version=1.0.544 package
uid=<HEX> platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManag
Show 3 logs ▼
[1]
[09:28:12] uid=6E3246461A5873FCBC367361534A14F48469D4DFDB4B26EC71C45DC7240A21C4 platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[2]
[09:26:37] uid=6E3246461A5873FCBC367361534A14F48469D4DFDB4B26EC71C45DC7240A21C4 platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[3]
[12:28:38] uid=6E3246461A5873FCBC367361534A14F48469D4DFDB4B26EC71C45DC7240A21C4 platform=ios device=iPhone18,2 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
#18 [INFO]
pusher
— 11x in 24h
utils.app_integrations
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data in
Show 10 logs ▼
[1] 2026-07-26T06:38:29
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[2] 2026-07-26T06:38:20
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[3] 2026-07-26T06:38:17
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[4] 2026-07-26T06:37:14
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[5] 2026-07-26T06:30:12
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[6] 2026-07-26T06:30:12
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[7] 2026-07-26T06:30:10
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[8] 2026-07-26T06:30:07
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[9] 2026-07-26T06:27:37
pod:pusher-58b94bc575-zcbbz
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[10] 2026-07-26T06:27:30
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
#19 [INFO]
backend-sync
— 10x in 24h
utils.sync.pipeline
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=upstream_error status=failed provider=unknown model=unkno
Show 10 logs ▼
[1] 2026-07-26 06:39:06
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[2] 2026-07-26 06:39:04
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[3] 2026-07-26 06:39:01
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[4] 2026-07-26 06:38:57
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[5] 2026-07-26 06:37:46
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[6] 2026-07-26 06:30:20
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[7] 2026-07-26 06:30:18
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[8] 2026-07-26 06:30:16
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[9] 2026-07-26 06:28:47
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
[10] 2026-07-26 06:28:39
pod:backend-sync
commit:8fba2060
ERROR:utils.sync.pipeline:event=sync_transcription_job outcome=invalid_input status=failed provider=unknown model=unknown lane=backfill reason_code=sync_invalid_audio
#20 [INFO]
backend-sync
— 10x in 24h
routers.sync
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 excep
Show 10 logs ▼
[1] 2026-07-25 10:51:51
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[2] 2026-07-25 10:51:05
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[3] 2026-07-25 10:50:18
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[4] 2026-07-25 10:50:15
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[5] 2026-07-25 10:50:13
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[6] 2026-07-25 10:50:06
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[7] 2026-07-25 10:50:04
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[8] 2026-07-25 10:50:02
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[9] 2026-07-25 10:50:02
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
[10] 2026-07-25 10:49:55
pod:backend-sync
commit:462adc7d
ERROR:routers.sync:event=sync_transcription_job outcome=upstream_error status=failed_final lane=backfill attempt=5 exception_type=RuntimeError
#21 [INFO]
pusher
— 9x in 24h
utils.webhooks
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
Show 10 logs ▼
[1] 2026-07-26T06:41:37
pod:pusher-58b94bc575-tq72z
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[2] 2026-07-26T06:39:13
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[3] 2026-07-26T06:38:58
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[4] 2026-07-26T06:38:43
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[5] 2026-07-26T06:38:17
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[6] 2026-07-26T06:38:15
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[7] 2026-07-26T06:37:34
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[8] 2026-07-26T06:37:32
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[9] 2026-07-26T06:37:29
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
[10] 2026-07-26T06:36:55
pod:pusher-58b94bc575-h44z8
ERROR:utils.webhooks:Error sending audio bytes to developer webhook:
#22 [INFO]
backend-sync
— 6x in 24h
utils.llm.conversation_processing
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
Show 10 logs ▼
[1] 2026-07-26 06:37:18
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[2] 2026-07-26 06:36:10
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[3] 2026-07-26 06:35:54
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[4] 2026-07-26 06:25:52
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[5] 2026-07-26 06:05:19
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: {"discard": True}
[6] 2026-07-26 06:02:56
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[7] 2026-07-26 05:52:04
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[8] 2026-07-26 05:51:09
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[9] 2026-07-26 05:44:49
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
[10] 2026-07-26 05:44:01
pod:backend-sync
commit:8fba2060
ERROR:utils.llm.conversation_processing:Error determining memory discard: Invalid json output: discard = False
#23 [INFO]
pusher
— 6x in 24h
utils.app_integrations
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid CInQQMaPJvZfaP
Show 10 logs ▼
[1] 2026-07-26T06:50:41
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[2] 2026-07-26T06:50:40
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[3] 2026-07-26T06:50:39
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[4] 2026-07-26T06:50:39
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[5] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[6] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[7] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[8] 2026-07-26T06:50:37
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[9] 2026-07-26T01:16:45
pod:pusher-58b94bc575-4b7qv
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid CInQQMaPJvZfaPJTml1djJKxseL2
[10] 2026-07-26T01:16:42
pod:pusher-58b94bc575-4b7qv
ERROR:utils.app_integrations:App 01K8FRZA3XV2T4N1VD9CMTW8HV is not proactive_notification or data invalid CInQQMaPJvZfaPJTml1djJKxseL2
#24 [INFO]
backend-listen
— 6x in 24h
utils.stt.streaming
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
Show 10 logs ▼
[1] 2026-07-25T09:33:59
pod:listen-684bc45446-kx2f9
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[2] 2026-07-25T09:33:35
pod:listen-684bc45446-f8cjb
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[3] 2026-07-25T09:33:12
pod:listen-684bc45446-8s85x
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[4] 2026-07-25T09:32:48
pod:listen-684bc45446-lz7x7
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[5] 2026-07-25T09:32:25
pod:listen-684bc45446-lz7x7
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[6] 2026-07-25T09:32:02
pod:listen-684bc45446-8s85x
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[7] 2026-07-25T09:31:38
pod:listen-684bc45446-hpphj
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[8] 2026-07-25T09:31:15
pod:listen-684bc45446-fmd9j
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[9] 2026-07-25T09:30:52
pod:listen-684bc45446-mrnpg
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
[10] 2026-07-25T09:30:28
pod:listen-684bc45446-f8cjb
ERROR:utils.stt.streaming:Deepgram start() returned False on all 3 attempts — giving up
#25 [INFO]
backend-listen
— 5x in 24h
utils.other.endpoints
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id <HEX> not found.
Show 10 logs ▼
[1] 2026-07-26T06:50:32
pod:listen-74b88657b-p2bc5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[2] 2026-07-26T06:50:29
pod:listen-74b88657b-pcbh6
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[3] 2026-07-26T06:50:25
pod:listen-74b88657b-p2bc5
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[4] 2026-07-26T06:50:22
pod:listen-74b88657b-fww5p
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[5] 2026-07-26T06:50:19
pod:listen-74b88657b-wpkzh
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[6] 2026-07-26T06:50:16
pod:listen-74b88657b-skjld
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[7] 2026-07-26T06:50:12
pod:listen-74b88657b-9p669
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[8] 2026-07-26T06:50:09
pod:listen-74b88657b-l85sg
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[9] 2026-07-26T06:50:06
pod:listen-74b88657b-fww5p
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
[10] 2026-07-26T06:50:02
pod:listen-74b88657b-wbs8p
ERROR:utils.other.endpoints:WebSocket auth failed: code=4001 error=Certificate for key id 65cb3f020caf7b2a9e86ead901d89f4812cbacbc not found.
#26 [INFO]
pusher
— 5x in 24h
utils.app_integrations
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 9l9OUOTgDnT3ZF
Show 10 logs ▼
[1] 2026-07-26T06:50:43
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[2] 2026-07-26T06:50:42
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[3] 2026-07-26T06:50:41
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[4] 2026-07-26T06:50:40
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[5] 2026-07-26T06:50:39
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[6] 2026-07-26T06:50:39
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[7] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[8] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[9] 2026-07-26T06:50:38
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
[10] 2026-07-26T06:50:37
pod:pusher-58b94bc575-tq72z
ERROR:utils.app_integrations:App 01KF1M5S7Y9QG9V6758W60TXFA is not proactive_notification or data invalid 5kb5okd5ctdEjFLIAxS3fDUMIPu1
#27 [INFO]
backend-sync
— 5x in 24h
utils.conversations.finalizer
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=s14He2nS6ic2d1HIwKTbbLn81e02 conversa
Show 10 logs ▼
[1] 2026-07-26 06:39:33
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[2] 2026-07-26 06:38:42
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[3] 2026-07-26 06:38:14
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[4] 2026-07-26 06:37:57
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[5] 2026-07-26 06:37:35
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[6] 2026-07-26 06:29:24
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=o5BAaYc2j7Z5oh5KaFjj1jchss62 conversation=e15304c6-2961-41fd-887a-c3b2a94ed70e failure=processing_failed
[7] 2026-07-26 06:17:22
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=MBCS9r3MamPDLFzBKVJnKSPfiGj1 conversation=5dab8765-1bd5-4b8c-9395-51a1c5676453 failure=processing_failed
[8] 2026-07-26 06:00:37
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
[9] 2026-07-26 06:00:29
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
[10] 2026-07-26 06:00:21
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
#28 [INFO]
crashlytics
— 5x in 24h
uid=<HEX> platform=ios device=iPhone8,1 version=1.0.544 package
uid=<HEX> platform=ios device=iPhone8,1 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManage
Show 3 logs ▼
[1]
[06:27:08] uid=BED63995211659C3CA4DD7043BD8C479EBCE465948C970A27D983B68DB6AEB18 platform=ios device=iPhone8,1 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[2]
[06:26:34] uid=BED63995211659C3CA4DD7043BD8C479EBCE465948C970A27D983B68DB6AEB18 platform=ios device=iPhone8,1 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
[3]
[06:26:14] uid=BED63995211659C3CA4DD7043BD8C479EBCE465948C970A27D983B68DB6AEB18 platform=ios device=iPhone8,1 version=1.0.544 package:omi/backend/http/http_pool_manager.dart - HttpPoolManager._executeWithRetry — FlutterError - TimeoutException: Request timeout
#29 [INFO]
backend-sync
— 5x in 24h
utils.conversations.finalizer
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversa
Show 10 logs ▼
[1] 2026-07-26 06:39:33
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[2] 2026-07-26 06:38:42
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[3] 2026-07-26 06:38:14
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[4] 2026-07-26 06:37:57
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[5] 2026-07-26 06:37:35
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=ef9089d4-2722-4341-8231-1014a1754f3d failure=processing_failed
[6] 2026-07-26 06:29:24
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=o5BAaYc2j7Z5oh5KaFjj1jchss62 conversation=e15304c6-2961-41fd-887a-c3b2a94ed70e failure=processing_failed
[7] 2026-07-26 06:17:22
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=MBCS9r3MamPDLFzBKVJnKSPfiGj1 conversation=5dab8765-1bd5-4b8c-9395-51a1c5676453 failure=processing_failed
[8] 2026-07-26 06:00:37
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
[9] 2026-07-26 06:00:29
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
[10] 2026-07-26 06:00:21
pod:backend-sync
commit:8fba2060
ERROR:utils.conversations.finalizer:persisted conversation finalization failed uid=KsDfQaxJBAepV8W3FOB1HxtFqTx1 conversation=999cab55-5597-44fb-a083-01fe3a562318 failure=processing_failed
#30 [INFO]
pusher
— 5x in 24h
utils.app_integrations
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data in
Show 10 logs ▼
[1] 2026-07-26T06:38:29
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[2] 2026-07-26T06:38:20
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[3] 2026-07-26T06:38:17
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[4] 2026-07-26T06:37:14
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[5] 2026-07-26T06:30:12
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[6] 2026-07-26T06:30:12
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[7] 2026-07-26T06:30:10
pod:pusher-58b94bc575-rgp69
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[8] 2026-07-26T06:30:07
pod:pusher-58b94bc575-rgp69
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid 6PltTw1PrranTK4pXa30gP7OP8u2
[9] 2026-07-26T06:27:37
pod:pusher-58b94bc575-zcbbz
INFO:utils.app_integrations:trigger_realtime_integrations supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ status: 500 results: Internal Server Error
[10] 2026-07-26T06:27:30
pod:pusher-58b94bc575-zcbbz
ERROR:utils.app_integrations:App supermemory-for-omi-01JE0442S4S20TRAHVYAXGQ2RQ is not proactive_notification or data invalid Nn38QjrnfQZAV9tilbsFoKh5xTl1
Infrastructure Health
| Check | Status | Details |
|---|
| Grafana Alerts | CRITICAL | 73 alert instances, 7 firing, 3 pending |
| Cloud Run 5xx | WARNING | 9390 total 5xx across 10 services |
| Pod Lifecycle | CRITICAL | 3095 events (Unhealthy: 1817, Started: 331, Created: 331) |
| Autoscaler | WARNING | 1168 events (1168 FailedScheduling, 0 scale failures) |
| Deploys | WARNING | 15 deploy runs, 2 failed |
| Node Maint | OK | no events |
| HPA | OK | no rescales |
Signal Tier Classification (Gap 2)
| Tier | Weight | Active Signals |
|---|
| critical_user_facing | 3.0x | GPU exhaustion, transcription pipeline |
| user_facing | 1.0x | Desktop timeouts, sync 5xx, WebSocket auth, iOS crashes |
| internal | 0.3x | MCP SSE 401s, Cloud Armor |
| known_noise | 0.1x | Pusher webhooks, firmware 404s |
Detection Gap Coverage
| Gap | Status | Coverage |
|---|
| 1. Per-status latency | LIVE | beast --per-status + Grafana 5 panels + 2 alerts |
| 2. Route tiers | LIVE | beast --tier-filter + 19 route rules |
| 3. GPU capacity | LIVE | beast --gpu + Grafana 10 panels + 3 alerts |
Sources: beast omi ops health-report --since 24h --drill-down | beast omi ops signal detect --per-status --gpu |
Cloud Logging (GKE k8s_container) + BQ (Cloud Run stderr) for full trace logs
Patterns with full logs: 29/30 |
Grafana dashboards: Per-Status-Code Latency + GPU Capacity & Health | 5 alert rules
- mon, by AI for @beastoin