Use a separate agent computer to inspect approved dashboards, run small read-only checks, and explain relevant logs without using your everyday browser session. Keep ordinary uptime and log tools responsible for continuous detection. In BotHearth, live investigations and scheduled routines are different: routines require a standalone model adapter, an awake host, tested notifications, limited access, and an explicit recovery plan. Prometheus: alerting overview
Let monitoring tools detect; let the agent investigate
Keep your uptime probes, metrics, log collection and paging in place. Prometheus, for example, evaluates alert rules; Alertmanager handles grouping and notification routing. Use BotHearth for the investigation alongside that system—not as its replacement. Prometheus: alerting overview
For example, ask the agent to open a service dashboard in its browser, compare the error timeline with deployment markers, inspect a narrow log window and save an evidence-based incident note. Ask for observed facts, possible explanations and missing evidence separately. First consider whether the work needs a separate computer.
Choose live observation or a recurring review
For live observation, give the agent a bounded job: inspect one service for the next 20 minutes while you supervise. BotHearth’s native Codex or Claude Code connection can run interactive tasks; it is not the current recurring-schedule setup.
For recurring review, the current path is the daemon’s routine scheduler plus a configured standalone API adapter. Each firing creates a task on an existing bot computer. The daemon, container runtime, network and model credentials must remain usable. A recurring browser review samples conditions; it is not continuous log ingestion. Choose a local host or VM that can stay available for the review.
Give it visibility, not production control
Use a dedicated monitoring account limited to the relevant dashboard and redacted logs. Enforce read-only permissions in the service itself. Do not supply deployment privileges, unrestricted SSH, database administration or cloud billing access. A prompt saying “read only” is an instruction, not an access-control boundary.
Redact before content reaches the agent’s browser or model. Remove tokens, cookies, passwords, connection strings and unnecessary personal data. Prefer approved fields such as timestamp, service, severity, error code and an opaque correlation ID over unrestricted request bodies. Treat log text as untrusted evidence, never as instructions to execute. OWASP recommends restrictive log access and excluding sensitive values. OWASP: logging guidance
Self-hosting does not keep every log excerpt offline: remote providers receive model-visible content, and notification services receive configured messages. BotHearth privacy notice
Keep healthchecks small and thresholds explicit
Have a maintainer test a lightweight, approved endpoint from a machine with network access and curl installed. Replace this illustrative address with your own:
curl --silent --show-error --fail \
--connect-timeout 3 --max-time 10 \
--output /dev/null \
--write-out 'http=%{http_code} total_s=%{time_total}\n' \
https://status.example.net/healthz
This limits connection and total request time and discards the response body. Preserve the exit code and validate the expected HTTP status separately: --fail does not reject every unexpected status, including redirects. An authentication failure means the check could not assess health, not that the service is down. curl manual
Define thresholds in the ordinary monitoring system. These are illustrative starting points, not BotHearth defaults:
| Signal | Example rule | Destination |
|---|---|---|
| Availability | Two consecutive failed one-minute probes | Existing on-call channel |
| Disk capacity | Less than 15% free for ten minutes | Operations warning queue |
| Data freshness | No sample for two expected intervals | Monitoring-failure alert |
Tune rules to the service’s normal behavior. Use sustained conditions and a deliberate recovery rule to avoid repeated alerts during brief fluctuations; Prometheus documents for and keep_firing_for for this purpose. Prometheus: alerting rules
Set up the actual BotHearth routine path
First complete a supervised dashboard review. Then configure a supported standalone adapter—such as the registered anthropic or openai_compat route—with a compatible model and its own credentials. A working native CLI login is not evidence that the standalone route works; model compatibility must be checked. Set and test task limits before scheduling; usage estimates are not provider-enforced spending caps.
The CLI reference documents the routine commands. On the daemon host, use the same configuration and data directory as the running installation. Replace the computer name and both example destinations. The webhook receiver must accept BotHearth’s JSON payload; an arbitrary chat-service webhook is not automatically compatible.
bothearth routine add \
--name service-review \
--cron '*/15 * * * *' \
--computer ops-review \
--origin https://status.example.net \
--goal 'Open https://status.example.net/overview and https://status.example.net/logs. Inspect the last 15 minutes of health data and redacted logs. Save a timestamped incident note in the workspace, separating observations, hypotheses and missing data. Do not change the service or send messages from its pages.' \
--notify 'webhook:https://alerts.example.net/bothearth'
bothearth routine run service-review
bothearth routine ls
This creates an enabled browser-only routine; --shell is intentionally absent. The cron expression uses the daemon process’s local timezone. routine run queues an enabled routine for the daemon’s next tick—it does not execute a check inside the CLI. Inspect the resulting task and saved file.
--origin is not a read-only permission. The task path uses supplied origins for both readable and writable destination sets; the monitoring account must supply the real restriction.
Remove an unwanted routine explicitly with bothearth routine rm service-review; confirm that no active task remains. Do not rely on an untested configuration change to stop scheduled work.
Separate server alerts from routine notifications
The CLI configures notifications for routine failure, not server-health thresholds. An agent can successfully finish a report describing an outage. Conversely, its review can fail while the server remains healthy. Never equate “task completed” with “service healthy.”
Use a notification destination whose delivery you have tested on your installed build. Keep independent alerting in place for delivery failures and missing reviews.
Test the chosen destination with harmless data. Keep webhook URLs and tokens out of prompts and reports. Keep notification text minimal; URL stripping is not secret redaction. Keep server threshold alerts in your existing monitoring system, and connect an independent missing-completion watchdog separately. Healthchecks.io documents this heartbeat pattern; it is not a built-in BotHearth integration. Healthchecks.io: monitoring cron jobs
Set expiry, retention and recovery before leaving
Give the pilot an owner and an end time; routines have no automatic end-date field. At that time, remove the schedule with bothearth routine rm service-review, stop any active task separately, and revoke temporary monitoring credentials. Removing the routine is not a purge of its history, task records or saved files.
BotHearth has no general transcript/screenshot retention scheduler or per-task purge command. Set an operator-managed retention policy covering task data, workspaces, audit files, browser profiles and backups. Provider and notification copies have separate retention. Stopping the daemon does not erase records.
The scheduler prevents overlapping runs of the same routine. It can fire once when less than an hour late, skips older missed runs, and disables a routine after ten consecutive failures. Do not expect every missed check to replay.
After a restart, inspect routine state and history, repair expired authentication, confirm a fresh sample, and retest delivery. Do not assume that a schedule marked “running” is still producing fresh results; investigate a missing completion before relying on the next review. BotHearth’s /healthz only reports that the daemon responds—not that reviews succeed.
Practical questions
Does this require SSH into production?
No. This browser-first workflow can use an authorized dashboard and redacted log viewer. Any additional server connection must be configured and tested separately.
Does HTTP 200 prove the whole application works?
Only the endpoint’s implemented checks. Define dependency or synthetic-transaction checks separately rather than treating one response as universal health.
Should the agent restart a failing service?
Not in this workflow. Keep diagnosis read-only; require separate authorization and an approved runbook for remediation.
Start with one supervised review
Use one service. For a suspected repair, test it in a temporary environment with separate approval. Before relying on recurrence, demonstrate a harmless failure, a recovery, a missing check and delivery to the intended recipient. Use the routine example as a pilot, and verify each part before leaving it unattended.