Monitoring

PocketCmd Server runs metric commands on the machine it is installed on, keeps the samples in ~/.pocketcmd/metrics.db and evaluates alert rules against every new sample. The app shows the metrics on dashboard cards and receives alerts as push notifications. Server metrics and alerts require PocketCmd Pro in the app.

The server has no built-in metrics. Every metric is a shell command that a team member proposed and someone with the Add monitoring commands permission approved. Only the owner holds that permission automatically. Admins and members hold it only when the owner grants it, and only the owner can grant or revoke it. A server with no approved commands runs nothing and stores nothing, and the app’s card editor lists no server metrics for it until a command is approved.

Dashboard cards can also run a shell command over SSH from the phone. Those cards never involve the server and cannot carry alerts. See Dashboards.

Custom command metrics

A metric runs a shell command on the server and records the first number it prints. Because this executes code on the host, every command goes through an approval step:

1. A team member proposes the command in the app with Request metric, on the team’s Metric requests page or in Your metric requests. It is stored as a pending request and nothing runs. Members see their own requests; members with Add monitoring commands see every request.
2. Someone with Add monitoring commands reviews it under Metric requests (it also appears in Needs attention) and approves it with Approve & Run or declines it. The approval dialog always shows the full command and its interval. On the host, pcmd requests approve does the same. Their own commands go in with Add metric, which files and approves them in one step.
3. The approved command runs on its own interval, or on the server’s default interval if the request does not set one. Its metric ID is custom:<request-id>.

Adding a command that runs, or changing a command’s shell code, needs Add monitoring commands, also when it arrives through a monitoring_config sync record; changing a command also needs Edit monitoring commands. Granting Add monitoring commands lets a member run code on the host. A requester can withdraw a request while it is pending. Deleting an approved request stops its command; that needs Remove monitoring commands.

Execution rules:

Rule	Value
Shell	sh -c as the server’s user. Under systemd it runs inside the service sandbox: read-only file system except ~/.pocketcmd, private /tmp.
Environment	Only PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, HOME and LANG.
Timeout	10 seconds, after which the whole process group is killed.
Output	The first 64 KB of stdout is read. A non-zero exit records no sample.
Parsing	The first run of digits, with an optional decimal part. A leading minus sign is not part of the match, so negative values are recorded as positive.
Length	Commands are limited to 4096 characters and are not otherwise validated.

Command metrics have no unit.

# CPU busy percentage over one second (Linux, sysstat)
mpstat 1 1 | awk '/Average/ {print 100 - $NF}'

# Disk usage of / in percent
df --output=pcent / | tail -n 1 | tr -dc '0-9'

# Established connections to port 22
ss -tn state established '( sport = :22 )' | tail -n +2 | wc -l

# Running Docker containers
docker ps -q | wc -l

Monitoring configuration

The approved commands and the alert rules set on them live in one team-wide record, monitoring_config. Approving a request adds its command to the record on the server. The app adds, changes and removes alert rules on commands that already exist; a dashboard card never creates, enables or changes anything on the server.

The server applies each change in the same database transaction that accepts it and rebuilds its collection timers at once, without a restart. A record that fails validation (an unknown operator, duplicate IDs, an interval out of range) is rejected as a whole, and the previous configuration stays in effect.

monitoring_config is the one sync record the server can read: it is stored in plaintext because the scheduler runs it. Do not put secrets in metric commands or labels.

The server checks each change against the writer’s permissions: Add alert rules, Edit alert rules and Remove alert rules for rules, Edit monitoring commands for intervals and retention. Admins hold every permission except Add monitoring commands. Reading metrics needs View metrics, and alert notifications go only to members with Receive alerts.

Intervals and retention

Commands run every 30 seconds unless their request sets another interval, and samples are kept for 7 days. The server accepts intervals from 5 seconds to 6 hours and retention from 1 to 30 days. Samples older than the retention are deleted when the server starts and every 24 hours after that.

Alert rules

Alert rules are set on server metrics in the app (Alerts). Each rule has a threshold and a direction, stored as greater (value strictly above the threshold) or less (strictly below). Rules have no severity level.

Every new sample is checked against the metric’s rules:

• The first breach sends a notification, provided the rule has not notified in the last 10 minutes.
• While the breach continues, the rule notifies again at most once an hour.
• When the value returns to the normal range, the rule re-arms. The 10-minute floor still applies, so a metric that flaps around its threshold notifies at most once every 10 minutes.

The server keeps this state across restarts, so restarting it neither repeats nor loses a notification. The notification title names the metric and the server’s hostname. The body gives the value and the threshold, for example Containers at 12.0 (threshold: 10.0).

The server does not store a history of fired alerts. The app’s alert history lists the notifications the device received.

Push notifications

Alerts reach a member’s phone or tablet once that device allows notifications for PocketCmd. The app asks for permission when you set up your first alert. When you allow it, the device registers with the server for push, and the server sends the alerts you are permitted to receive through PocketCmd’s push relay. A device that doesn’t allow notifications is not registered, and nothing is sent to it.

How delivery works:

1. The app registers its FCM (Android) or APNs (Apple) token with PocketCmd’s push relay and receives a signed token, then registers both with each paired server.
2. When a rule fires, the server sends the title, the body and the device’s tokens to the relay, which forwards the notification to FCM or APNs. Only devices whose user holds Receive alerts for that metric get it.
3. When the relay reports a token as gone, the server deletes that registration.

Notification text is not end-to-end encrypted

The title and body pass through the relay and through Apple’s or Google’s push service in cleartext: the server’s hostname, the metric label, the value and the threshold. Don’t put secrets in metric labels.

The signed token lets anyone holding it send notifications to that device, which is why the server deletes a device’s registration as soon as the device is revoked or leaves.

Push notifications reach the iOS and Android apps only; the desktop apps do not register for push.

Metrics API

All endpoints require mTLS, a session token and the X-PocketCmd-Api-Version header. The three read endpoints also require View metrics.

Endpoint	Method	Purpose
/api/metrics/available	GET	The approved commands, in configuration order. A server with none answers an empty list.
/api/metrics/latest	GET	The most recent sample of every metric.
/api/metrics/series	GET	Time series for metric=<id> over range=1m|1h|6h|24h|7d|30d|1y.
/api/metrics/requests	GET, POST	List requests, or propose a command (any member).
/api/metrics/requests/{id}/approve	POST	Approve a request (owner only).
/api/metrics/requests/{id}/decline	POST	Decline a request (owner only).
/api/metrics/requests/{id}	DELETE	Withdraw a pending request, or remove an approved command.

/api/metrics/series returns raw samples for 1m and 1h and averages per bucket for longer ranges:

Range	Bucket
1m, 1h	none (raw samples)
6h	1 minute
24h	5 minutes
7d	1 hour
30d	6 hours
1y	1 day

Retention is at most 30 days, so 1y returns whatever is still stored. Decided requests older than 30 days are pruned automatically.

Stopping collection

The server has no switch for monitoring. To stop a metric, delete its approved request under Metric requests in the app. Its command stops running, and its alert rules go with it.