Fix persistent log collector liveness and BusyBox lock waits
This commit is contained in:
1 parent
b62db40daf
commit
e738d6fa69
8 files changed
+846
-37
No files matched your search
@@ -0,0 +1,111 @@
|
||||
# Journal collector liveness fix (local v117)
|
||||
|
||||
## What the reported evidence does and does not prove
|
||||
|
||||
The previous `log_new_fact_locked()` normalized syslog timestamps and stored the
|
||||
first successful `Heartbeat: OK; ...` in `heartbeat-ok`. The identical success
|
||||
was then suppressed **without an expiry**, even after hours. A 136-byte chunk,
|
||||
`ack=total`, empty pending and live logread therefore do not prove a stalled
|
||||
collector. They also do not establish a memory leak. A shell syntax check and
|
||||
`persistent=1` prove neither event delivery nor process health.
|
||||
|
||||
The old supervisor checked only `kill -0` for its background writer and reader;
|
||||
it ignored `log_append` failure. This could conceal write/lock errors while the
|
||||
supervisor remained alive. These failure paths are corrected independently of
|
||||
the deduplication issue. No claim is made that they caused this particular
|
||||
AX3000T report without measurements from that device.
|
||||
|
||||
## Changes
|
||||
|
||||
- One supervising collect shell consumes its private FIFO with BusyBox ash
|
||||
timed reads. There is one logread child, no separate persistent writer shell
|
||||
and no five-second sleep child. At most one 4096-byte input fragment is kept
|
||||
in a shell variable. Oversized/fragmented input is diagnosed and discarded;
|
||||
it cannot grow a shell string without a bound.
|
||||
- Each complete line is handed to `log_append` over an explicit short pipe,
|
||||
isolated from the live FIFO. Append failure is checked, recorded and causes
|
||||
a nonzero exit so procd can recover. Lock-operation traps run in subshells
|
||||
and cannot replace the collector's cleanup trap.
|
||||
- Normal stop terminates/reaps logread, closes/removes the FIFO, flushes a final
|
||||
bounded batch and releases only its own collector lock. SIGKILL recovery
|
||||
recognizes a stale owner and terminates its recorded orphan reader only if
|
||||
PID **and kernel start time** match. Stale-lock reaping is serialized.
|
||||
- procd's journal instance retries every 10 seconds without exhausting a
|
||||
five-crash budget. The archive instance and networking services are unchanged.
|
||||
- Builds without fractional BusyBox sleep use whole-second waits. Lock and
|
||||
child cleanup deadlines account for that duration; unsupported `sleep 0.1`
|
||||
cannot produce a fast loop, repeated errors or a 100-second lock wait.
|
||||
- A successful heartbeat is saved at most once per 300 seconds when unchanged,
|
||||
plus changes and recovery. Intervening duplicates update small RAM health
|
||||
metadata only. Flash flush is still once per 60 seconds, <=16 KiB per batch,
|
||||
with existing size/age/part limits. No per-heartbeat Flash write is introduced.
|
||||
- Starting logread requests only a 64-message system-ring tail plus follow,
|
||||
to recover a short interruption. Existing fact dedup and bounded pending
|
||||
apply; this is not a guarantee of lossless collection during a long outage.
|
||||
- `status` adds input and accepted-event times, counters, collector state,
|
||||
append/flush errors, the last completed batch/commit, allocated runtime bytes
|
||||
and RSS for the collector/reader when proc provides it. `receiving` means
|
||||
recently processed input; `idle-unverified` explicitly does not assert health.
|
||||
- `diagnostics` is a private RAM-only <=4 KiB reason-code history (plus <=2 KiB
|
||||
compaction), with duplicate-rate limiting. It records FIFO creation/open,
|
||||
fragmented input, lock timeout, append, flush and reader exit failures without
|
||||
storing event text, keys or command lines. It is not sent back through logger,
|
||||
avoiding diagnostic feedback loops.
|
||||
- Existing validated persistent storage, immutable checksummed chunks, ack
|
||||
protocol and limited RAM fallback remain. Runtime health is lost on reboot;
|
||||
retained journal/ack data are not cleared by a collector restart.
|
||||
|
||||
## Local install and rollback (Windows PowerShell 5.1/11)
|
||||
|
||||
The dedicated installer replaces **only** `logs.sh` and `owrt-remote-logs`.
|
||||
It backs up code/service state, preserves journal chunks and UCI configuration,
|
||||
restarts only the log service, verifies a real logger marker through the FIFO
|
||||
and performs one explicit deployment flush. Failure restores the previous code.
|
||||
It does not remove `xray.zip`, update Xray, restart the tunnel, modify Podkop,
|
||||
WAN/DNS or policy routing. VPS files and hosting URLs are untouched.
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Install
|
||||
```
|
||||
|
||||
Retained backup: `/root/owrt-remote-backup-before-log-collector-v117`.
|
||||
Rollback leaves journal data intact:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
|
||||
```
|
||||
|
||||
After installation compare two samples a minute apart. `last_input_epoch` and
|
||||
`suppressed_events` should advance on stable heartbeat. An unchanged heartbeat
|
||||
becomes a new retained sample after five minutes. A distinct marker is visible
|
||||
immediately in read, then committed by the next normal flush (up to ~65 seconds).
|
||||
|
||||
```powershell
|
||||
ssh root@192.168.2.1 'logger -t owrt-remote collector-user-check; sleep 3; sh /usr/lib/owrt-remote/logs.sh read | tail -n 12; sh /usr/lib/owrt-remote/logs.sh status; sh /usr/lib/owrt-remote/logs.sh diagnostics; free'
|
||||
```
|
||||
|
||||
Measure memory using `MemAvailable`, tmpfs allocation and process RSS over time;
|
||||
`free` memory alone includes reclaimable kernel/filesystem cache. Do not attribute
|
||||
all process RSS or low available RAM to the log spool without these measurements.
|
||||
No router or VPS is modified by automated tests.
|
||||
|
||||
## Automated coverage and limits
|
||||
|
||||
- Real FIFO tests using MSYS shell on Windows: live heartbeat sampling, distinct
|
||||
events through pending/Flash, duplicate collector rejection, reader death,
|
||||
append failure, owner SIGKILL and successor recovery, TERM cleanup/final flush.
|
||||
- Real BusyBox ash/applets: spool bounds, atomic/checksum/ack recovery, sampling,
|
||||
limited diagnostics, fallback and cleanup paths; Windows BusyBox lacks FIFOs.
|
||||
- Sandboxed installer: install, first-backup preservation, rollback, bad syntax,
|
||||
failed verification restoration, journal retention.
|
||||
- Existing redaction/CGI/Hub transport regressions and shell syntax checks.
|
||||
|
||||
This is not a claim of physical AX3000T / OpenWrt 24.10.3 UBIFS testing, hardware
|
||||
power-cut testing or a measured RAM reduction in MiB. The actual device still
|
||||
needs event/flush/reboot and memory observations before release.
|
||||
|
||||
Implementation references checked:
|
||||
[OpenWrt 24.10 ubox version](https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/ubox/Makefile),
|
||||
[its logread](https://github.com/openwrt/ubox/blob/85f1053019caf4cd333795760950235ee4529ba7/log/logread.c),
|
||||
[BusyBox ash timed read](https://github.com/mirror/busybox/blob/1_36_stable/shell/ash.c),
|
||||
[procd respawn](https://github.com/openwrt/procd/blob/master/service/instance.c).
|
||||
Reference in new issue
Block a user