Fix persistent log collector liveness and BusyBox lock waits

This commit is contained in:
kzolotarev95 committed 2026-10-16 00:12:54 +04:00
1 parent b62db40daf
commit e738d6fa69
8 files changed
+846 -37

No files matched your search

+111
View File
@@ -0,0 +1,111 @@
# Journal collector liveness fix (local v117)
## What the reported evidence does and does not prove
The previous `log_new_fact_locked()` normalized syslog timestamps and stored the
first successful `Heartbeat: OK; ...` in `heartbeat-ok`. The identical success
was then suppressed **without an expiry**, even after hours. A 136-byte chunk,
`ack=total`, empty pending and live logread therefore do not prove a stalled
collector. They also do not establish a memory leak. A shell syntax check and
`persistent=1` prove neither event delivery nor process health.
The old supervisor checked only `kill -0` for its background writer and reader;
it ignored `log_append` failure. This could conceal write/lock errors while the
supervisor remained alive. These failure paths are corrected independently of
the deduplication issue. No claim is made that they caused this particular
AX3000T report without measurements from that device.
## Changes
- One supervising collect shell consumes its private FIFO with BusyBox ash
timed reads. There is one logread child, no separate persistent writer shell
and no five-second sleep child. At most one 4096-byte input fragment is kept
in a shell variable. Oversized/fragmented input is diagnosed and discarded;
it cannot grow a shell string without a bound.
- Each complete line is handed to `log_append` over an explicit short pipe,
isolated from the live FIFO. Append failure is checked, recorded and causes
a nonzero exit so procd can recover. Lock-operation traps run in subshells
and cannot replace the collector's cleanup trap.
- Normal stop terminates/reaps logread, closes/removes the FIFO, flushes a final
bounded batch and releases only its own collector lock. SIGKILL recovery
recognizes a stale owner and terminates its recorded orphan reader only if
PID **and kernel start time** match. Stale-lock reaping is serialized.
- procd's journal instance retries every 10 seconds without exhausting a
five-crash budget. The archive instance and networking services are unchanged.
- Builds without fractional BusyBox sleep use whole-second waits. Lock and
child cleanup deadlines account for that duration; unsupported `sleep 0.1`
cannot produce a fast loop, repeated errors or a 100-second lock wait.
- A successful heartbeat is saved at most once per 300 seconds when unchanged,
plus changes and recovery. Intervening duplicates update small RAM health
metadata only. Flash flush is still once per 60 seconds, <=16 KiB per batch,
with existing size/age/part limits. No per-heartbeat Flash write is introduced.
- Starting logread requests only a 64-message system-ring tail plus follow,
to recover a short interruption. Existing fact dedup and bounded pending
apply; this is not a guarantee of lossless collection during a long outage.
- `status` adds input and accepted-event times, counters, collector state,
append/flush errors, the last completed batch/commit, allocated runtime bytes
and RSS for the collector/reader when proc provides it. `receiving` means
recently processed input; `idle-unverified` explicitly does not assert health.
- `diagnostics` is a private RAM-only <=4 KiB reason-code history (plus <=2 KiB
compaction), with duplicate-rate limiting. It records FIFO creation/open,
fragmented input, lock timeout, append, flush and reader exit failures without
storing event text, keys or command lines. It is not sent back through logger,
avoiding diagnostic feedback loops.
- Existing validated persistent storage, immutable checksummed chunks, ack
protocol and limited RAM fallback remain. Runtime health is lost on reboot;
retained journal/ack data are not cleared by a collector restart.
## Local install and rollback (Windows PowerShell 5.1/11)
The dedicated installer replaces **only** `logs.sh` and `owrt-remote-logs`.
It backs up code/service state, preserves journal chunks and UCI configuration,
restarts only the log service, verifies a real logger marker through the FIFO
and performs one explicit deployment flush. Failure restores the previous code.
It does not remove `xray.zip`, update Xray, restart the tunnel, modify Podkop,
WAN/DNS or policy routing. VPS files and hosting URLs are untouched.
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Install
```
Retained backup: `/root/owrt-remote-backup-before-log-collector-v117`.
Rollback leaves journal data intact:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
```
After installation compare two samples a minute apart. `last_input_epoch` and
`suppressed_events` should advance on stable heartbeat. An unchanged heartbeat
becomes a new retained sample after five minutes. A distinct marker is visible
immediately in read, then committed by the next normal flush (up to ~65 seconds).
```powershell
ssh root@192.168.2.1 'logger -t owrt-remote collector-user-check; sleep 3; sh /usr/lib/owrt-remote/logs.sh read | tail -n 12; sh /usr/lib/owrt-remote/logs.sh status; sh /usr/lib/owrt-remote/logs.sh diagnostics; free'
```
Measure memory using `MemAvailable`, tmpfs allocation and process RSS over time;
`free` memory alone includes reclaimable kernel/filesystem cache. Do not attribute
all process RSS or low available RAM to the log spool without these measurements.
No router or VPS is modified by automated tests.
## Automated coverage and limits
- Real FIFO tests using MSYS shell on Windows: live heartbeat sampling, distinct
events through pending/Flash, duplicate collector rejection, reader death,
append failure, owner SIGKILL and successor recovery, TERM cleanup/final flush.
- Real BusyBox ash/applets: spool bounds, atomic/checksum/ack recovery, sampling,
limited diagnostics, fallback and cleanup paths; Windows BusyBox lacks FIFOs.
- Sandboxed installer: install, first-backup preservation, rollback, bad syntax,
failed verification restoration, journal retention.
- Existing redaction/CGI/Hub transport regressions and shell syntax checks.
This is not a claim of physical AX3000T / OpenWrt 24.10.3 UBIFS testing, hardware
power-cut testing or a measured RAM reduction in MiB. The actual device still
needs event/flush/reboot and memory observations before release.
Implementation references checked:
[OpenWrt 24.10 ubox version](https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/ubox/Makefile),
[its logread](https://github.com/openwrt/ubox/blob/85f1053019caf4cd333795760950235ee4529ba7/log/logread.c),
[BusyBox ash timed read](https://github.com/mirror/busybox/blob/1_36_stable/shell/ash.c),
[procd respawn](https://github.com/openwrt/procd/blob/master/service/instance.c).
+2 -2
View File
@@ -108,7 +108,7 @@ class GlobalLogTests(unittest.TestCase):
self.helper_call('append', good * 3)
self.helper_call('append', 'Other action\n')
self.helper_call('append', good, env={**self.env, 'TEST_UPTIME': '2000'})
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2000'}).count('Heartbeat: OK;'), 1)
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2000'}).count('Heartbeat: OK;'), 2)
failed = 'Heartbeat: ошибка; endpoint=https://hub.example; rc=7; connection refused\n'
self.helper_call('append', failed * 3, env={**self.env, 'TEST_UPTIME': '2001'})
self.helper_call('append', good, env={**self.env, 'TEST_UPTIME': '2002'})
@@ -117,7 +117,7 @@ class GlobalLogTests(unittest.TestCase):
env={**self.env, 'TEST_UPTIME': '2004'})
stored = self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2004'})
self.assertEqual(stored.count('[ERROR] Heartbeat:'), 2, 'same failure after recovery must appear')
self.assertEqual(stored.count('[OK] Heartbeat:'), 3, 'first, recovery and another WAN')
self.assertEqual(stored.count('[OK] Heartbeat:'), 4, 'first, periodic sample, recovery and another WAN')
def test_recurring_failure_is_recorded_again_after_a_quiet_period(self):
failure = 'owrt-remote: cannot download Xray\n'
+172
View File
@@ -0,0 +1,172 @@
"""Real FIFO lifecycle tests; MSYS provides FIFOs on Windows, BusyBox tests the applets separately."""
import os
from pathlib import Path
import subprocess
import time
import unittest
import test_persistent_logs as spool
class CollectorTests(unittest.TestCase):
setUp = spool.PersistentLogsTests.setUp
call = spool.PersistentLogsTests.call
status = spool.PersistentLogsTests.status
flush = spool.PersistentLogsTests.flush
journal = spool.PersistentLogsTests.journal
# Reuse filesystem fixtures without inheriting the persistent spool test cases.
def start(self, extra=""):
self.input = self.root / 'input'
self.env['TEST_INPUT'] = self.input.as_posix()
self.env['TEST_CLOCK_FILE'] = (self.root / 'clock').as_posix()
(self.root / 'clock').write_text('1000\n')
mocks = r'''
log_clock() { cat "$TEST_CLOCK_FILE"; }
logread() {
exec 4<>"$TEST_INPUT"
while :; do if IFS= read -r -t 1 item <&4; then printf '%s\n' "$item"; fi; done
}
'''
source = self.source.replace('case "${1:-}" in', mocks + extra + '\ncase "${1:-}" in', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
if not getattr(self, 'input_created', False):
result = subprocess.run([*self.shell, '-c', 'mkfifo "$TEST_INPUT"'], env=self.env,
capture_output=True, timeout=10)
if result.returncode:
self.skipTest('This shell has no FIFO support')
self.input_created = True
output = (self.root / ('process-%d.log' % len(getattr(self, 'processes', [])))).open('wb')
proc = subprocess.Popen([*self.shell, self.helper.as_posix(), 'collect'], env=self.env,
stdout=output, stderr=output)
if not hasattr(self, 'processes'):
self.processes = []
self.addCleanup(self.stop_processes)
self.processes.append((proc, output))
return proc
def wait_for(self, predicate, seconds=20):
deadline = time.monotonic() + seconds
while time.monotonic() < deadline:
value = predicate()
if value:
return value
time.sleep(.15)
self.fail('Collector did not reach expected state')
def ready(self):
return self.wait_for(lambda: (self.run_dir / 'collector-lock/reader').exists())
def signal(self, pid, signal='TERM'):
return subprocess.run([*self.shell, '-c', 'kill -"$1" "$2"', '--', signal, str(pid)],
env=self.env, capture_output=True, timeout=10)
def stop_processes(self):
lock = self.run_dir / 'collector-lock/pid'
if lock.exists():
self.signal(lock.read_text().strip())
for proc, output in self.processes:
try:
proc.wait(timeout=15)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=5)
output.close()
def send(self, *lines):
result = subprocess.run([*self.shell, '-c', 'cat >"$TEST_INPUT"'], input='\n'.join(lines) + '\n',
encoding='utf-8', env=self.env, capture_output=True, timeout=15)
self.assertEqual(result.returncode, 0, result.stderr)
def test_actual_fifo_heartbeat_freshness_sample_and_batch_flush(self):
proc = self.start(); self.ready()
good = 'Thu Oct 15 22:52:36 2026 user.notice owrt-remote: Heartbeat: OK; Hub=https://hub.example; WAN_direct=1; WAN_active=wan'
self.send(good)
self.wait_for(lambda: self.status()['accepted_events'] == '1')
self.send(good.replace('22:52:36', '22:53:08'))
self.wait_for(lambda: self.status()['suppressed_events'] == '1')
state = self.status()
self.assertEqual(state['collector_state'], 'receiving')
self.assertEqual(state['received_lines'], '2')
self.assertGreater(int(state['last_input_epoch']), 0)
self.assertEqual(state['committed_bytes'], '0')
(self.root / 'clock').write_text('1301\n')
self.send(good.replace('22:52:36', '22:57:38'), 'owrt-remote: panel action after heartbeat')
self.wait_for(lambda: self.status()['accepted_events'] == '3')
read = self.call('read').stdout
self.assertEqual(read.count('Heartbeat: OK;'), 2)
self.assertIn('panel action after heartbeat', read)
self.flush()
self.assertEqual(self.status()['pending_bytes'], '0')
self.assertIn(b'panel action after heartbeat', b''.join(p.read_bytes() for p in self.journal.glob('part.*')))
self.assertIsNone(proc.poll())
def test_duplicate_collect_does_not_remove_owner_fifo(self):
proc = self.start(); self.ready()
owner = (self.run_dir / 'collector-lock/pid').read_text()
fifo = list(self.run_dir.glob('stream.*'))
other = self.start(); self.assertEqual(other.wait(timeout=15), 0)
self.assertEqual((self.run_dir / 'collector-lock/pid').read_text(), owner)
self.assertEqual(list(self.run_dir.glob('stream.*')), fifo)
self.send('owrt-remote: owner still receives')
self.wait_for(lambda: self.status()['accepted_events'] == '1')
self.assertIsNone(proc.poll())
def test_reader_death_exits_nonzero_and_retains_journal_for_restart(self):
proc = self.start(); self.ready()
self.send('owrt-remote: before logread failure')
self.wait_for(lambda: self.status()['accepted_events'] == '1')
reader = (self.run_dir / 'collector-lock/reader').read_text().split()[0]
self.signal(reader, 'KILL')
self.assertNotEqual(proc.wait(timeout=20), 0)
self.assertFalse((self.run_dir / 'collector-lock').exists())
self.assertFalse(list(self.run_dir.glob('stream.*')))
self.assertIn('reader-exited', self.call('diagnostics').stdout)
self.assertIn('before logread failure', self.call('read').stdout)
new = self.start(); self.ready()
self.send('owrt-remote: after reader recovery')
self.wait_for(lambda: self.status()['accepted_events'] == '2')
self.assertIsNone(new.poll())
def test_sigkill_owner_reaps_identified_reader_and_stale_fifo(self):
proc = self.start(); self.ready()
owner = (self.run_dir / 'collector-lock/pid').read_text().strip()
self.send('owrt-remote: retained before killed collector')
self.wait_for(lambda: self.status()['accepted_events'] == '1')
old_reader = (self.run_dir / 'collector-lock/reader').read_text().split()[0]
self.signal(owner, 'KILL'); proc.wait(timeout=10)
new = self.start()
self.wait_for(lambda: (self.run_dir / 'collector-lock/pid').exists() and
(self.run_dir / 'collector-lock/pid').read_text().strip() != owner)
self.wait_for(lambda: (self.run_dir / 'collector-lock/reader').exists() and
(self.run_dir / 'collector-lock/reader').read_text().split()[0] != old_reader)
self.assertEqual(len(list(self.run_dir.glob('stream.*'))), 1)
self.send('owrt-remote: recovered collector')
self.wait_for(lambda: self.status()['accepted_events'] == '2')
text = self.call('read').stdout
self.assertIn('retained before killed collector', text)
self.assertIn('recovered collector', text)
self.assertIsNone(new.poll())
def test_append_failure_is_not_silently_ignored(self):
proc = self.start('\nlog_collect_line() { return 1; }\n'); self.ready()
self.send('owrt-remote: rejected input')
self.assertNotEqual(proc.wait(timeout=20), 0)
self.assertIn('collector-append-failed', self.call('diagnostics').stdout)
self.assertEqual(self.status()['append_errors'], '1')
self.assertFalse((self.run_dir / 'collector-lock').exists())
def test_term_flushes_and_cleans_children_fifo_and_owner_lock(self):
proc = self.start(); self.ready()
self.send('owrt-remote: shutdown flush')
self.wait_for(lambda: self.status()['accepted_events'] == '1')
owner = (self.run_dir / 'collector-lock/pid').read_text().strip()
self.signal(owner)
self.assertEqual(proc.wait(timeout=15), 0)
self.assertEqual(self.status()['pending_bytes'], '0')
self.assertIn('shutdown flush', self.call('read').stdout)
self.assertFalse(list(self.run_dir.glob('stream.*')))
self.assertFalse((self.run_dir / 'collector-lock').exists())
if __name__ == '__main__':
unittest.main()
+95
View File
@@ -0,0 +1,95 @@
"""Deploy only the journal helper/service, preserving data and networking."""
import os
from pathlib import Path
import re
import shlex
import shutil
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
class CollectorDeploymentTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory(prefix='collector-deploy-')
self.addCleanup(self.tmp.cleanup)
self.root = Path(self.tmp.name)
self.targets = {'logs.sh': '/usr/lib/owrt-remote/logs.sh', 'logs-init': '/etc/init.d/owrt-remote-logs'}
self.initial = {}
for item, target in self.targets.items():
path = self.root / target.lstrip('/')
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text('#!/bin/sh\necho old-' + item + '-$1\n', newline='\n')
self.initial[item] = path.read_bytes()
self.backup = self.root / 'backup'
self.journal = self.root / 'retained-journal'; self.journal.write_text('keep history')
text = (ROOT / 'vps/deploy-log-collector-fix.ps1').read_text(encoding='utf-8')
self.script = text.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
mapping = {target: (self.root / target.lstrip('/')).as_posix() for target in self.targets.values()}
mapping['/root/owrt-remote-backup-before-log-collector-v117'] = self.backup.as_posix()
pattern = '|'.join(re.escape(path) for path in sorted(mapping, key=len, reverse=True))
self.script = re.sub(pattern, lambda m: mapping[m.group()], self.script)
mocks = r'''
ubus() { printf '{"running":true}\n'; }
jsonfilter() { printf true; }
logread() { return 0; }
mkfifo() { return 0; }
logger() { printf '%s\n' "$3" >"$TEST_MARKER"; }
sleep() { return 0; }
'''
self.script = self.script.replace('trap cleanup EXIT', mocks + '\ntrap cleanup EXIT')
self.env = {**os.environ, 'PATH': str(GIT_BIN) + os.pathsep + os.environ['PATH'],
'TEST_MARKER': (self.root / 'marker').as_posix()}
def deploy(self, action, bad=False, fail=False):
stage = self.root / 'stage'; stage.mkdir()
if action == 'Install':
helper = r'''#!/bin/sh
# log_collector_progress()
# LOG_HEARTBEAT_SECONDS=300
case "$1" in
read) cat "$TEST_MARKER" ;;
status) echo persistent=1 ;;
esac
'''
if bad: helper = '#!/bin/sh\nif true; then\n'
(stage / 'logs.sh').write_text(helper, newline='\n')
(stage / 'logs-init').write_text('#!/bin/sh\necho new-service-$1\n', newline='\n')
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(stage.as_posix()))
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
if fail: script = script.replace('cmp "$source_dir/$item" "$path"', 'false')
result = subprocess.run([SH], input=script, text=True, encoding='utf-8', capture_output=True, env=self.env, timeout=20)
self.assertFalse(stage.exists())
self.assertEqual(self.journal.read_text(), 'keep history')
return result
def test_install_and_rollback_keep_first_backup_and_journal(self):
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('LOG_COLLECTOR_ROUTER_Install_OK', result.stdout)
for item, initial in self.initial.items(): self.assertEqual((self.backup / item).read_bytes(), initial)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
for item, target in self.targets.items(): self.assertEqual((self.root / target.lstrip('/')).read_bytes(), self.initial[item])
def test_missing_backup_is_noop(self):
result = self.deploy('Rollback')
self.assertNotEqual(result.returncode, 0)
self.assertIn('Nothing changed', result.stderr)
def test_invalid_shell_rejected_before_backup(self):
result = self.deploy('Install', bad=True)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(self.backup.exists())
def test_failed_verify_restores_previous_code(self):
result = self.deploy('Install', fail=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn('Restoring journal code', result.stderr)
for item, target in self.targets.items(): self.assertEqual((self.root / target.lstrip('/')).read_bytes(), self.initial[item])
if __name__ == '__main__': unittest.main()
+70 -3
View File
@@ -128,7 +128,7 @@ class PersistentLogsTests(unittest.TestCase):
(self.run_dir / 'facts').write_text(''.join('1000\ts256-%035d\n' % i for i in range(63)))
self.append('distinct event 63\ndistinct event 64\n')
self.assertEqual(len((self.run_dir / 'facts').read_text().splitlines()), 64)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size if (self.run_dir / 'facts').exists() else 0, 4096)
def test_empty_digest_files_falsely_acked_by_old_hub_are_recovered(self):
self.append('retained restart event\n'); self.flush()
@@ -242,7 +242,7 @@ class PersistentLogsTests(unittest.TestCase):
self.flush()
self.assertEqual({p.name: p.read_bytes() for p in parts}, original)
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size if (self.run_dir / 'facts').exists() else 0, 4096)
first = self.export()
self.acknowledge(first)
self.flush()
@@ -353,6 +353,32 @@ class PersistentLogsTests(unittest.TestCase):
self.assertIn('after killed reaper', self.call('read').stdout)
self.assertFalse(lock.exists())
def test_integer_only_sleep_keeps_lock_timeout_bounded_and_quiet(self):
self.call('check')
source = self.source.split('case "${1:-}" in', 1)[0]
commands = r'''
log_init
mkdir "$RUN_DIR/lock"
printf '12345\n' >"$RUN_DIR/lock/pid"
sleep() {
printf '%s\n' "$1" >>"$RUN_DIR/pauses"
if [ "$1" = 0.1 ]; then echo "sleep: invalid number '0.1'" >&2; return 1; fi
[ "$1" = 1 ]
}
kill() { return 0; }
if log_lock; then exit 17; fi
'''
script = self.root / 'integer-sleep.sh'
script.write_text(source + commands, encoding='utf-8', newline='\n')
result = subprocess.run([*self.shell, script.as_posix()], env=self.env,
capture_output=True, text=True, timeout=15)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stderr, '')
pauses = (self.run_dir / 'pauses').read_text().splitlines()
self.assertEqual(pauses.count('0.1'), 1, 'probe once per process')
self.assertEqual(pauses.count('1'), 10, 'ten-second deadline, no busy spin or 100-second wait')
self.assertIn('lock-timeout', self.call('diagnostics').stdout)
def test_collector_failure_reaps_children_and_removes_fifo(self):
# Actual FIFO integration is tested on POSIX; Windows BusyBox has no
# FIFO support. Exercise its traps with mocked process/FIFO operations.
@@ -373,7 +399,48 @@ wait() { return 0; }
self.assertFalse(list(self.run_dir.glob('stream.*')))
self.assertFalse((self.run_dir / 'collector-lock').exists())
lines = (self.run_dir / 'kills').read_text().splitlines()
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 2)
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 1)
def test_heartbeat_sampling_and_freshness_are_independent_of_flash_writes(self):
good = 'Thu Oct 15 22:52:36 2026 user.notice owrt-remote: Heartbeat: OK; Hub=https://hub.example; WAN_direct=1; WAN_active=wan\n'
self.append(good)
self.append(good.replace('22:52:36', '22:53:08'), TEST_UPTIME='1032')
state = self.status()
self.assertEqual((state['accepted_events'], state['suppressed_events']), ('1', '1'))
self.assertEqual(state['committed_bytes'], '0')
self.assertGreater(int(state['last_input_epoch']), 0)
self.assertEqual(state['collector_state'], 'stopped', 'persistent storage does not imply a running collector')
self.append(good.replace('22:52:36', '22:57:38'), TEST_UPTIME='1302')
self.assertEqual(self.call('read').stdout.count('Heartbeat: OK;'), 2)
self.flush()
self.assertEqual(self.status()['pending_bytes'], '0')
self.assertGreater(int(self.status()['last_commit_epoch']), 0)
def test_private_diagnostics_are_bounded_rate_limited_and_no_syslog_feedback(self):
self.call('check')
source = self.source.split('case "${1:-}" in', 1)[0]
commands = r'''
log_init
logger() { echo forbidden-syslog-feedback >&2; return 1; }
i=0
while [ "$i" -lt 140 ]; do
if [ "$((i % 2))" = 0 ]; then log_diag flush-failed; else log_diag lock-timeout; fi
i=$((i + 1))
done
log_diag lock-timeout
'''
script = self.root / 'diag-test.sh'
script.write_text(source + commands, encoding='utf-8', newline='\n')
result = subprocess.run([*self.shell, script.as_posix()], env=self.env,
capture_output=True, text=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stderr, '')
diagnostics = self.run_dir / 'diagnostics'
self.assertLessEqual(diagnostics.stat().st_size, 4096)
self.assertFalse((self.run_dir / 'diagnostics.new').exists())
self.assertFalse((self.run_dir / 'diag-lock').exists())
self.assertEqual(self.call('diagnostics').stdout, diagnostics.read_text())
self.assertFalse((self.journal / 'diagnostics').exists())
def test_legacy_migration_preserves_identity_ack_and_only_undelivered_bytes(self):
legacy = self.root / 'legacy'