Fix persistent log collector liveness and BusyBox lock waits
This commit is contained in:
1 parent
b62db40daf
commit
e738d6fa69
8 files changed
+846
-37
No files matched your search
@@ -0,0 +1,111 @@
|
||||
# Journal collector liveness fix (local v117)
|
||||
|
||||
## What the reported evidence does and does not prove
|
||||
|
||||
The previous `log_new_fact_locked()` normalized syslog timestamps and stored the
|
||||
first successful `Heartbeat: OK; ...` in `heartbeat-ok`. The identical success
|
||||
was then suppressed **without an expiry**, even after hours. A 136-byte chunk,
|
||||
`ack=total`, empty pending and live logread therefore do not prove a stalled
|
||||
collector. They also do not establish a memory leak. A shell syntax check and
|
||||
`persistent=1` prove neither event delivery nor process health.
|
||||
|
||||
The old supervisor checked only `kill -0` for its background writer and reader;
|
||||
it ignored `log_append` failure. This could conceal write/lock errors while the
|
||||
supervisor remained alive. These failure paths are corrected independently of
|
||||
the deduplication issue. No claim is made that they caused this particular
|
||||
AX3000T report without measurements from that device.
|
||||
|
||||
## Changes
|
||||
|
||||
- One supervising collect shell consumes its private FIFO with BusyBox ash
|
||||
timed reads. There is one logread child, no separate persistent writer shell
|
||||
and no five-second sleep child. At most one 4096-byte input fragment is kept
|
||||
in a shell variable. Oversized/fragmented input is diagnosed and discarded;
|
||||
it cannot grow a shell string without a bound.
|
||||
- Each complete line is handed to `log_append` over an explicit short pipe,
|
||||
isolated from the live FIFO. Append failure is checked, recorded and causes
|
||||
a nonzero exit so procd can recover. Lock-operation traps run in subshells
|
||||
and cannot replace the collector's cleanup trap.
|
||||
- Normal stop terminates/reaps logread, closes/removes the FIFO, flushes a final
|
||||
bounded batch and releases only its own collector lock. SIGKILL recovery
|
||||
recognizes a stale owner and terminates its recorded orphan reader only if
|
||||
PID **and kernel start time** match. Stale-lock reaping is serialized.
|
||||
- procd's journal instance retries every 10 seconds without exhausting a
|
||||
five-crash budget. The archive instance and networking services are unchanged.
|
||||
- Builds without fractional BusyBox sleep use whole-second waits. Lock and
|
||||
child cleanup deadlines account for that duration; unsupported `sleep 0.1`
|
||||
cannot produce a fast loop, repeated errors or a 100-second lock wait.
|
||||
- A successful heartbeat is saved at most once per 300 seconds when unchanged,
|
||||
plus changes and recovery. Intervening duplicates update small RAM health
|
||||
metadata only. Flash flush is still once per 60 seconds, <=16 KiB per batch,
|
||||
with existing size/age/part limits. No per-heartbeat Flash write is introduced.
|
||||
- Starting logread requests only a 64-message system-ring tail plus follow,
|
||||
to recover a short interruption. Existing fact dedup and bounded pending
|
||||
apply; this is not a guarantee of lossless collection during a long outage.
|
||||
- `status` adds input and accepted-event times, counters, collector state,
|
||||
append/flush errors, the last completed batch/commit, allocated runtime bytes
|
||||
and RSS for the collector/reader when proc provides it. `receiving` means
|
||||
recently processed input; `idle-unverified` explicitly does not assert health.
|
||||
- `diagnostics` is a private RAM-only <=4 KiB reason-code history (plus <=2 KiB
|
||||
compaction), with duplicate-rate limiting. It records FIFO creation/open,
|
||||
fragmented input, lock timeout, append, flush and reader exit failures without
|
||||
storing event text, keys or command lines. It is not sent back through logger,
|
||||
avoiding diagnostic feedback loops.
|
||||
- Existing validated persistent storage, immutable checksummed chunks, ack
|
||||
protocol and limited RAM fallback remain. Runtime health is lost on reboot;
|
||||
retained journal/ack data are not cleared by a collector restart.
|
||||
|
||||
## Local install and rollback (Windows PowerShell 5.1/11)
|
||||
|
||||
The dedicated installer replaces **only** `logs.sh` and `owrt-remote-logs`.
|
||||
It backs up code/service state, preserves journal chunks and UCI configuration,
|
||||
restarts only the log service, verifies a real logger marker through the FIFO
|
||||
and performs one explicit deployment flush. Failure restores the previous code.
|
||||
It does not remove `xray.zip`, update Xray, restart the tunnel, modify Podkop,
|
||||
WAN/DNS or policy routing. VPS files and hosting URLs are untouched.
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Install
|
||||
```
|
||||
|
||||
Retained backup: `/root/owrt-remote-backup-before-log-collector-v117`.
|
||||
Rollback leaves journal data intact:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-log-collector-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
|
||||
```
|
||||
|
||||
After installation compare two samples a minute apart. `last_input_epoch` and
|
||||
`suppressed_events` should advance on stable heartbeat. An unchanged heartbeat
|
||||
becomes a new retained sample after five minutes. A distinct marker is visible
|
||||
immediately in read, then committed by the next normal flush (up to ~65 seconds).
|
||||
|
||||
```powershell
|
||||
ssh root@192.168.2.1 'logger -t owrt-remote collector-user-check; sleep 3; sh /usr/lib/owrt-remote/logs.sh read | tail -n 12; sh /usr/lib/owrt-remote/logs.sh status; sh /usr/lib/owrt-remote/logs.sh diagnostics; free'
|
||||
```
|
||||
|
||||
Measure memory using `MemAvailable`, tmpfs allocation and process RSS over time;
|
||||
`free` memory alone includes reclaimable kernel/filesystem cache. Do not attribute
|
||||
all process RSS or low available RAM to the log spool without these measurements.
|
||||
No router or VPS is modified by automated tests.
|
||||
|
||||
## Automated coverage and limits
|
||||
|
||||
- Real FIFO tests using MSYS shell on Windows: live heartbeat sampling, distinct
|
||||
events through pending/Flash, duplicate collector rejection, reader death,
|
||||
append failure, owner SIGKILL and successor recovery, TERM cleanup/final flush.
|
||||
- Real BusyBox ash/applets: spool bounds, atomic/checksum/ack recovery, sampling,
|
||||
limited diagnostics, fallback and cleanup paths; Windows BusyBox lacks FIFOs.
|
||||
- Sandboxed installer: install, first-backup preservation, rollback, bad syntax,
|
||||
failed verification restoration, journal retention.
|
||||
- Existing redaction/CGI/Hub transport regressions and shell syntax checks.
|
||||
|
||||
This is not a claim of physical AX3000T / OpenWrt 24.10.3 UBIFS testing, hardware
|
||||
power-cut testing or a measured RAM reduction in MiB. The actual device still
|
||||
needs event/flush/reboot and memory observations before release.
|
||||
|
||||
Implementation references checked:
|
||||
[OpenWrt 24.10 ubox version](https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/ubox/Makefile),
|
||||
[its logread](https://github.com/openwrt/ubox/blob/85f1053019caf4cd333795760950235ee4529ba7/log/logread.c),
|
||||
[BusyBox ash timed read](https://github.com/mirror/busybox/blob/1_36_stable/shell/ash.c),
|
||||
[procd respawn](https://github.com/openwrt/procd/blob/master/service/instance.c).
|
||||
@@ -108,7 +108,7 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.helper_call('append', good * 3)
|
||||
self.helper_call('append', 'Other action\n')
|
||||
self.helper_call('append', good, env={**self.env, 'TEST_UPTIME': '2000'})
|
||||
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2000'}).count('Heartbeat: OK;'), 1)
|
||||
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2000'}).count('Heartbeat: OK;'), 2)
|
||||
failed = 'Heartbeat: ошибка; endpoint=https://hub.example; rc=7; connection refused\n'
|
||||
self.helper_call('append', failed * 3, env={**self.env, 'TEST_UPTIME': '2001'})
|
||||
self.helper_call('append', good, env={**self.env, 'TEST_UPTIME': '2002'})
|
||||
@@ -117,7 +117,7 @@ class GlobalLogTests(unittest.TestCase):
|
||||
env={**self.env, 'TEST_UPTIME': '2004'})
|
||||
stored = self.helper_call('read', env={**self.env, 'TEST_UPTIME': '2004'})
|
||||
self.assertEqual(stored.count('[ERROR] Heartbeat:'), 2, 'same failure after recovery must appear')
|
||||
self.assertEqual(stored.count('[OK] Heartbeat:'), 3, 'first, recovery and another WAN')
|
||||
self.assertEqual(stored.count('[OK] Heartbeat:'), 4, 'first, periodic sample, recovery and another WAN')
|
||||
|
||||
def test_recurring_failure_is_recorded_again_after_a_quiet_period(self):
|
||||
failure = 'owrt-remote: cannot download Xray\n'
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Real FIFO lifecycle tests; MSYS provides FIFOs on Windows, BusyBox tests the applets separately."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import unittest
|
||||
|
||||
import test_persistent_logs as spool
|
||||
|
||||
|
||||
class CollectorTests(unittest.TestCase):
|
||||
setUp = spool.PersistentLogsTests.setUp
|
||||
call = spool.PersistentLogsTests.call
|
||||
status = spool.PersistentLogsTests.status
|
||||
flush = spool.PersistentLogsTests.flush
|
||||
journal = spool.PersistentLogsTests.journal
|
||||
# Reuse filesystem fixtures without inheriting the persistent spool test cases.
|
||||
def start(self, extra=""):
|
||||
self.input = self.root / 'input'
|
||||
self.env['TEST_INPUT'] = self.input.as_posix()
|
||||
self.env['TEST_CLOCK_FILE'] = (self.root / 'clock').as_posix()
|
||||
(self.root / 'clock').write_text('1000\n')
|
||||
mocks = r'''
|
||||
log_clock() { cat "$TEST_CLOCK_FILE"; }
|
||||
logread() {
|
||||
exec 4<>"$TEST_INPUT"
|
||||
while :; do if IFS= read -r -t 1 item <&4; then printf '%s\n' "$item"; fi; done
|
||||
}
|
||||
'''
|
||||
source = self.source.replace('case "${1:-}" in', mocks + extra + '\ncase "${1:-}" in', 1)
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
if not getattr(self, 'input_created', False):
|
||||
result = subprocess.run([*self.shell, '-c', 'mkfifo "$TEST_INPUT"'], env=self.env,
|
||||
capture_output=True, timeout=10)
|
||||
if result.returncode:
|
||||
self.skipTest('This shell has no FIFO support')
|
||||
self.input_created = True
|
||||
output = (self.root / ('process-%d.log' % len(getattr(self, 'processes', [])))).open('wb')
|
||||
proc = subprocess.Popen([*self.shell, self.helper.as_posix(), 'collect'], env=self.env,
|
||||
stdout=output, stderr=output)
|
||||
if not hasattr(self, 'processes'):
|
||||
self.processes = []
|
||||
self.addCleanup(self.stop_processes)
|
||||
self.processes.append((proc, output))
|
||||
return proc
|
||||
|
||||
def wait_for(self, predicate, seconds=20):
|
||||
deadline = time.monotonic() + seconds
|
||||
while time.monotonic() < deadline:
|
||||
value = predicate()
|
||||
if value:
|
||||
return value
|
||||
time.sleep(.15)
|
||||
self.fail('Collector did not reach expected state')
|
||||
|
||||
def ready(self):
|
||||
return self.wait_for(lambda: (self.run_dir / 'collector-lock/reader').exists())
|
||||
|
||||
def signal(self, pid, signal='TERM'):
|
||||
return subprocess.run([*self.shell, '-c', 'kill -"$1" "$2"', '--', signal, str(pid)],
|
||||
env=self.env, capture_output=True, timeout=10)
|
||||
|
||||
def stop_processes(self):
|
||||
lock = self.run_dir / 'collector-lock/pid'
|
||||
if lock.exists():
|
||||
self.signal(lock.read_text().strip())
|
||||
for proc, output in self.processes:
|
||||
try:
|
||||
proc.wait(timeout=15)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=5)
|
||||
output.close()
|
||||
|
||||
def send(self, *lines):
|
||||
result = subprocess.run([*self.shell, '-c', 'cat >"$TEST_INPUT"'], input='\n'.join(lines) + '\n',
|
||||
encoding='utf-8', env=self.env, capture_output=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
|
||||
def test_actual_fifo_heartbeat_freshness_sample_and_batch_flush(self):
|
||||
proc = self.start(); self.ready()
|
||||
good = 'Thu Oct 15 22:52:36 2026 user.notice owrt-remote: Heartbeat: OK; Hub=https://hub.example; WAN_direct=1; WAN_active=wan'
|
||||
self.send(good)
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '1')
|
||||
self.send(good.replace('22:52:36', '22:53:08'))
|
||||
self.wait_for(lambda: self.status()['suppressed_events'] == '1')
|
||||
state = self.status()
|
||||
self.assertEqual(state['collector_state'], 'receiving')
|
||||
self.assertEqual(state['received_lines'], '2')
|
||||
self.assertGreater(int(state['last_input_epoch']), 0)
|
||||
self.assertEqual(state['committed_bytes'], '0')
|
||||
(self.root / 'clock').write_text('1301\n')
|
||||
self.send(good.replace('22:52:36', '22:57:38'), 'owrt-remote: panel action after heartbeat')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '3')
|
||||
read = self.call('read').stdout
|
||||
self.assertEqual(read.count('Heartbeat: OK;'), 2)
|
||||
self.assertIn('panel action after heartbeat', read)
|
||||
self.flush()
|
||||
self.assertEqual(self.status()['pending_bytes'], '0')
|
||||
self.assertIn(b'panel action after heartbeat', b''.join(p.read_bytes() for p in self.journal.glob('part.*')))
|
||||
self.assertIsNone(proc.poll())
|
||||
|
||||
def test_duplicate_collect_does_not_remove_owner_fifo(self):
|
||||
proc = self.start(); self.ready()
|
||||
owner = (self.run_dir / 'collector-lock/pid').read_text()
|
||||
fifo = list(self.run_dir.glob('stream.*'))
|
||||
other = self.start(); self.assertEqual(other.wait(timeout=15), 0)
|
||||
self.assertEqual((self.run_dir / 'collector-lock/pid').read_text(), owner)
|
||||
self.assertEqual(list(self.run_dir.glob('stream.*')), fifo)
|
||||
self.send('owrt-remote: owner still receives')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '1')
|
||||
self.assertIsNone(proc.poll())
|
||||
|
||||
def test_reader_death_exits_nonzero_and_retains_journal_for_restart(self):
|
||||
proc = self.start(); self.ready()
|
||||
self.send('owrt-remote: before logread failure')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '1')
|
||||
reader = (self.run_dir / 'collector-lock/reader').read_text().split()[0]
|
||||
self.signal(reader, 'KILL')
|
||||
self.assertNotEqual(proc.wait(timeout=20), 0)
|
||||
self.assertFalse((self.run_dir / 'collector-lock').exists())
|
||||
self.assertFalse(list(self.run_dir.glob('stream.*')))
|
||||
self.assertIn('reader-exited', self.call('diagnostics').stdout)
|
||||
self.assertIn('before logread failure', self.call('read').stdout)
|
||||
new = self.start(); self.ready()
|
||||
self.send('owrt-remote: after reader recovery')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '2')
|
||||
self.assertIsNone(new.poll())
|
||||
|
||||
def test_sigkill_owner_reaps_identified_reader_and_stale_fifo(self):
|
||||
proc = self.start(); self.ready()
|
||||
owner = (self.run_dir / 'collector-lock/pid').read_text().strip()
|
||||
self.send('owrt-remote: retained before killed collector')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '1')
|
||||
old_reader = (self.run_dir / 'collector-lock/reader').read_text().split()[0]
|
||||
self.signal(owner, 'KILL'); proc.wait(timeout=10)
|
||||
new = self.start()
|
||||
self.wait_for(lambda: (self.run_dir / 'collector-lock/pid').exists() and
|
||||
(self.run_dir / 'collector-lock/pid').read_text().strip() != owner)
|
||||
self.wait_for(lambda: (self.run_dir / 'collector-lock/reader').exists() and
|
||||
(self.run_dir / 'collector-lock/reader').read_text().split()[0] != old_reader)
|
||||
self.assertEqual(len(list(self.run_dir.glob('stream.*'))), 1)
|
||||
self.send('owrt-remote: recovered collector')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '2')
|
||||
text = self.call('read').stdout
|
||||
self.assertIn('retained before killed collector', text)
|
||||
self.assertIn('recovered collector', text)
|
||||
self.assertIsNone(new.poll())
|
||||
|
||||
def test_append_failure_is_not_silently_ignored(self):
|
||||
proc = self.start('\nlog_collect_line() { return 1; }\n'); self.ready()
|
||||
self.send('owrt-remote: rejected input')
|
||||
self.assertNotEqual(proc.wait(timeout=20), 0)
|
||||
self.assertIn('collector-append-failed', self.call('diagnostics').stdout)
|
||||
self.assertEqual(self.status()['append_errors'], '1')
|
||||
self.assertFalse((self.run_dir / 'collector-lock').exists())
|
||||
|
||||
def test_term_flushes_and_cleans_children_fifo_and_owner_lock(self):
|
||||
proc = self.start(); self.ready()
|
||||
self.send('owrt-remote: shutdown flush')
|
||||
self.wait_for(lambda: self.status()['accepted_events'] == '1')
|
||||
owner = (self.run_dir / 'collector-lock/pid').read_text().strip()
|
||||
self.signal(owner)
|
||||
self.assertEqual(proc.wait(timeout=15), 0)
|
||||
self.assertEqual(self.status()['pending_bytes'], '0')
|
||||
self.assertIn('shutdown flush', self.call('read').stdout)
|
||||
self.assertFalse(list(self.run_dir.glob('stream.*')))
|
||||
self.assertFalse((self.run_dir / 'collector-lock').exists())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Deploy only the journal helper/service, preserving data and networking."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
|
||||
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
|
||||
|
||||
class CollectorDeploymentTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory(prefix='collector-deploy-')
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.root = Path(self.tmp.name)
|
||||
self.targets = {'logs.sh': '/usr/lib/owrt-remote/logs.sh', 'logs-init': '/etc/init.d/owrt-remote-logs'}
|
||||
self.initial = {}
|
||||
for item, target in self.targets.items():
|
||||
path = self.root / target.lstrip('/')
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text('#!/bin/sh\necho old-' + item + '-$1\n', newline='\n')
|
||||
self.initial[item] = path.read_bytes()
|
||||
self.backup = self.root / 'backup'
|
||||
self.journal = self.root / 'retained-journal'; self.journal.write_text('keep history')
|
||||
text = (ROOT / 'vps/deploy-log-collector-fix.ps1').read_text(encoding='utf-8')
|
||||
self.script = text.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
mapping = {target: (self.root / target.lstrip('/')).as_posix() for target in self.targets.values()}
|
||||
mapping['/root/owrt-remote-backup-before-log-collector-v117'] = self.backup.as_posix()
|
||||
pattern = '|'.join(re.escape(path) for path in sorted(mapping, key=len, reverse=True))
|
||||
self.script = re.sub(pattern, lambda m: mapping[m.group()], self.script)
|
||||
mocks = r'''
|
||||
ubus() { printf '{"running":true}\n'; }
|
||||
jsonfilter() { printf true; }
|
||||
logread() { return 0; }
|
||||
mkfifo() { return 0; }
|
||||
logger() { printf '%s\n' "$3" >"$TEST_MARKER"; }
|
||||
sleep() { return 0; }
|
||||
'''
|
||||
self.script = self.script.replace('trap cleanup EXIT', mocks + '\ntrap cleanup EXIT')
|
||||
self.env = {**os.environ, 'PATH': str(GIT_BIN) + os.pathsep + os.environ['PATH'],
|
||||
'TEST_MARKER': (self.root / 'marker').as_posix()}
|
||||
|
||||
def deploy(self, action, bad=False, fail=False):
|
||||
stage = self.root / 'stage'; stage.mkdir()
|
||||
if action == 'Install':
|
||||
helper = r'''#!/bin/sh
|
||||
# log_collector_progress()
|
||||
# LOG_HEARTBEAT_SECONDS=300
|
||||
case "$1" in
|
||||
read) cat "$TEST_MARKER" ;;
|
||||
status) echo persistent=1 ;;
|
||||
esac
|
||||
'''
|
||||
if bad: helper = '#!/bin/sh\nif true; then\n'
|
||||
(stage / 'logs.sh').write_text(helper, newline='\n')
|
||||
(stage / 'logs-init').write_text('#!/bin/sh\necho new-service-$1\n', newline='\n')
|
||||
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(stage.as_posix()))
|
||||
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
|
||||
if fail: script = script.replace('cmp "$source_dir/$item" "$path"', 'false')
|
||||
result = subprocess.run([SH], input=script, text=True, encoding='utf-8', capture_output=True, env=self.env, timeout=20)
|
||||
self.assertFalse(stage.exists())
|
||||
self.assertEqual(self.journal.read_text(), 'keep history')
|
||||
return result
|
||||
|
||||
def test_install_and_rollback_keep_first_backup_and_journal(self):
|
||||
for _ in range(2):
|
||||
result = self.deploy('Install')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('LOG_COLLECTOR_ROUTER_Install_OK', result.stdout)
|
||||
for item, initial in self.initial.items(): self.assertEqual((self.backup / item).read_bytes(), initial)
|
||||
result = self.deploy('Rollback')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
for item, target in self.targets.items(): self.assertEqual((self.root / target.lstrip('/')).read_bytes(), self.initial[item])
|
||||
|
||||
def test_missing_backup_is_noop(self):
|
||||
result = self.deploy('Rollback')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('Nothing changed', result.stderr)
|
||||
|
||||
def test_invalid_shell_rejected_before_backup(self):
|
||||
result = self.deploy('Install', bad=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(self.backup.exists())
|
||||
|
||||
def test_failed_verify_restores_previous_code(self):
|
||||
result = self.deploy('Install', fail=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('Restoring journal code', result.stderr)
|
||||
for item, target in self.targets.items(): self.assertEqual((self.root / target.lstrip('/')).read_bytes(), self.initial[item])
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -128,7 +128,7 @@ class PersistentLogsTests(unittest.TestCase):
|
||||
(self.run_dir / 'facts').write_text(''.join('1000\ts256-%035d\n' % i for i in range(63)))
|
||||
self.append('distinct event 63\ndistinct event 64\n')
|
||||
self.assertEqual(len((self.run_dir / 'facts').read_text().splitlines()), 64)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size if (self.run_dir / 'facts').exists() else 0, 4096)
|
||||
|
||||
def test_empty_digest_files_falsely_acked_by_old_hub_are_recovered(self):
|
||||
self.append('retained restart event\n'); self.flush()
|
||||
@@ -242,7 +242,7 @@ class PersistentLogsTests(unittest.TestCase):
|
||||
self.flush()
|
||||
self.assertEqual({p.name: p.read_bytes() for p in parts}, original)
|
||||
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size if (self.run_dir / 'facts').exists() else 0, 4096)
|
||||
first = self.export()
|
||||
self.acknowledge(first)
|
||||
self.flush()
|
||||
@@ -353,6 +353,32 @@ class PersistentLogsTests(unittest.TestCase):
|
||||
self.assertIn('after killed reaper', self.call('read').stdout)
|
||||
self.assertFalse(lock.exists())
|
||||
|
||||
def test_integer_only_sleep_keeps_lock_timeout_bounded_and_quiet(self):
|
||||
self.call('check')
|
||||
source = self.source.split('case "${1:-}" in', 1)[0]
|
||||
commands = r'''
|
||||
log_init
|
||||
mkdir "$RUN_DIR/lock"
|
||||
printf '12345\n' >"$RUN_DIR/lock/pid"
|
||||
sleep() {
|
||||
printf '%s\n' "$1" >>"$RUN_DIR/pauses"
|
||||
if [ "$1" = 0.1 ]; then echo "sleep: invalid number '0.1'" >&2; return 1; fi
|
||||
[ "$1" = 1 ]
|
||||
}
|
||||
kill() { return 0; }
|
||||
if log_lock; then exit 17; fi
|
||||
'''
|
||||
script = self.root / 'integer-sleep.sh'
|
||||
script.write_text(source + commands, encoding='utf-8', newline='\n')
|
||||
result = subprocess.run([*self.shell, script.as_posix()], env=self.env,
|
||||
capture_output=True, text=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(result.stderr, '')
|
||||
pauses = (self.run_dir / 'pauses').read_text().splitlines()
|
||||
self.assertEqual(pauses.count('0.1'), 1, 'probe once per process')
|
||||
self.assertEqual(pauses.count('1'), 10, 'ten-second deadline, no busy spin or 100-second wait')
|
||||
self.assertIn('lock-timeout', self.call('diagnostics').stdout)
|
||||
|
||||
def test_collector_failure_reaps_children_and_removes_fifo(self):
|
||||
# Actual FIFO integration is tested on POSIX; Windows BusyBox has no
|
||||
# FIFO support. Exercise its traps with mocked process/FIFO operations.
|
||||
@@ -373,7 +399,48 @@ wait() { return 0; }
|
||||
self.assertFalse(list(self.run_dir.glob('stream.*')))
|
||||
self.assertFalse((self.run_dir / 'collector-lock').exists())
|
||||
lines = (self.run_dir / 'kills').read_text().splitlines()
|
||||
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 2)
|
||||
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 1)
|
||||
|
||||
def test_heartbeat_sampling_and_freshness_are_independent_of_flash_writes(self):
|
||||
good = 'Thu Oct 15 22:52:36 2026 user.notice owrt-remote: Heartbeat: OK; Hub=https://hub.example; WAN_direct=1; WAN_active=wan\n'
|
||||
self.append(good)
|
||||
self.append(good.replace('22:52:36', '22:53:08'), TEST_UPTIME='1032')
|
||||
state = self.status()
|
||||
self.assertEqual((state['accepted_events'], state['suppressed_events']), ('1', '1'))
|
||||
self.assertEqual(state['committed_bytes'], '0')
|
||||
self.assertGreater(int(state['last_input_epoch']), 0)
|
||||
self.assertEqual(state['collector_state'], 'stopped', 'persistent storage does not imply a running collector')
|
||||
self.append(good.replace('22:52:36', '22:57:38'), TEST_UPTIME='1302')
|
||||
self.assertEqual(self.call('read').stdout.count('Heartbeat: OK;'), 2)
|
||||
self.flush()
|
||||
self.assertEqual(self.status()['pending_bytes'], '0')
|
||||
self.assertGreater(int(self.status()['last_commit_epoch']), 0)
|
||||
|
||||
def test_private_diagnostics_are_bounded_rate_limited_and_no_syslog_feedback(self):
|
||||
self.call('check')
|
||||
source = self.source.split('case "${1:-}" in', 1)[0]
|
||||
commands = r'''
|
||||
log_init
|
||||
logger() { echo forbidden-syslog-feedback >&2; return 1; }
|
||||
i=0
|
||||
while [ "$i" -lt 140 ]; do
|
||||
if [ "$((i % 2))" = 0 ]; then log_diag flush-failed; else log_diag lock-timeout; fi
|
||||
i=$((i + 1))
|
||||
done
|
||||
log_diag lock-timeout
|
||||
'''
|
||||
script = self.root / 'diag-test.sh'
|
||||
script.write_text(source + commands, encoding='utf-8', newline='\n')
|
||||
result = subprocess.run([*self.shell, script.as_posix()], env=self.env,
|
||||
capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(result.stderr, '')
|
||||
diagnostics = self.run_dir / 'diagnostics'
|
||||
self.assertLessEqual(diagnostics.stat().st_size, 4096)
|
||||
self.assertFalse((self.run_dir / 'diagnostics.new').exists())
|
||||
self.assertFalse((self.run_dir / 'diag-lock').exists())
|
||||
self.assertEqual(self.call('diagnostics').stdout, diagnostics.read_text())
|
||||
self.assertFalse((self.journal / 'diagnostics').exists())
|
||||
|
||||
def test_legacy_migration_preserves_identity_ack_and_only_undelivered_bytes(self):
|
||||
legacy = self.root / 'legacy'
|
||||
|
||||
Reference in new issue
Block a user