Release v117 with persistent logs and WAN tunnel recovery

This commit is contained in:
kzolotarev95 committed 2026-10-15 23:08:19 +04:00
1 parent 5f65a3d811
commit b62db40daf
61 files changed
+3080 -438

No files matched your search

+2 -2
View File
@@ -1,4 +1,4 @@
# Heartbeat: HTTPS, авторизация и диагностика (v116)
# Heartbeat: HTTPS, авторизация и диагностика (v117)
## Разбор сбоя 15.10.2026
@@ -82,7 +82,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Обновляются агент и WAN helper. Перезапускается только экземпляр Heartbeat
под procd; Xray и удалённый туннель скрипт не перезапускает. Остановленный
экземпляр остаётся остановленным. Резервная копия первых файлов сохраняется
в `/root/owrt-remote-backup-before-heartbeat-transport-v116`.
в `/root/owrt-remote-backup-before-heartbeat-transport-v117`.
Маркер `HEARTBEAT_TRANSPORT_ROUTER_Install_OK` подтверждает установку кода,
а доступность Hub проверяется отдельно:
+12 -12
View File
@@ -1,4 +1,4 @@
# Фиксация VPS host и независимое соединение Remote через WAN (v116)
# Фиксация VPS host и независимое соединение Remote через WAN (v117)
`vps_host_mode=manual` сохраняет заданный на роутере адрес. Это значение
используется и когда опция отсутствует у старой установки. Только явно
@@ -42,8 +42,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-hub-url-fix.
```
Откат: те же команды с `-Action Rollback`, без `-HubUrl`. Резервные копии:
`/root/owrt-remote-backup-before-hub-url-v116` на роутере и
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v116` на VPS.
`/root/owrt-remote-backup-before-hub-url-v117` на роутере и
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v117` на VPS.
`HEARTBEAT_OK` подтверждает ответ Hub. Если файлы установлены, но Hub недоступен,
скрипт отдельно сообщает `HEARTBEAT_FAILED`; это не подтверждение связи.
@@ -125,21 +125,21 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.
Установка загружает локальные файлы, без Git. До остановки службы проверяются
доступность WAN и поддержка таблицы реальной командой ip на роутере.
На роутер загружается и файл LuCI: версия v116 появляется и в основной
На роутер загружается и файл LuCI: версия v117 появляется и в основной
CGI-панели, и в странице входа в неё через LuCI.
Сначала сохраняются файлы
и конфигурация; после замены проверяется heartbeat/health. При ошибке
автоматически восстанавливается состояние перед текущей попыткой.
Сохранённая резервная копия для ручного отката не перезаписывается.
* Роутер: `/root/owrt-remote-backup-before-host-policy-v116` — агент,
* Роутер: `/root/owrt-remote-backup-before-host-policy-v117` — агент,
helper, init, CGI, UCI-конфигурация и Xray-конфиг (в закрытом каталоге).
* VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v116`.
* VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v117`.
База роутеров и настройки авторизации не заменяются.
Успешная установка заканчивается `HOST_POLICY_ROUTER_V116_OK` на роутере
и `HOST_POLICY_VPS_V116_Install_OK` на VPS. Старые резервные копии предыдущей
версии сохраняются; новые команды отката используют отдельные копии для v116.
Успешная установка заканчивается `HOST_POLICY_ROUTER_V117_OK` на роутере
и `HOST_POLICY_VPS_V117_Install_OK` на VPS. Старые резервные копии предыдущей
версии сохраняются; новые команды отката используют отдельные копии для v117.
### Переход с nftables-маркировки на адресную маршрутизацию
@@ -148,11 +148,11 @@ CGI-панели, и в странице входа в неё через LuCI.
Обновление VPS для этого изменения не требуется.
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v116" -Action Install
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v116" -Action Rollback
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v117" -Action Install
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v117" -Action Rollback
```
Копия: `/root/owrt-remote-backup-before-podkop-compat-v116`.
Копия: `/root/owrt-remote-backup-before-podkop-compat-v117`.
После установки повторите диагностику Podkop. Предупреждение от прежних правил
Remote должно исчезнуть. Настоящие дополнительные правила других приложений
по-прежнему обнаруживаются штатной диагностикой.
+155
View File
@@ -0,0 +1,155 @@
# Persistent client journal (v117 local fix)
The helper is `/usr/lib/owrt-remote/logs.sh`. It changes only Remote's private
journal, not logd, network configuration, policy routing or Xray configuration.
## Storage and limits
- Persistent: normally `/overlay/owrt-remote/logs`, only when `/overlay` is an
actual writable JFFS2/UBIFS/ext4/f2fs/ext2/ext3/btrfs mount. A directly mounted
persistent root uses `/etc/owrt-remote/logs`. Root OverlayFS and tmpfs are
rejected. Existing ancestors/symlinks are resolved against `/proc/mounts`;
the longest containing mount wins. An explicit `OWRT_REMOTE_LOG_DIR` is
validated the same way.
- Runtime: `/tmp/owrt-remote-log-runtime`, which must be on tmpfs. Contains a
buffer <=16 KiB, <=64 hashed dedup entries, a heartbeat fingerprint, PID locks,
one export <=16 KiB, one HTTP body about 22 KiB, one viewer snapshot <=16 KiB
plus a bounded filename list. No whole-journal shell variable or RAM archive.
HTTP response files are limited by a sender-only RLIMIT_FSIZE (32 KiB in
BusyBox ash); this does not alter heartbeat or Xray process limits.
- Flash data: <=2 MiB across <=256 immutable chunks, each <=16 KiB. This is an
upper bound, not guaranteed capacity: many small chunks reach the file-count
bound first. Metadata, filesystem allocation units and the one quarantined
damaged chunk add small overhead beyond the raw-data bound.
- Rotation runs each minute. Acknowledged chunks expire after 24 hours or
earlier to admit new chunks. Unacknowledged chunks do not expire. If the
bounded queue/RAM buffer fills, **new** events are skipped; existing
unacknowledged chunks remain intact. A warning and dropped-event count make
that loss visible. A bounded queue cannot preserve infinitely many events
during an infinitely long outage.
- Default batch flush: once per 60 seconds, at most 16 KiB of new data; explicit
`flush force`, normal service stop and shutdown can flush sooner. There is no
fsync per line. Idle polls do not rewrite persistent state. Metadata and
filesystem write amplification are additional to the data rate; no universal
Flash-lifetime guarantee is possible across devices/filesystems.
- Less than 2 MiB free space or write/mount failure stops Flash writes and
switches this boot to a warning-marked RAM fallback: <=64 KiB data in <=8
chunks +16 KiB buffer. The old Flash queue is left intact. Restore storage
and reboot to resume that queue. No automatic mid-boot merging of two journal
identities, which could acknowledge the wrong records.
## Crash and synchronization rules
Each chunk filename includes journal identity, start/end byte offsets and a validated SHA-256/MD5/POSIX cksum digest. Data is written to `chunk.new` on the persistent filesystem and renamed;
state is written to `state.new` on that filesystem and atomically replaced.
Batch sync (`sync -f`, or BusyBox `sync` fallback) completes durability. A small
RAM in-flight marker reconciles a killed flusher before another append. At
boot, checksums reject partial/corrupt chunks, incomplete temporary files are
removed, and a completed rename beyond the saved cursor is recovered. One
corrupt chunk can be retained as `damaged.log` for inspection. A reboot never
clears a valid journal.
Last-minute events still in RAM can be lost on sudden power removal. This is
the explicit buffering tradeoff, not a claim of per-line durable logging.
Only a successful authenticated Hub reply advances the ack. The recent ack is
checkpointed with the next periodic batch before pruning; if power is lost
first, records may replay with the same journal/offset. Existing Hub ingestion
suppresses these duplicates. Packets are <=16 KiB raw and transport them from
files without an entire packet in a shell variable. One `logs-sync` sender is
allowed; stale locks and stale payloads recover on the next attempt. Explicit
local/Hub clear intentionally discards both confirmed and unconfirmed data.
Log-sender curl temporaries use a private runtime subdirectory, reclaimed on
the next attempt after SIGKILL; heartbeat's temporary directory is untouched.
The local installer replaces the helper, log service, agent, CGI and LuCI view. It
does not restart the tunnel, heartbeat or network. With the previous collector
stopped, it streams the old bounded RAM journal into Flash chunks, preserving
its journal ID, total cursor, ack and clear token. Already delivered bytes are
not resent; undelivered records remain eligible for delivery. No FIFO, lock or
temporary request is migrated. A same-filesystem staging manifest allows a
power-interrupted migration to finish even after the old /tmp has disappeared.
Migration needs 6 MiB free Flash; if it cannot safely migrate legacy data, the
local installer fails and restores code instead of silently deleting history.
With no usable Flash, only a bounded legacy tail is retained in RAM with an
explicit loss warning, and the former large RAM files are removed. An ordinary
upgrade stops the previous collector before replacing files and invokes this
migration on log-service startup. Rollback restores code/service state and
leaves Flash logs intact for recovery; the old helper starts a fresh RAM log.
## Automated checks
```powershell
$env:OWRT_LOG_TEST_SHELL = "$PWD\.test-output\busybox\busybox64.exe"
python -m unittest discover -s tests -p test_persistent_logs.py -v
Remove-Item Env:OWRT_LOG_TEST_SHELL
python -m unittest discover -s tests -p test_global_logs.py -v
python -m unittest discover -s tests -p test_router_log_transport.py -v
```
Tests use real shell functions and disposable mount/filesystem fixtures. The
BusyBox Windows build exercises ash and its applets; it is **not** a claim of
testing OpenWrt 24.10.3's exact BusyBox/kernel/UBIFS or physical power loss.
## Device checks before release
Install locally, then inspect from Windows PowerShell:
```powershell
ssh root@192.168.2.1 'sh /usr/lib/owrt-remote/logs.sh status; df -h /overlay /tmp; free; ubus call service list; owrt-remote heartbeat-diagnostics'
```
Expect `persistent=1`, a persistent storage path and tmpfs runtime. Confirm the
mount explicitly using `mount` and verify `state` and `part.*` appear on Flash.
Append a marker and flush before a user-planned reboot:
```powershell
ssh root@192.168.2.1 'printf "persistent-check\n" | sh /usr/lib/owrt-remote/logs.sh append; sh /usr/lib/owrt-remote/logs.sh flush force; sh /usr/lib/owrt-remote/logs.sh status'
```
After reboot, `read` must still show the marker and the same journal ID. Test an
outage only on a controlled router with a separate management path; do not
change firewall/WAN settings on a remote production router merely for this
test. During outage, watch data/chunk/runtime bounds; after reconnect, watch
`ack_bytes` advance without duplicate Hub entries. Launch concurrent
`owrt-remote logs-sync` calls and inspect service/process counts. Confirm UI,
heartbeat, Xray and WAN direct remain functional. Actual sustained RAM,
Flash-write and reboot observations must be collected on the device.
## VPS archive and malformed-fragment recovery
The Hub retains its existing persistent SQLite archive, normally at
`/var/lib/owrt-remote/hub.db`, <=2 MiB of content per router. No new RAM archive
or whole-journal queue is introduced on the VPS. The live view still transfers
only the last 64 KiB. A storage diagnostic resolves DB symlinks and the longest
containing Linux mount, marking tmpfs/ramfs as nonpersistent. The dedicated
`deploy-persistent-logs-vps.ps1` verifies the running Hub's actual DB path and
writable disk before installing code. It preserves the database and all settings;
rollback replaces code only. It refuses installation over an unconfirmed/RAM DB
rather than automatically relocating a live database.
Some OpenWrt builds lack the optional `cksum` applet. The previous local helper
failed to check its exit status and could publish names ending in an empty digest.
Export skipped these files, but the empty status packet claimed the full cursor;
the previous Hub accepted it and falsely acknowledged missing data. Regression
checks now simulate missing cksum and failed checksum tools. New chunks use an
available SHA-256/MD5/POSIX checksum with validated output. An empty status packet
cannot advance the delivered cursor on either side.
On upgrade, a versioned recovery marker forces inspection even during the same
boot. Retained malformed fragments of the expected byte length are copied into a
same-filesystem recovery generation with real digests and a committed migration
manifest. Old falsely acknowledged offsets cannot suppress that generation.
Recovery requires enough free disk for the bounded copy and retains originals
until the new manifest is committed. This recovers retained bytes, not already
removed files, and cannot retroactively verify their former integrity. A warning
records that limitation. A new generation can repeat already delivered history
if mixed with affected fragments; normal retries remain deduplicated by offsets.
Run the real client spool-to-Hub ingestion regression as well as storage/deploy
checks:
```powershell
python -m unittest discover -s tests -p test_router_log_archive.py -v
python -m unittest discover -s tests -p test_persistent_logs_vps_deploy.py -v
```
+3 -3
View File
@@ -1,4 +1,4 @@
# reCAPTCHA v3 в OpenWrt Remote Hub v116
# reCAPTCHA v3 в OpenWrt Remote Hub v117
В меню владельца, в блоке «Капча на экране входа», доступны цифры,
Google reCAPTCHA v2 Checkbox и Google reCAPTCHA v3. Установка обновления
@@ -74,8 +74,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-recaptcha-v3
непосредственно перед попыткой. Резервные копии при повторном запуске
не перезаписываются:
* `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v116`
* `/opt/owrt-remote/captcha-before-recaptcha-v3-v116.json` (режим 0600).
* `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v117`
* `/opt/owrt-remote/captcha-before-recaptcha-v3-v117.json` (режим 0600).
Скрипт рассчитан на стандартные пути установленного Hub:
`/opt/owrt-remote/owrt-remote-hub.py`, `/var/lib/owrt-remote/hub-auth.json`
+3 -3
View File
@@ -1,6 +1,6 @@
# Аудит ресурсов OpenWrt Remote Hub v116
# Аудит ресурсов OpenWrt Remote Hub v117
Исправления внесены поверх `7f809fd`. Версия панели остаётся v116.
Исправления внесены поверх `7f809fd`. Версия панели остаётся v117.
## SQLite
@@ -115,7 +115,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix
Установщик загружает локальный файл без Git, запускает изолированные тесты,
сохраняет исходный серверный файл:
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v116`,
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v117`,
атомарно заменяет Python файл, перезапускает `owrt-remote` и проверяет health.
Существующая резервная копия не перезаписывается. При ошибке установки
восстанавливается файл, работавший непосредственно перед текущей попыткой.
+4 -4
View File
@@ -1,4 +1,4 @@
# Виджеты ресурсов VPS (v116)
# Виджеты ресурсов VPS (v117)
Три карточки ЦП, ОЗУ и ДИСК находятся над кнопками «Рестарт Xray VPS»,
профилем и выходом. На ПК ширина и высота совпадают с кнопками под ними.
@@ -23,7 +23,7 @@ vCPU и объёмы занятой/общей памяти и диска.
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-vps-widgets.ps1" -Vps "root@193.233.82.38" -Action Install
```
После `VPS_WIDGETS_V116_Install_OK` обновите страницу через Ctrl+F5.
После `VPS_WIDGETS_V117_Install_OK` обновите страницу через Ctrl+F5.
Скрипт загружает локальный файл, проверяет его до замены, перезапускает
службу и проверяет health. При ошибке после замены восстанавливает файл
перед текущей попыткой. База данных и настройки не заменяются.
@@ -33,8 +33,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
```
Откат использует неизменяемую копию первого состояния до установки:
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v116`.
Успешное завершение: `VPS_WIDGETS_V116_Rollback_OK`.
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v117`.
Успешное завершение: `VPS_WIDGETS_V117_Rollback_OK`.
## Локальные проверки
+2 -2
View File
@@ -1,4 +1,4 @@
# WAN IPv4 DNS fallback (local v116 fix)
# WAN IPv4 DNS fallback (local v117 fix)
`files/usr/lib/owrt-remote/wan.sh` now explicitly queries
`nslookup -type=A -timeout=3 -retry=1 "$host" "$dns"`.
@@ -58,7 +58,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
```
The installer saves the first backup at
`/root/owrt-remote-backup-before-wan-dns-v116/wan.sh`, checks shell syntax and
`/root/owrt-remote-backup-before-wan-dns-v117/wan.sh`, checks shell syntax and
reloads only the supervised heartbeat instance so it reads the new helper.
It does not restart Xray or regenerate configuration/routing. Installation
verification/reload failures restore the file from that attempt.
+2 -2
View File
@@ -1,4 +1,4 @@
# Несколько WAN интерфейсов и резервирование (v116)
# Несколько WAN интерфейсов и резервирование (v117)
Кнопка **Выбрать интерфейсы** раскрывает список с галочками. Можно отметить
один, два, три и больше интерфейсов. Список содержит имена
@@ -46,7 +46,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Скрипт устанавливает CGI, LuCI, агент, WAN helper и hotplug вместе. До замены
сохраняются прежние файлы, UCI-настройки и Xray config в
`/root/owrt-remote-backup-before-wan-multi-v116`.
`/root/owrt-remote-backup-before-wan-multi-v117`.
Успех подтверждает `WAN_MULTI_ROUTER_Install_OK`. После установки обновить
панель роутера через Ctrl+F5, отметить интерфейсы и сохранить выбор.
VPS обновлять не требуется. Старый `deploy-wan-interface-picker.ps1`
+96
View File
@@ -0,0 +1,96 @@
# Восстановление Xray после пропажи выбранного WAN (v117)
Сообщения `No selected WAN has a physical IPv4 default route`, ошибки
генерации конфигурации и `crash loop 6 crashes` описывают последовательность:
временно исчез маршрут на выбранном uplink, каждый запуск агента завершался,
а procd прекратил повторные попытки запуска Xray. Затем Heartbeat смог
восстановиться самостоятельно. Он делает HTTP-запрос, поэтому его успех
не подтверждает работу двух обратных соединений Xray для Админки и SSH.
В прежнем `wan_reconcile_xray` остановленный Xray игнорировался; при возврате
того же интерфейса также ничего не происходило. Это подтверждённый дефект
восстановления. Причину исчезновения исходного маршрута и фактическое
текущее состояние Xray на удалённом устройстве только эти логи не определяют.
## Исправление
- Перед генерацией JSON служебный процесс ожидает выбранный WAN с физическим
IPv4 default route. Проверка каждые 5 секунд, одна запись о начале ожидания
и одна о возвращении WAN. Конфигурация в ожидании не заменяется.
- Ошибки UUID, JSON или проверки Xray при доступном WAN не маскируются
ожиданием сети: генерация по-прежнему завершается с настоящей ошибкой.
- Heartbeat не перезапускает всю службу, пока все выбранные WAN недоступны.
- После подготовки WAN проверяются процесс Xray и состояние procd. Если
Xray отсутствует и его instance не работает, но служба включена и Heartbeat
работает, повторно регистрируется только instance `xray` через `service add`.
Работающий процесс запуска, который ожидает WAN или готовит Xray, не трогается.
- Живой Heartbeat и остальные instance не удаляются. При прежнем здоровом
туннеле на том же интерфейсе перезапуск не запрашивается.
- Блокировка в RAM исключает одновременные запросы восстановления; PID
завершившегося worker освобождается. Повторные запросы ограничены интервалом
300 секунд, в том числе при ошибке ubus. Успешный ответ ubus означает принятие
запроса запуска, а не доказанную доступность Админки.
- Явная остановка через панель (`enabled=0`) или остановка всей службы
не отменяется автоматикой. Невыбранные интерфейсы не используются.
- Источники установки, настройки LAN, DNS, Podkop и логика WAN policy routing
не меняются этим исправлением.
Формат `service add` и строковых параметров respawn сверялся с исходниками:
- https://github.com/openwrt/procd/blob/master/service/service.c
- https://github.com/openwrt/procd/blob/master/service/instance.c
- https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/procd/files/procd.sh
## Установка и откат
Windows PowerShell 5.1, роутер пользователя:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Install
```
Скрипт обновляет агент и WAN helper парой, сохраняет первую копию в
`/root/owrt-remote-backup-before-wan-recovery-v117`, проверяет shell-синтаксис
и побайтовое совпадение. Затем перезагружает только работающий Heartbeat,
чтобы в RAM не остался прежний shell с уже загруженными функциями.
Туннель при исправном Xray не перезапускается. Остановленный Heartbeat
не запускается принудительно. Ошибка установки восстанавливает файлы
до текущей попытки. UCI и рабочий JSON скрипт установки не заменяет;
обычный запуск Xray выполняет штатную проверку и генерацию конфигурации.
Установка подтверждается `WAN_RECOVERY_ROUTER_Install_OK`, это подтверждение
установки кода. Проверка состояния:
```powershell
ssh root@192.168.2.1 'owrt-remote status; owrt-remote heartbeat-diagnostics; logread -e owrt-remote | tail -n 60'
```
Откат к коду до первого применения:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
```
Для роутера друга нужен его собственный доступный SSH-адрес, VPN или человек
в локальной сети. Через оборванные обратные SSH/HTTP-туннели установить код
невозможно. Если WAN уже вернулся, для разового восстановления на самом
роутере можно выполнить `/etc/init.d/owrt-remote restart`, затем проверить
`owrt-remote status`. Сам по себе этот перезапуск не устанавливает исправление.
## Локальная проверка
`test_wan_recovery.py` моделирует длительное отсутствие маршрута, возврат
того же интерфейса, остановленный Xray при работающем Heartbeat, начальную
подготовку, исправный туннель, ручную остановку, параллельные запросы,
оставшуюся блокировку, отказ ubus и настоящую ошибку генерации JSON.
Поддерживает `OWRT_RECOVERY_TEST_SHELL` для BusyBox ash.
`test_wan_recovery_deploy.py` проверяет установку, повторную установку,
сохранение первой резервной копии, откат, отсутствие копии, повреждённый
payload, отказ проверки и отказ перезапуска Heartbeat без реального SSH.
Дополнительно запускаются регрессии WAN failover, host policy, HTTP/HTTPS
авторизации и перенаправлений. Проверка BusyBox ash выполняется локальной
Windows-сборкой BusyBox; OpenWrt 24.10.3 и настоящий procd на устройстве
требуют проверки пользователем. Прямой WAN на удалённом устройстве не
отключать для искусственного теста без независимого канала доступа.
+3 -3
View File
@@ -1,4 +1,4 @@
# Исправление запуска прямого WAN в v116
# Исправление запуска прямого WAN в v117
На проблемном роутере обычный `wan` был поднят, имел IPv4-адрес,
маршрут и рабочий DNS. Проверка `ip -4 route show table 210` возвращала
@@ -16,7 +16,7 @@ instance после шести падений. Из-за остановленн
конфигурации останавливают агент до запуска Xray.
Изменены только `files/usr/lib/owrt-remote/wan.sh` и
`files/usr/sbin/owrt-remote`. Версия остаётся v116. Настройки Podkop,
`files/usr/sbin/owrt-remote`. Версия остаётся v117. Настройки Podkop,
LAN и источники установки GitHub/GitLab не меняются. Обновление VPS
для этого исправления не требуется.
@@ -28,7 +28,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Скрипт перед заменой проверяет синтаксис и подготовку WAN новым helper,
сохраняет агент, helper, UCI и Xray-конфигурацию в
`/root/owrt-remote-backup-before-wan-stability-v116`. Сохранённая копия
`/root/owrt-remote-backup-before-wan-stability-v117`. Сохранённая копия
не перезаписывается при повторной установке. После замены выполняются
перезапуск, проверка процесса Xray и настоящий heartbeat до Hub.
При ошибке восстанавливается состояние до текущей попытки.
+6 -6
View File
@@ -1,4 +1,4 @@
# Применение OpenWrt config через панель роутера (v116)
# Применение OpenWrt config через панель роутера (v117)
При отправке textarea браузер кодирует переносы строк как CRLF. CGI
декодировал их без нормализации и записывал CR в исполняемый shell-скрипт.
@@ -22,7 +22,7 @@ CRLF и LF, heredoc, пустые строки, кодированные сим
Скрипт обновляет только CGI на роутере. VPS обновлять для этого фикса
не требуется. До замены сохраняется первая резервная копия CGI:
`/root/owrt-remote-backup-before-xray-apply-v116/owrt-remote`.
`/root/owrt-remote-backup-before-xray-apply-v117/owrt-remote`.
Из PowerShell Windows 11; замените IP, если адрес роутера отличается:
@@ -53,7 +53,7 @@ python tests/test_xray_apply.py -v
Переносы строк в формах описаны в [стандарте HTML](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html).
## Мобильная вставка команд (локальный фикс v116)
## Мобильная вставка команд (локальный фикс v117)
Воспроизведён отдельный случай `rc=255`: в строке
`uci set owrtremote.main.router_name=‘Дом у моря’` типографские кавычки
@@ -77,7 +77,7 @@ python tests/test_xray_apply.py -v
Обновляется только `/www/cgi-bin/owrt-remote` на роутере, без перезапуска
служб и изменений настроек при установке. Резервная копия отдельная:
`/root/owrt-remote-backup-before-xray-mobile-v116/owrt-remote`.
`/root/owrt-remote-backup-before-xray-mobile-v117/owrt-remote`.
Установка из PowerShell Windows 11:
@@ -101,7 +101,7 @@ python tests/test_xray_mobile_deploy.py -v
```
## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v116)
## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v117)
В предоставленном журнале Xray 26.3.27 запускается успешно, а завершающая
команда `owrt-remote heartbeat` возвращает `rc=1` с `Failed to allocate
@@ -153,7 +153,7 @@ CGI получает этап выполнения от агента. Ошибк
Новый скрипт `vps/deploy-xray-validation-fix.ps1` обновляет CGI и агент,
не пересобирает настройки и не перезапускает службы при установке/откате.
Первая резервная копия обоих файлов:
`/root/owrt-remote-backup-before-xray-validation-v116`. Если установка
`/root/owrt-remote-backup-before-xray-validation-v117`. Если установка
оборвалась или проверка файлов не прошла, возвращаются файлы до этой попытки.
На роутере друга нужно указать его доступный SSH-адрес вместо локального IP.
+12 -90
View File
@@ -1,4 +1,4 @@
"""Exercise the real RAM journal and authorized router CGI with temporary files."""
"""Exercise the bounded persistent journal and authorized router CGI with temporary files."""
import concurrent.futures
import base64
import json
@@ -22,12 +22,14 @@ class GlobalLogTests(unittest.TestCase):
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.log_dir = self.root / 'journal'
self.run_dir = self.root / 'runtime'
self.helper = self.root / 'logs.sh'
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
self.helper_source = source
source = source.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
source = source.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.cgi = self.root / 'cgi'
self.cgi.write_text((ROOT / 'files/www/cgi-bin/owrt-remote').read_text(encoding='utf-8'),
@@ -35,12 +37,15 @@ class GlobalLogTests(unittest.TestCase):
self.key = self.root / 'key'
self.key.write_text('test-key', encoding='utf-8')
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': self.log_dir.as_posix(),
'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
'OWRT_REMOTE_LOG_HELPER': self.helper.as_posix(), 'TMPDIR': self.root.as_posix(),
'KEY_FILE': self.key.as_posix(), 'TEST_UPTIME': '1000'}
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def helper_call(self, command, text='', *args, env=None):
if command == 'export':
self.helper_call('flush', '', 'force', env=env)
result = subprocess.run([SH, self.helper.as_posix(), command, *args], input=text,
encoding='utf-8', capture_output=True, env=env or self.env, timeout=20)
self.assertEqual(result.returncode, 0, result.stderr)
@@ -61,8 +66,9 @@ class GlobalLogTests(unittest.TestCase):
self.assertIn('Xray connected', downloaded)
self.assertIn('Cache-Control: no-store', downloaded)
# The bounded read is independent of append filtering; supply an existing large journal.
with (self.log_dir / 'current.log').open('a', encoding='utf-8') as journal:
journal.write(''.join(f'{i}: ' + 'y' * 1024 + '\n' for i in range(70)))
state = (self.log_dir / 'state').read_text().split()
for i in range(5):
(self.log_dir / f'part.{state[0]}.{i:018d}.{i+1:018d}.1').write_text('y' * 15000 + '\n')
live = self.cgi_call('logs_live').split('\n\n', 1)[1]
# Git for Windows can translate CRLF inside native pipes.
self.assertLessEqual(len(live.encode()), 65536)
@@ -141,32 +147,6 @@ class GlobalLogTests(unittest.TestCase):
lines = self.helper_call('read').splitlines()
self.assertEqual([line.split(']', 1)[0] for line in lines], ['[ERROR', '[ERROR', '[ERROR', '[WARN'])
def test_persistent_storage_is_rejected_before_any_file_is_written(self):
mounts = self.root / 'mounts'
mounts.write_text('/dev/root / overlay rw 0 0\n', encoding='utf-8')
source = self.helper_source.replace('/proc/mounts', mounts.as_posix())
self.helper.write_text(source, encoding='utf-8', newline='\n')
env = self.env
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not be written\n',
encoding='utf-8', capture_output=True, env=env, timeout=20)
self.assertNotEqual(result.returncode, 0)
self.assertIn('refusing persistent storage', result.stderr)
self.assertFalse(self.log_dir.exists())
self.env = env
self.assertIn('503 Service Unavailable', self.cgi_call('logs_live'))
self.assertFalse(self.log_dir.exists())
# The same location is permitted only when its containing mount is tmpfs.
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix() + ' tmpfs rw 0 0\n', encoding='utf-8')
self.helper_call('append', 'RAM fact\n')
self.assertEqual(self.helper_call('read'), '[OK] RAM fact\n')
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix()
+ ' tmpfs rw 0 0\n/dev/flash ' + self.log_dir.as_posix()
+ ' overlay rw 0 0\n', encoding='utf-8')
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not enter nested Flash mount\n',
encoding='utf-8', capture_output=True, env=env, timeout=20)
self.assertNotEqual(result.returncode, 0)
self.assertEqual((self.log_dir / 'current.log').read_text(), '[OK] RAM fact\n')
def test_clear_requires_post_and_correct_key(self):
self.helper_call('append', 'retain this event\n')
self.assertIn('Доступ запрещен', self.cgi_call('logs_download', key='wrong'))
@@ -178,13 +158,6 @@ class GlobalLogTests(unittest.TestCase):
self.helper_call('append', 'new event\n')
self.assertEqual(self.helper_call('read'), '[OK] new event\n')
def test_retention_uses_uptime_and_clears_on_reboot(self):
self.helper_call('append', 'old event\n')
self.assertIn('old event', self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87399'}))
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87400'}), '')
self.helper_call('append', 'after daily clear\n', env={**self.env, 'TEST_UPTIME': '87401'})
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '20'}), '')
def test_export_acknowledges_only_received_bytes_and_remote_clear_is_idempotent(self):
self.helper_call('append', 'First event\n')
first = json.loads(self.helper_call('export'))
@@ -207,39 +180,6 @@ class GlobalLogTests(unittest.TestCase):
self.helper_call('ack', '', second['journal_id'], str(second['to']), 'clear-123')
self.assertIn('After clear', self.helper_call('read'), 'retrying clear must not clear new events')
def test_export_catches_up_in_bounded_batches_after_rotation(self):
self.helper_call('clear')
raw = b'[OK] ' + b'x' * 100000 + b'\n'
(self.log_dir / 'current.log').write_bytes(raw)
first = json.loads(self.helper_call('export'))
first_raw = base64.b64decode(first['data'])
self.assertLessEqual(len(first_raw), 65536)
self.helper_call('ack', '', first['journal_id'], str(first['to']))
second = json.loads(self.helper_call('export'))
self.assertEqual(first_raw + base64.b64decode(second['data']), raw)
# Bytes dropped by local rotation are detected by the server from the gap.
(self.log_dir / 'current.log').write_bytes(b'[OK] Latest\n')
(self.log_dir / 'total').write_text('200000')
late = json.loads(self.helper_call('export'))
self.assertEqual(late['from'], 200000 - len(b'[OK] Latest\n'))
self.assertEqual(base64.b64decode(late['data']), b'[OK] Latest\n')
def test_rotation_retains_recent_log_and_clear_removes_both(self):
self.helper_call('clear')
current = self.log_dir / 'current.log'
current.write_text('A' * 1048576, encoding='utf-8')
self.helper_call('append', 'second file\n')
self.assertTrue((self.log_dir / 'previous.log').exists())
self.assertEqual(current.read_text(), '[OK] second file\n')
current.write_text('B' * 1048576, encoding='utf-8')
self.helper_call('append', 'third file\n')
output = self.helper_call('read')
self.assertNotIn('A', output)
self.assertTrue(output.endswith('third file\n'))
self.helper_call('clear')
self.assertFalse((self.log_dir / 'previous.log').exists())
self.assertEqual(current.read_text(), '')
def test_secrets_are_redacted_before_storage_and_html_is_literal(self):
text = ('Heartbeat OK\nuci set owrtremote.main.hub_token=SECRET_TOKEN\n'
'Authorization: Bearer SECRET_AUTH\nhttps://hub.test/?key=SECRET_KEY\n'
@@ -256,11 +196,11 @@ class GlobalLogTests(unittest.TestCase):
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
list(pool.map(lambda i: self.helper_call('append', f'event-{i}\n'), range(12)))
self.assertEqual(set(self.helper_call('read').splitlines()), {f'[OK] event-{i}' for i in range(12)})
self.assertFalse((self.log_dir / 'lock').exists())
self.assertFalse((self.run_dir / 'lock').exists())
def test_stale_lock_recovers(self):
self.helper_call('clear')
lock = self.log_dir / 'lock'
lock = self.run_dir / 'lock'
lock.mkdir()
(lock / 'pid').write_text('99999999')
self.helper_call('append', 'recovered\n')
@@ -272,28 +212,10 @@ class GlobalLogTests(unittest.TestCase):
def test_empty_lock_left_by_killed_writer_recovers(self):
self.helper_call('clear')
(self.log_dir / 'lock').mkdir()
(self.run_dir / 'lock').mkdir()
self.helper_call('append', 'after empty lock\n')
self.assertEqual(self.helper_call('read'), '[OK] after empty lock\n')
def test_collector_failure_cleans_both_children_and_fifo(self):
source = self.helper.read_text(encoding='utf-8')
mocks = '''
log_init() { return 0; }
mkfifo() { return 0; }
logread() { return 0; }
log_lock() { return 1; }
kill() { printf 'KILL=%s\\n' "$*"; }
wait() { return 0; }
rm() { printf 'REMOVE=%s\\n' "$*"; }
'''
self.helper.write_text(source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1),
encoding='utf-8', newline='\n')
result = subprocess.run([SH, self.helper.as_posix(), 'collect'], capture_output=True,
encoding='utf-8', env=self.env, timeout=10)
self.assertNotEqual(result.returncode, 0)
self.assertEqual(result.stdout.count('KILL='), 2)
self.assertIn('REMOVE=-f ' + self.log_dir.as_posix() + '/stream.', result.stdout)
if __name__ == '__main__':
+5 -4
View File
@@ -43,7 +43,7 @@ class GlobalLogsDeploymentTests(unittest.TestCase):
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
mappings['/root/owrt-remote-backup-before-global-logs-stop-v116'] = self.backup.as_posix()
mappings['/root/owrt-remote-backup-before-global-logs-stop-v117'] = self.backup.as_posix()
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
# Existing sandbox service scripts only print markers. Mock readiness.
@@ -55,7 +55,7 @@ jsonfilter() { return 0; }
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
'''
self.script = self.script.replace('for item in cgi agent init;', mocks + '\nfor item in cgi agent init;', 1)
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix()}
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
@@ -75,6 +75,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
if item == 'logs.sh':
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
if item == 'logs-init':
text = '#!/bin/sh\necho logs-service-$1\n'
if bad and item == 'cgi':
@@ -103,7 +104,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
self.assertIn('GLOBAL_LOGS_ROUTER_Install_OK', result.stdout)
for item in ('cgi', 'luci'):
installed = self.root / self.targets[item].lstrip('/')
self.assertIn('v116', installed.read_text(encoding='utf-8'))
self.assertIn('v117', installed.read_text(encoding='utf-8'))
for item, data in self.initial.items():
self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
result = self.deploy('Rollback')
@@ -130,7 +131,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
self.assert_initial()
def test_rollback_restores_existing_log_service_and_helper(self):
# This fix is normally applied on top of v116's already installed journal.
# This fix is normally applied on top of v117's already installed journal.
for item in ('logs.sh', 'logs-init'):
target = self.root / self.targets[item].lstrip('/')
target.parent.mkdir(parents=True, exist_ok=True)
+2 -2
View File
@@ -20,7 +20,7 @@ class VpsLogDeployTests(unittest.TestCase):
self.target = self.root / 'hub.py'
self.previous = b'print("previous Hub")\n'
self.target.write_bytes(self.previous)
self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v116'
self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v117'
source = (ROOT / 'vps/deploy-global-logs-vps.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
@@ -55,7 +55,7 @@ sleep() { return 0; }
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('GLOBAL_LOGS_VPS_V116_Install_OK', result.stdout)
self.assertIn('GLOBAL_LOGS_VPS_V117_Install_OK', result.stdout)
self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
self.assertEqual(self.backup.read_bytes(), self.previous)
result = self.deploy('Rollback')
+1 -1
View File
@@ -25,7 +25,7 @@ class HeartbeatTransportDeploymentTests(unittest.TestCase):
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-heartbeat-transport-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v116",
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
+423
View File
@@ -0,0 +1,423 @@
"""Persistent spool recovery/resource limits using the real shell helper.
Mount fixtures replace /proc/mounts only in disposable copies. Set
OWRT_LOG_TEST_SHELL to a BusyBox executable to exercise its actual ash/applets.
"""
import base64
import concurrent.futures
import json
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import time
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
BUSYBOX = os.environ.get('OWRT_LOG_TEST_SHELL', '')
class PersistentLogsTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='persistent-logs-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name).resolve()
self.run_dir = self.root / 'runtime'
self.flash = self.root / 'flash'
self.flash.mkdir()
self.helper = self.root / 'logs.sh'
self.mounts = self.root / 'mounts'
self.root_mount = self.root.as_posix()
self.flash_mount = self.flash.as_posix()
if os.name == 'nt' and not BUSYBOX:
# Git's bundled readlink may emit native C:/ or MSYS /c paths.
# The fixture must use what the actual applet returns, not assume it.
env = {**os.environ, 'PATH': str(GIT_BIN) + os.pathsep + os.environ['PATH']}
def resolved_mount(path):
return subprocess.check_output([SH, '-c', 'readlink -f "$1"', '--', path.as_posix()],
env=env, text=True).strip()
self.root_mount = resolved_mount(self.root)
self.flash_mount = resolved_mount(self.flash)
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
f'/dev/ubi0_1 {self.flash_mount} ubifs rw 0 0\n')
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
source = source.replace('/proc/mounts', self.mounts.as_posix())
if os.name == 'nt':
source = source.replace('case "$path" in /*)', 'case "$path" in [a-zA-Z]:/*|/*)')
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
source = source.replace('log_epoch() { date +%s; }',
'log_epoch() { echo "${TEST_EPOCH:-' + str(int(time.time())) + '}"; }')
# Real df remains covered in integration tests. Fixtures emulate free space.
source = source.replace('free="$(df -Pk "$LOG_DIR" | awk \'NR==2 {print $4}\')"',
'free="${TEST_FREE_KB:-8192}"')
self.source = source
self.source = self.source.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + (self.root / 'legacy').as_posix())
self.helper.write_text(self.source, encoding='utf-8', newline='\n')
self.env = {**os.environ, 'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
'OWRT_REMOTE_LOG_DIR': (self.flash / 'journal').as_posix()}
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
self.shell = [BUSYBOX, 'ash'] if BUSYBOX else [SH]
if BUSYBOX:
self.env['BB_OVERRIDE_APPLETS'] = '1'
def call(self, command, text='', *args, check=True, **flags):
result = subprocess.run([*self.shell, self.helper.as_posix(), command, *args],
input=text, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25)
if check:
self.assertEqual(result.returncode, 0, result.stderr)
return result
def status(self):
return dict(line.split('=', 1) for line in self.call('status').stdout.splitlines())
def flush(self, **flags):
return self.call('flush', '', 'force', **flags)
def export(self):
return json.loads(self.call('export').stdout)
def append(self, text, **flags):
return self.call('append', text, **flags)
def acknowledge(self, batch):
self.call('ack', '', batch['journal_id'], str(batch['to']))
@property
def journal(self):
return self.flash / 'journal'
def reboot(self):
shutil.rmtree(self.run_dir)
def test_persistent_storage_selected_and_survives_runtime_loss(self):
self.append('before reboot\n')
self.flush()
first = self.export()
self.acknowledge(first)
self.flush()
self.reboot()
self.assertEqual(self.status()['persistent'], '1')
self.assertIn('before reboot', self.call('read').stdout)
after = self.export()
self.assertEqual(after['journal_id'], first['journal_id'])
self.assertEqual(after['from'], first['to'])
self.assertEqual(after['data'], '')
self.assertFalse(any(self.journal.glob('stream.*')))
self.assertFalse((self.journal / 'lock').exists())
def test_missing_optional_cksum_does_not_create_empty_digest_or_drop_facts(self):
source = self.source.replace('case "${1:-}" in', 'cksum() { return 127; }\ncase "${1:-}" in', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.append('first event\nsecond event\n')
self.flush()
part = next(self.journal.glob('part.*'))
self.assertIn('.s256-', part.name)
self.assertIn(b'second event', base64.b64decode(self.export()['data']))
self.reboot()
self.assertIn(b'first event', base64.b64decode(self.export()['data']))
def test_digest_fingerprint_cache_stays_small_at_64_entries(self):
self.call('check')
# Seed 63 existing fingerprints; exercise eviction with two real events
# instead of starting thousands of Windows utility processes for a fill.
(self.run_dir / 'facts').write_text(''.join('1000\ts256-%035d\n' % i for i in range(63)))
self.append('distinct event 63\ndistinct event 64\n')
self.assertEqual(len((self.run_dir / 'facts').read_text().splitlines()), 64)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
def test_empty_digest_files_falsely_acked_by_old_hub_are_recovered(self):
self.append('retained restart event\n'); self.flush()
before = self.export()
self.acknowledge(before); self.flush()
part = next(self.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
# Upgrade during the same boot must override the former recovery marker.
(self.run_dir / 'recovered-v2').unlink()
(self.run_dir / 'recovered').touch()
repaired = self.export()
self.assertNotEqual(repaired['journal_id'], before['journal_id'])
self.assertTrue(repaired['journal_id'].startswith('recovered-'))
self.assertEqual(repaired['from'], 0)
self.assertIn(b'retained restart event', base64.b64decode(repaired['data']))
self.assertFalse(any(p.name.endswith('.') for p in self.journal.glob('part.*')))
self.acknowledge(repaired)
self.assertEqual(self.export()['data'], '')
self.flush()
# Recovery's warning was buffered after the replay chunk; deliver it too.
self.acknowledge(self.export())
self.flush()
self.reboot()
self.assertEqual(self.export()['data'], '')
def test_checksum_failure_keeps_pending_and_does_not_advance_total(self):
self.append('keep until checksum works\n')
before = self.status()['committed_bytes']
source = self.source.replace('log_checksum() {', 'log_checksum() { return 1;', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.call('flush', '', 'force', check=False)
self.assertFalse(list(self.journal.glob('part.*')))
self.assertEqual(self.status()['committed_bytes'], before)
self.assertIn('keep until checksum works', (self.run_dir / 'pending').read_text())
def test_md5_and_legacy_numeric_checksum_formats_remain_readable(self):
for digest in ('md5', 'crc'):
with self.subTest(digest=digest):
self.call('clear')
source = self.source.replace('if command -v sha256sum >/dev/null 2>&1; then method=s256',
'if false; then method=s256')
if digest == 'crc':
source = source.replace('elif command -v md5sum >/dev/null 2>&1; then method=md5',
'elif false; then method=md5')
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.append('digest compatibility ' + digest + '\n'); self.flush()
value = self.export()
self.reboot()
self.assertEqual(self.export(), value)
self.assertIn(digest.encode(), base64.b64decode(value['data']))
name = next(self.journal.glob('part.*')).name.rsplit('.', 1)[1]
self.assertEqual(name.startswith('md5-'), digest == 'md5')
def test_empty_export_never_claims_unreadable_chunks_as_delivered(self):
self.append('not yet delivered\n'); self.flush()
part = next(self.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.invalid'))
empty = self.export()
self.assertEqual((empty['from'], empty['to'], empty['data']), (0, 0, ''))
def test_failed_or_unmounted_overlay_does_not_receive_journal(self):
for kind, flags in [('overlay', 'rw'), ('tmpfs', 'rw'), ('ubifs', 'ro')]:
with self.subTest(kind=kind, flags=flags):
if self.run_dir.exists(): self.reboot()
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
f'fake {self.flash_mount} {kind} {flags} 0 0\n')
self.append('RAM fallback\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertIn('Flash недоступна', self.call('read').stdout)
self.assertFalse(self.journal.exists())
def test_nested_tmpfs_and_symlink_do_not_pass_flash_check(self):
nested = self.flash / 'ram'
nested.mkdir()
self.mounts.write_text(self.mounts.read_text() + f'tmpfs {self.flash_mount}/ram tmpfs rw 0 0\n')
self.env['OWRT_REMOTE_LOG_DIR'] = (nested / 'logs').as_posix()
self.append('nested tmpfs\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertFalse((nested / 'logs').exists())
def test_periodic_flush_and_idle_polling_do_not_rewrite_flash(self):
self.call('check')
state = self.journal / 'state'
before = state.stat().st_mtime_ns
self.append('buffered record\n')
self.assertEqual(list(self.journal.glob('part.*')), [])
self.assertEqual(self.export()['data'], '')
self.call('flush', TEST_UPTIME='1059')
self.assertEqual(list(self.journal.glob('part.*')), [])
self.call('flush', TEST_UPTIME='1060')
self.assertEqual(len(list(self.journal.glob('part.*'))), 1)
self.assertNotEqual(state.stat().st_mtime_ns, before)
stable = {p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}
for _ in range(3):
self.call('flush', TEST_UPTIME='1120')
self.export()
self.assertEqual({p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}, stable)
def test_buffer_and_number_of_archives_are_bounded_without_dropping_unacked(self):
self.helper.write_text(self.source.replace('LOG_LIMIT=2097152', 'LOG_LIMIT=32768')
.replace('LOG_PART_LIMIT=256', 'LOG_PART_LIMIT=2'),
encoding='utf-8', newline='\n')
for i in range(4):
self.append(f'event {i} ' + 'x' * 4000 + '\n')
self.flush()
parts = sorted(self.journal.glob('part.*'))
self.assertEqual(len(parts), 2)
original = {p.name: p.read_bytes() for p in parts}
for i in range(5):
self.append(f'overflow {i} ' + 'y' * 4000 + '\n')
self.flush()
self.assertEqual({p.name: p.read_bytes() for p in parts}, original)
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
first = self.export()
self.acknowledge(first)
self.flush()
self.assertFalse(parts[0].exists(), 'only acknowledged chunks make room')
self.assertIn('пропущено новых событий', self.call('read').stdout)
def test_ack_is_monotonic_and_does_not_delete_unconfirmed_after_one_day(self):
self.append('not delivered\n')
self.flush()
batch = self.export()
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
self.assertEqual(self.export(), batch)
invalid = self.call('ack', '', batch['journal_id'], str(batch['to'] + 1), check=False)
self.assertNotEqual(invalid.returncode, 0)
self.acknowledge(batch)
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
self.assertEqual(list(self.journal.glob('part.*')), [])
invalid = self.call('ack', '', batch['journal_id'], '0', check=False)
self.assertNotEqual(invalid.returncode, 0)
def test_crash_after_chunk_rename_recovers_original_id_and_offsets(self):
self.append('committed first\n')
self.flush()
first = self.export()
previous_state = (self.journal / 'state').read_bytes()
self.append('second chunk before state checkpoint\n')
self.flush()
(self.journal / 'state').write_bytes(previous_state)
(self.journal / 'chunk.new').write_text('interrupted write')
(self.journal / 'state.new').write_text('partial state')
self.reboot()
self.assertEqual(self.export()['journal_id'], first['journal_id'])
self.acknowledge(first)
second = self.export()
self.assertEqual(second['from'], first['to'])
self.assertIn(b'second chunk', base64.b64decode(second['data']))
self.assertFalse((self.journal / 'chunk.new').exists())
self.assertFalse((self.journal / 'state.new').exists())
def test_corrupt_chunk_is_quarantined_and_later_chunks_still_deliver(self):
self.append('first\n'); self.flush()
self.append('second\n'); self.flush()
first = sorted(self.journal.glob('part.*'))[0]
first.write_bytes(b'partial')
self.reboot()
batch = self.export()
self.assertIn(b'second', base64.b64decode(batch['data']))
self.assertTrue((self.journal / 'damaged.log').exists())
self.assertIn('Повреждённый фрагмент', self.call('read').stdout)
def test_disk_full_falls_back_without_touching_router_configuration(self):
self.append('retained on flash\n'); self.flush()
self.append('buffered before ENOSPC\n')
self.call('flush', '', 'force', TEST_FREE_KB='100', check=False)
self.append('after disk full\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertIn('after disk full', self.call('read').stdout)
self.assertTrue(list(self.journal.glob('part.*')))
def test_confirmed_ack_can_replay_after_crash_without_changing_cursor_identity(self):
self.append('duplicate safe\n'); self.flush()
batch = self.export()
self.acknowledge(batch)
self.assertEqual(self.export()['data'], '')
self.reboot() # ack not yet checkpointed: safe replay, Hub suppresses it.
self.assertEqual(self.export(), batch)
def test_killed_flusher_reconciles_inflight_without_duplicate_data(self):
self.append('already renamed\n')
self.flush()
batch = self.export()
part = next(self.journal.glob('part.*'))
fields = part.name.rsplit('.', 3)
(self.run_dir / 'pending').write_bytes(base64.b64decode(batch['data']))
(self.run_dir / 'inflight').write_text(f'{batch["journal_id"]} 0 {batch["to"]} {fields[-1]}\n')
self.append('after killed writer\n')
text = self.call('read').stdout
self.assertEqual(text.count('already renamed'), 1)
self.assertIn('after killed writer', text)
self.assertFalse((self.run_dir / 'inflight').exists())
def test_corrupt_state_recovers_committed_generation_and_stale_temporaries(self):
self.append('state recovery\n'); self.flush()
batch = self.export()
(self.journal / 'state').write_text('partial garbage')
self.reboot()
self.assertEqual(self.export(), batch)
(self.run_dir / 'export.12345').write_text('orphan')
self.export()
self.assertFalse(list(self.run_dir.glob('export.*')))
def test_parallel_writers_and_stale_lock_recovery(self):
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
list(pool.map(lambda i: self.append(f'concurrent {i}\n'), range(6)))
self.assertEqual(len(self.call('read').stdout.splitlines()), 6)
(self.run_dir / 'lock').mkdir()
(self.run_dir / 'lock/pid').write_text('99999999')
self.append('recovered lock\n')
self.assertIn('recovered lock', self.call('read').stdout)
self.assertFalse((self.run_dir / 'lock').exists())
def test_stale_lock_reaper_itself_recovers_after_process_crash(self):
self.call('check')
lock = self.run_dir / 'lock'; lock.mkdir()
(lock / 'pid').write_text('99999999')
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
self.append('after killed reaper\n')
self.assertIn('after killed reaper', self.call('read').stdout)
self.assertFalse(lock.exists())
def test_collector_failure_reaps_children_and_removes_fifo(self):
# Actual FIFO integration is tested on POSIX; Windows BusyBox has no
# FIFO support. Exercise its traps with mocked process/FIFO operations.
mocks = r'''
mkfifo() { : >"$1"; }
logread() { return 0; }
sleep() { return 0; }
kill() {
printf '%s\n' "$*" >>"$RUN_DIR/kills"
[ "$1" != -0 ]
}
wait() { return 0; }
'''
source = self.source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
result = self.call('collect', check=False)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(list(self.run_dir.glob('stream.*')))
self.assertFalse((self.run_dir / 'collector-lock').exists())
lines = (self.run_dir / 'kills').read_text().splitlines()
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 2)
def test_legacy_migration_preserves_identity_ack_and_only_undelivered_bytes(self):
legacy = self.root / 'legacy'
legacy.mkdir()
(legacy / 'current.log').write_bytes(b'legacy current\n')
(legacy / 'previous.log').write_bytes(b'legacy previous\n')
raw = b'legacy previous\nlegacy current\n'
(legacy / 'total').write_text(str(len(raw)))
(legacy / 'journal-id').write_text('legacy-journal')
(legacy / 'sync-ack').write_text('legacy-journal 16')
(legacy / 'stream.dead').write_text('must stay transient')
self.call('migrate')
self.assertEqual(self.call('read').stdout, raw.decode())
self.assertFalse((self.journal / 'stream.dead').exists())
self.assertFalse((legacy / 'current.log').exists())
self.assertFalse((legacy / 'total').exists())
self.assertTrue((legacy / 'stream.dead').exists())
batch = self.export()
self.assertEqual(batch['journal_id'], 'legacy-journal')
self.assertEqual(batch['from'], 16)
self.assertEqual(base64.b64decode(batch['data']), b'legacy current\n')
def test_unavailable_flash_legacy_migration_does_not_keep_large_ram_archive(self):
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n')
legacy = self.root / 'legacy'; legacy.mkdir()
(legacy / 'current.log').write_text('old data\n' * 10000)
self.call('migrate')
self.assertFalse((legacy / 'current.log').exists())
self.assertEqual(self.status()['persistent'], '0')
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
self.assertIn('ограниченный хвост', self.call('read').stdout)
def test_interrupted_legacy_migration_finishes_after_old_ram_is_gone(self):
self.append('staged persistent data\n'); self.flush()
batch = self.export()
staging = self.journal / 'migrating'; staging.mkdir()
for part in self.journal.glob('part.*'):
part.rename(staging / part.name)
shutil.copyfile(self.journal / 'state', staging / 'state')
(self.journal / 'state').write_text('interrupted state replacement')
self.reboot()
self.assertEqual(self.export(), batch)
self.assertFalse(staging.exists())
if __name__ == '__main__':
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
"""Install/rollback journal files against a sandbox, never a real router."""
import os
import re
from pathlib import Path
import shlex
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
@unittest.skipUnless(SH, 'POSIX shell required')
class PersistentLogsDeploymentTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='persistent-log-deploy-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.targets = {
'cgi': '/www/cgi-bin/owrt-remote', 'agent': '/usr/sbin/owrt-remote',
'init': '/etc/init.d/owrt-remote', 'logs.sh': '/usr/lib/owrt-remote/logs.sh',
'logs-init': '/etc/init.d/owrt-remote-logs',
'luci': '/www/luci-static/resources/view/owrt_remote.js',
}
self.initial = {}
for item in ('cgi', 'agent', 'logs.sh', 'logs-init', 'luci'):
path = self.root / self.targets[item].lstrip('/')
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text('#!/bin/sh\necho previous-' + item + '\n', encoding='utf-8', newline='\n')
self.initial[item] = path.read_bytes()
self.backup = self.root / 'backup'
source = (ROOT / 'vps/deploy-persistent-logs.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
# Absolute paths are replaced only in this disposable test copy.
mappings = {target: (self.root / target.lstrip('/')).as_posix() for target in self.targets.values()}
mappings['/usr/lib/owrt-remote'] = (self.root / 'usr/lib/owrt-remote').as_posix()
mappings['/www/luci-static/resources/view'] = (self.root / 'www/luci-static/resources/view').as_posix()
mappings['/tmp/luci-indexcache'] = (self.root / 'luci-indexcache').as_posix()
mappings['/tmp/luci-modulecache'] = (self.root / 'luci-modulecache').as_posix()
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
mappings['/root/owrt-remote-backup-before-persistent-logs-v117'] = self.backup.as_posix()
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
# Existing sandbox service scripts only print markers. Mock readiness.
mocks = '''
logread() { return 0; }
ubus() { printf '{"running": true}\\n'; }
mkfifo() { return 0; }
jsonfilter() { return 0; }
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
'''
self.script = self.script.replace('for item in cgi agent logs.sh logs-init;', mocks + '\nfor item in cgi agent logs.sh logs-init;', 1)
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def deploy(self, action, bad=False, fail_verify=False):
stage = self.root / 'stage'
stage.mkdir()
candidates = {
'cgi': 'files/www/cgi-bin/owrt-remote', 'agent': 'files/usr/sbin/owrt-remote',
'init': 'files/etc/init.d/owrt-remote', 'logs.sh': 'files/usr/lib/owrt-remote/logs.sh',
'logs-init': 'files/etc/init.d/owrt-remote-logs',
'luci': 'files/www/luci-static/resources/view/owrt_remote.js',
}
if action == 'Install':
for item, source in candidates.items():
text = (ROOT / source).read_text(encoding='utf-8')
if item == 'logs.sh':
text = text.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + shlex.quote((self.root / 'legacy').as_posix()))
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
if item == 'logs-init':
text = '#!/bin/sh\necho logs-service-$1\n'
if bad and item == 'cgi':
text = '#!/bin/sh\nif true; then\n'
stage.joinpath(item).write_text(text, encoding='utf-8', newline='\n')
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(stage.as_posix()))
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
if fail_verify:
script = script.replace('cmp "$stage/$item" "$path"', 'false')
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env=self.env, timeout=30)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.rglob('*.new-sandbox')))
return result
def assert_initial(self):
for item, data in self.initial.items():
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
def test_install_then_rollback_retains_first_backup(self):
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_ROUTER_Install_OK', result.stdout)
for item in ('cgi', 'luci'):
installed = self.root / self.targets[item].lstrip('/')
self.assertIn('v117', installed.read_text(encoding='utf-8'))
for item, data in self.initial.items():
self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_ROUTER_Rollback_OK', result.stdout)
self.assert_initial()
def test_missing_backup_changes_nothing(self):
result = self.deploy('Rollback')
self.assertNotEqual(result.returncode, 0)
self.assertIn('No complete backup:', result.stderr)
self.assert_initial()
def test_invalid_candidate_rejected_before_backup(self):
result = self.deploy('Install', bad=True)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(self.backup.exists())
self.assert_initial()
def test_failed_verification_restores_all_previous_files(self):
result = self.deploy('Install', fail_verify=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn('Restoring files from this attempt.', result.stderr)
self.assert_initial()
def test_rollback_restores_existing_log_service_and_helper(self):
# This fix is normally applied on top of v117's already installed journal.
for item in ('logs.sh', 'logs-init'):
target = self.root / self.targets[item].lstrip('/')
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text('#!/bin/sh\necho previous-' + item + '-$1\n', encoding='utf-8', newline='\n')
self.initial[item] = target.read_bytes()
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
for item, data in self.initial.items():
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
self.assertIn('previous-logs-init-start', result.stdout)
if __name__ == '__main__':
unittest.main()
+106
View File
@@ -0,0 +1,106 @@
"""Validate VPS install, rollback and automatic recovery in a disposable tree."""
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
@unittest.skipUnless(SH, 'POSIX shell required')
class VpsLogDeployTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='vps-log-deploy-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.target = self.root / 'hub.py'
self.previous = b'print("previous Hub")\n'
self.target.write_bytes(self.previous)
self.backup = self.root / 'hub.py.bak-before-persistent-logs-sync-v117'
source = (ROOT / 'vps/deploy-persistent-logs-vps.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
self.script = self.script.replace('python_bin=/opt/owrt-remote/venv/bin/python', 'python_bin=python')
self.script = self.script.replace('[ -x "$python_bin" ] || python_bin=python3', '')
mocks = '''
systemctl() { [ "${TEST_RESTART_FAIL:-0}" != 1 ] || [ "$1" != restart ]; }
journalctl() { echo sandbox-journal; }
curl() { [ "${TEST_HEALTH_FAIL:-0}" != 1 ] && printf '{"ok":true}'; }
sleep() { return 0; }
'''
self.script = self.script.replace('test -s "$target"', mocks + '\ntest -s "$target"', 1)
# Exercise the actual mount/path validation with an isolated process proc
# fixture. Storage classification is covered independently on real paths.
original = self.root / 'running.py'
original.write_text('print("running")')
proc = self.root / 'proc' / '123'
proc.mkdir(parents=True)
(proc / 'environ').write_bytes(('OWRT_REMOTE_STATE_DIR=' + self.root.as_posix()).encode() + b'\0')
(proc / 'cmdline').write_bytes(b'python\0' + original.as_posix().encode() + b'\0')
database = self.root / 'hub.db'
database.write_bytes(b'existing archive must not change')
self.database = database
self.script = self.script.replace("Path('/proc/' + pid", "Path('" + (self.root / 'proc').as_posix() + "/' + pid")
self.script = self.script.replace("storage = hub.router_log_storage(db_path)",
"storage = hub.router_log_storage(db_path)\n" +
"storage.update(persistent=os.environ.get('TEST_RAM_DISK') != '1', writable=True)")
self.script = self.script.replace('systemctl() { ',
'systemctl() { if [ "$1" = show ]; then echo 123; return 0; fi; ')
self.env = dict(os.environ)
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def deploy(self, action, invalid=False, **flags):
stage = self.root / 'stage'
stage.mkdir()
if action == 'Install':
data = 'if invalid:\n' if invalid else (ROOT / 'vps/owrt-remote-hub.py').read_text(encoding='utf-8')
stage.joinpath('hub.py').write_text(data, encoding='utf-8', newline='\n')
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage="' + stage.as_posix() + '"')
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.glob('*.new-sandbox')))
self.assertEqual(self.database.read_bytes(), b'existing archive must not change')
return result
def test_install_and_rollback_preserve_original_backup(self):
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_VPS_V117_Install_OK', result.stdout)
self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
self.assertEqual(self.backup.read_bytes(), self.previous)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(self.target.read_bytes(), self.previous)
def test_failed_health_restores_file_from_this_attempt(self):
self.assertEqual(self.deploy('Install').returncode, 0)
installed = self.target.read_bytes()
result = self.deploy('Rollback', TEST_HEALTH_FAIL='1')
self.assertNotEqual(result.returncode, 0)
self.assertIn('Restoring the previous Hub file', result.stderr)
self.assertEqual(self.target.read_bytes(), installed)
def test_ram_database_preflight_does_not_install_or_modify_archive(self):
result = self.deploy('Install', TEST_RAM_DISK='1')
self.assertNotEqual(result.returncode, 0)
self.assertIn('not confirmed on writable persistent disk', result.stderr)
self.assertEqual(self.target.read_bytes(), self.previous)
self.assertFalse(self.backup.exists())
def test_invalid_candidate_and_missing_backup_change_nothing(self):
self.assertNotEqual(self.deploy('Install', invalid=True).returncode, 0)
self.assertFalse(self.backup.exists())
self.assertNotEqual(self.deploy('Rollback').returncode, 0)
self.assertEqual(self.target.read_bytes(), self.previous)
if __name__ == '__main__':
unittest.main()
+1 -1
View File
@@ -131,7 +131,7 @@ class HubFixture:
assert "groups" in json.loads(self.request("/api/router-groups"))
assert "notifications" in json.loads(self.request("/api/notifications"))
assert self.request("/access/test-router/") == b"router proxy fixture"
assert b"v116" in self.request("/")
assert b"v117" in self.request("/")
self.app.snapshot_router_states()
def __exit__(self, *args):
+75
View File
@@ -113,6 +113,81 @@ class ArchiveTests(unittest.TestCase):
self.assertIn('новый журнал', self.snapshot()['text'])
self.assertIn('first', self.snapshot()['text'])
def test_empty_status_cannot_confirm_missing_records_or_invent_rotation_gap(self):
reply = self.ingest(batch(b'', start=298))
self.assertEqual(reply['offset'], 0)
self.assertEqual(self.snapshot()['text'], '')
raw = b'[OK] retained records arrive after status\n'
self.ingest(batch(raw))
reply = self.ingest(batch(b'', start=1000))
self.assertEqual(reply['offset'], len(raw))
self.assertEqual(self.snapshot()['text'].encode(), raw)
def test_actual_gap_with_data_is_reported_once_but_retries_keep_data(self):
value = batch(b'[OK] surviving records\n', start=50)
self.ingest(value)
self.ingest(value)
text = self.snapshot()['text']
self.assertEqual(text.count('не доставлена'), 1)
self.assertEqual(text.count('surviving records'), 1)
def test_empty_new_generation_cannot_erase_current_journal_cursor(self):
raw = b'[OK] previous generation\n'
self.ingest(batch(raw))
self.ingest(batch(b'', start=2000, journal='new-journal'))
self.ingest(batch(raw))
self.assertEqual(self.snapshot()['text'].encode(), raw)
def test_recovery_generation_replays_previously_falsely_confirmed_chunks(self):
# Reproduce the cursor left by the former empty-batch ingestion bug.
with self.app.conn() as conn:
conn.execute("insert into router_logs(router_id,journal_id,cursor) values ('main','broken',298)")
self.ingest(batch(b'[OK] retained restart event\n', journal='recovered-old'))
self.assertIn('retained restart event', self.snapshot()['text'])
def test_real_router_spool_recovers_bad_digest_and_delivers_after_restart(self):
from test_persistent_logs import PersistentLogsTests
client = PersistentLogsTests()
client.setUp()
self.addCleanup(client.doCleanups)
client.append('Панель: действие=stop; начало\nСервис: остановка туннеля\n')
client.flush()
original = client.export()
client.acknowledge(original); client.flush()
with self.app.conn() as conn:
conn.execute('insert into router_logs(router_id,journal_id,cursor) values (?,?,?)',
('main', original['journal_id'], original['to']))
part = next(client.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
(client.run_dir / 'recovered-v2').unlink()
repaired = client.export()
reply = self.ingest(repaired, {'enabled': False, 'xray': 'stopped'})
client.call('ack', '', reply['journal_id'], str(reply['offset']))
# A lost acknowledgment may replay, but the archive must stay unchanged.
self.ingest(repaired)
self.assertEqual(self.snapshot()['text'].count('Сервис: остановка туннеля'), 1)
client.append('Сервис: запуск\n'); client.flush()
second = client.export()
reply = self.ingest(second, {'enabled': True, 'xray': 'running'})
client.call('ack', '', reply['journal_id'], str(reply['offset']))
text = self.snapshot()['text']
self.assertIn('Сервис: запуск', text)
self.assertNotIn('не доставлена до ротации', text)
self.assertEqual(client.export()['data'], '')
def test_disk_mount_diagnostic_resolves_deepest_mount_and_escaped_paths(self):
path = self.app.db_path.resolve()
mount = str(path.parent).replace(' ', r'\040')
root = str(Path(path.anchor)).replace(' ', r'\040')
mountinfo = f'1 0 8:1 / {root} rw - ext4 /dev/vda1 rw\n'
mountinfo += f'2 1 0:1 / {mount} rw - tmpfs tmpfs rw\n'
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo)):
result = hub.router_log_storage(path)
self.assertIs(result['persistent'], False)
self.assertEqual(result['filesystem'], 'tmpfs')
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo.splitlines()[0])):
self.assertIs(hub.router_log_storage(path)['persistent'], True)
def test_offline_clear_does_not_restore_old_logs_on_reconnection(self):
old = b'[ERROR] old\n'
self.ingest(batch(old))
+3
View File
@@ -35,6 +35,7 @@ const server = http.createServer((req, res) => {
reads++;
res.setHeader('Content-Type', 'application/json');
return res.end(JSON.stringify({ok:true, online:false, text, bytes:Buffer.byteLength(text), last_sync:1791954000,
storage:{path:'/var/lib/owrt-remote/hub.db',filesystem:'ext4',persistent:true},
diagnosis:{level:'warn',text:'Remote выключен на роутере. Туннель и Heartbeat остановлены.',at:1791954000}}));
}
if (url.pathname === '/api/router/main/logs/clear') {
@@ -95,6 +96,8 @@ const server = http.createServer((req, res) => {
text=initialText();const before=reads;
await page.goto(`http://127.0.0.1:${server.address().port}/router/main/logs`);
await page.waitForFunction(()=>document.querySelector('#logsText').textContent.includes('Event 119'));
assert((await page.locator('#logsStorage').textContent()).includes('Архив VPS: постоянный диск.'));
assert((await page.locator('#logsStorage').textContent()).includes('/var/lib/owrt-remote/hub.db'));
assert.deepEqual(await page.locator('body').evaluate(el=>{
const s=getComputedStyle(el);return {image:s.backgroundImage,color:s.backgroundColor,attachment:s.backgroundAttachment};
}),terminalBackground,'log page uses the VPS terminal background');
+41 -8
View File
@@ -23,6 +23,8 @@ class RouterLogTransportTests(unittest.TestCase):
self.env = {**self.journal.env, 'OWRT_REMOTE_WAN_HELPER': (ROOT / 'files/usr/lib/owrt-remote/wan.sh').as_posix(),
'TEST_ROOT': self.journal.root.as_posix(), 'TEST_PYTHON': Path(sys.executable).as_posix()}
self.mocks = r'''
# Windows has no RLIMIT_FSIZE; production ash enforces the actual file limit.
ulimit() { return 0; }
uci_get() {
case "$1" in
enabled) printf '%s' "${TEST_ENABLED:-1}" ;; hub_token) printf test-token ;; router_id) printf main ;;
@@ -60,6 +62,7 @@ jsonfilter() {
'''
def sync(self, **flags):
self.journal.helper_call('flush', '', 'force')
script = self.source + '\n' + self.mocks + '\nsync_global_logs\n'
return subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25)
@@ -80,14 +83,34 @@ jsonfilter() {
self.assertEqual('direct' in requests, direct == '1')
self.assertEqual('prepare' in requests, direct == '1')
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
def test_failed_delivery_preserves_unsent_records_and_removes_payload_files(self):
before = json.loads(self.journal.helper_call('export'))
result = self.sync(TEST_HTTP_FAIL='1')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertEqual(json.loads(self.journal.helper_call('export')), before)
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
def test_agent_update_delivers_with_older_helper_without_its_own_runtime_directory(self):
# The pre-persistent helper creates only /tmp/owrt-remote-logs. A WAN
# agent update must not depend on that helper creating the new runtime.
helper = self.journal.root / 'old-helper'
record = base64.b64encode(b'[OK] older helper record\n').decode()
batch = json.dumps({'journal_id': 'older', 'from': 0, 'to': 25, 'data': record})
helper.write_text('#!/bin/sh\ncase "$1" in\ncheck) exit 0 ;;\nexport) printf \'%s\\n\' \'' + batch +
'\' ;;\nack) printf \'%s\\n\' "$2 $3" >"$TEST_ROOT/older-ack" ;;\nesac\n',
encoding='utf-8', newline='\n')
runtime = self.journal.root / 'new-runtime'
result = subprocess.run([SH], input=self.source + '\n' + self.mocks + '\nsync_global_logs\n',
encoding='utf-8', capture_output=True,
env={**self.env, 'OWRT_REMOTE_LOG_HELPER': helper.as_posix(),
'OWRT_REMOTE_LOG_RUNTIME': runtime.as_posix()}, timeout=25)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertTrue(runtime.is_dir())
self.assertTrue((self.journal.root / 'older-ack').is_file())
self.assertFalse((runtime / 'sync-lock').exists())
self.assertFalse(list(runtime.glob('sync.*')))
def test_disabled_remote_still_delivers_stop_reason_and_state_without_restart(self):
self.journal.helper_call('append', 'Сервис: остановка туннеля\n')
@@ -97,14 +120,14 @@ jsonfilter() {
self.assertEqual(result.returncode, 0, result.stderr)
payload = json.loads((self.journal.root / 'delivered.json').read_text())
self.assertEqual(payload['state'], {'enabled': False, 'xray': 'stopped'})
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
def test_overlapping_delivery_does_not_start_second_sender(self):
lock = self.journal.log_dir / 'sync-lock'
lock = self.journal.run_dir / 'sync-lock'
lock.mkdir()
# The shell's own PID is alive for the duration of sync_global_logs.
script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_DIR/sync-lock/pid"\nsync_global_logs\n'
script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_RUNTIME/sync-lock/pid"\nsync_global_logs\n'
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True, env=self.env, timeout=20)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((self.journal.root / 'requests').exists())
@@ -113,15 +136,25 @@ jsonfilter() {
def test_wan_failure_releases_delivery_lock_for_next_attempt(self):
result = self.sync(TEST_DIRECT='1', TEST_WAN_FAIL='1', TEST_ENABLED='0')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertFalse((self.journal.log_dir / 'sync-lock').exists())
self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
self.assertNotEqual(json.loads(self.journal.helper_call('export'))['data'], '')
self.assertEqual(self.sync(TEST_DIRECT='1', TEST_ENABLED='0').returncode, 0)
def test_abandoned_empty_delivery_lock_is_recovered(self):
(self.journal.log_dir / 'sync-lock').mkdir()
(self.journal.run_dir / 'sync-lock').mkdir()
result = self.sync(TEST_ENABLED='0')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((self.journal.log_dir / 'sync-lock').exists())
self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
self.assertTrue((self.journal.root / 'delivered.json').exists())
self.assertFalse(list(self.journal.run_dir.glob('http/owrt-remote-http.*')))
def test_abandoned_delivery_lock_reaper_is_cleaned_for_next_attempt(self):
lock = self.journal.run_dir / 'sync-lock'
lock.mkdir(); (lock / 'pid').write_text('99999999')
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
self.assertEqual(self.sync().returncode, 0)
self.assertEqual(self.sync().returncode, 0)
self.assertFalse(lock.exists())
self.assertTrue((self.journal.root / 'delivered.json').exists())
def test_panel_stop_sends_completed_result_without_waiting_for_restart(self):
+1 -1
View File
@@ -35,7 +35,7 @@ class WanDnsDeploymentTests(unittest.TestCase):
source = (ROOT / "vps/deploy-wan-dns-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("target=/usr/lib/owrt-remote/wan.sh", "target=" + shlex.quote(self.target.as_posix()))
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v116",
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix()))
def deploy(self, action="Install", bad=False, fail_verify=False, fail_signal=False):
+1 -1
View File
@@ -129,7 +129,7 @@ class WanFailoverTests(unittest.TestCase):
self.prepare()
config = self.directory / "xray.json"
config.write_text(json.dumps({"outbounds": [{"tag": "vps-interconn", "streamSettings": {"sockopt": {"interface": "eth0"}}}]}))
self.config(wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix())
self.config(enabled="1", wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix())
result = self.shell('''
xray_running() { return 0; }
wan_reconnect_xray() { printf '%s -> %s\n' "$1" "$2" >>"$TEST_ROOT/reconnect"; }
+208
View File
@@ -0,0 +1,208 @@
"""WAN outage/recovery using the real agent shell and a simulated procd."""
import json
import os
import subprocess
import unittest
import test_host_policy as base
from test_wan_failover import UPLINK_MOCKS, JSONFILTER
@unittest.skipUnless(base.SH, "POSIX shell required")
class WanRecoveryTests(unittest.TestCase):
setUp = base.RouterShellTests.setUp
config = base.RouterShellTests.config
def shell(self, commands):
shell = os.environ.get("OWRT_RECOVERY_TEST_SHELL", base.SH)
args = [shell, "ash"] if "busybox" in shell.lower() else [shell]
return subprocess.run(args, input=self.agent + "\n" + base.MOCKS + "\n" + UPLINK_MOCKS + "\n" + commands,
encoding="utf-8", capture_output=True, cwd=base.ROOT, env=self.env, timeout=40)
def prepare(self, enabled="1"):
self.config(enabled=enabled, wan_direct="1", wan_interface="wan", vps_host="192.0.2.10",
hub_url="https://hub.example", hub_token="secret-test-token")
self.directory.joinpath("jsonfilter.py").write_text(JSONFILTER, encoding="utf-8")
self.directory.joinpath("service.json").write_text(json.dumps({"owrt-remote": {"instances": {
"heartbeat": {"running": True, "pid": 123}, "xray": {"running": False}
}}}), encoding="utf-8")
PROCD = r'''
xray_running() { return 1; }
ubus() {
case "$1 $2 $3" in
'call service list') cat "$TEST_ROOT/service.json" ;;
'call service add')
printf '%s\n' "$4" >>"$TEST_ROOT/add-calls"
[ ! -e "$TEST_ROOT/add-fail" ] ;;
*) return 1 ;;
esac
}
jsonfilter() {
"$TEST_PYTHON" -c 'import json,sys
state=json.load(sys.stdin)
instance="heartbeat" if ".heartbeat." in sys.argv[1] else "xray"
value=state.get("owrt-remote",{}).get("instances",{}).get(instance,{}).get("running","")
print(str(value).lower())' "$2"
}
wan_select_interface() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf wan; }
wan_device() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf eth0; }
'''
def test_online_heartbeat_after_outage_recovers_dead_tunnel_on_same_uplink_once(self):
self.prepare()
result = self.shell(self.PROCD + r'''
touch "$TEST_ROOT/wan-down"
wan_reconcile_xray || true
test ! -e "$TEST_ROOT/add-calls" || exit 1
rm "$TEST_ROOT/wan-down"
wan_reconcile_xray || exit
wan_reconcile_xray
''')
self.assertEqual(result.returncode, 0, result.stderr)
requests = self.directory.joinpath("add-calls").read_text().splitlines()
self.assertEqual(len(requests), 1)
request = json.loads(requests[0])
self.assertEqual(set(request["instances"]), {"xray"})
self.assertEqual(request["instances"]["xray"]["command"], ["/usr/sbin/owrt-remote", "run"])
self.assertEqual(request["instances"]["xray"]["respawn"], ["3600", "5", "5"])
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
self.assertNotIn("secret-test-token", result.stderr)
def test_initial_start_waiting_for_wan_is_left_to_procd(self):
self.prepare()
state = self.directory / "service.json"
data = json.loads(state.read_text())
data["owrt-remote"]["instances"]["xray"]["running"] = True
state.write_text(json.dumps(data))
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_successful_heartbeat_recovers_dead_xray_and_still_returns_hub_response(self):
self.prepare()
result = self.shell(self.PROCD + r'''
wan_prepare() { return 0; }
heartbeat_json() { printf '{}'; }
heartbeat_candidate_endpoints() { printf 'https://hub.example/api/heartbeat'; }
post_json() { printf '{"ok":true}'; }
heartbeat_apply_hub_update() { return 0; }
send_heartbeat
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(json.loads(result.stdout), {"ok": True})
request = json.loads(self.directory.joinpath("add-calls").read_text())
self.assertEqual(set(request["instances"]), {"xray"})
self.assertIn("Heartbeat: OK", result.stderr)
def test_healthy_xray_is_not_restarted_when_wan_returns(self):
self.prepare()
result = self.shell(self.PROCD + "xray_running() { return 0; }; wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_panel_stop_and_external_service_stop_are_respected(self):
self.prepare(enabled="0")
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
self.prepare()
self.directory.joinpath("service.json").write_text('{}')
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_parallel_requests_and_stale_lock_produce_one_recovery(self):
self.prepare()
lock = self.directory / "owrt-remote-xray-recovery"
lock.mkdir()
lock.joinpath("pid").write_text("2147483000")
result = self.shell(self.PROCD + "wan_reconcile_xray & wan_reconcile_xray & wait")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(lock.exists())
def test_ubus_failure_reports_error_cleans_lock_and_honors_backoff(self):
self.prepare()
self.directory.joinpath("add-fail").touch()
result = self.shell(self.PROCD + "wan_reconcile_xray; rc=$?; wan_reconcile_xray; exit $rc")
self.assertEqual(result.returncode, 1, result.stderr)
self.assertIn("procd", result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
def test_empty_lock_and_dead_reaper_do_not_block_later_recovery(self):
self.prepare()
lock = self.directory / "owrt-remote-xray-recovery"
lock.mkdir()
(lock / "reap").mkdir()
(lock / "reap/pid").write_text("2147483000")
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(lock.exists())
def test_missing_wan_does_not_cause_repeated_service_restarts(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
result = self.shell(self.PROCD + r'''
active_ssh_session() { return 1; }
heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("owrt-remote-heartbeat.restart").exists())
self.assertIn("IPv4 default route", result.stderr)
def test_supervised_start_waits_beyond_six_retries_then_launches_with_valid_config(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
executable = self.directory / "fake-xray"
executable.write_text('#!/bin/sh\n[ "$1" != help ] || { echo run; exit; }; printf "%s\\n" "$*" >"$TEST_ROOT/xray-called"\n', newline="\n")
executable.chmod(0o755)
config = self.directory / "valid-config.json"
config.write_text('{"outbounds":[]}')
result = self.shell(r'''
sleep() {
local n
n=$(cat "$TEST_ROOT/waits" 2>/dev/null || echo 0)
n=$((n+1)); printf '%s' "$n" >"$TEST_ROOT/waits"
[ "$n" -lt 8 ] || rm -f "$TEST_ROOT/wan-down"
}
ensure_xray_available_with_retry() { return 0; }
xray_bin_path() { printf '%s' "$TEST_ROOT/fake-xray"; }
render_client_config() { printf '%s' "$TEST_ROOT/valid-config.json"; }
run_xray
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(self.directory.joinpath("waits").read_text(), "8")
self.assertEqual(self.directory.joinpath("xray-called").read_text().strip(), 'run -config ' + config.as_posix())
self.assertEqual(result.stderr.count("конфигурация сохранена"), 1)
self.assertIn("продолжается запуск", result.stderr)
def test_disabling_service_cancels_wait_without_rewriting_configuration(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
result = self.shell(r'''
sleep() { uci set owrtremote.main.enabled=0; }
render_client_config() { touch "$TEST_ROOT/rendered"; }
run_xray
''')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertFalse(self.directory.joinpath("rendered").exists())
def test_configuration_error_with_available_wan_is_not_retried_as_wan_outage(self):
self.prepare()
result = self.shell(r'''
sleep() { touch "$TEST_ROOT/slept"; }
ensure_xray_available_with_retry() { return 0; }
xray_bin_path() { printf /usr/bin/xray; }
render_client_config() { printf 'invalid configuration\n' >&2; return 7; }
run_xray
''')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertIn("invalid configuration", result.stderr)
self.assertFalse(self.directory.joinpath("slept").exists())
if __name__ == "__main__":
unittest.main()
+122
View File
@@ -0,0 +1,122 @@
"""Run installation and rollback in a sandbox, without SSH or services."""
import os
from pathlib import Path
import shlex
import subprocess
import tempfile
import unittest
from test_xray_mobile_deploy import SH, GIT_BIN, ROOT
@unittest.skipUnless(SH, "POSIX shell required")
class WanRecoveryDeploymentTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix="wan-recovery-deploy-")
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.targets = {"wan.sh": self.root / "wan.sh", "agent": self.root / "agent"}
for name, path in self.targets.items():
path.write_text("#!/bin/sh\necho previous-" + name + "\n", encoding="utf-8", newline="\n")
self.original = {name: path.read_bytes() for name, path in self.targets.items()}
self.backup = self.root / "backup"
self.env = dict(os.environ)
if os.name == "nt":
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-wan-recovery-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-recovery-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
def deploy(self, action, bad_item=None, fail_verify=False, fail_signal=False, stopped=False):
stage = self.root / "stage"
stage.mkdir()
for item, source in {"wan.sh": "files/usr/lib/owrt-remote/wan.sh", "agent": "files/usr/sbin/owrt-remote"}.items():
text = (ROOT / source).read_text(encoding="utf-8")
if bad_item == item:
text = "#!/bin/sh\nif true; then\n"
stage.joinpath(item).write_text(text, encoding="utf-8", newline="\n")
script = self.script.replace("stage=/tmp/__UPLOAD_NAME__", "stage=" + shlex.quote(stage.as_posix()))
script = script.replace("__ACTION__", action).replace("__UPLOAD_NAME__", "sandbox")
if fail_verify:
script = script.replace('cmp "$source_dir/$item" "$path"', "false")
mocks = r'''
ubus() {
printf '%s\n' "$*" >>"$TEST_ROOT/ubus-calls"
if [ "$3" = signal ]; then
[ "${TEST_SIGNAL_FAIL:-0}" = 0 ] || return 1
printf 202 >"$TEST_ROOT/pid"
fi
printf '{}'
}
jsonfilter() { cat >/dev/null; cat "$TEST_ROOT/pid"; }
sleep() { :; }
'''
self.root.joinpath("pid").write_text("" if stopped else "101")
result = subprocess.run([SH], input=mocks + script, encoding="utf-8", capture_output=True,
env={**self.env, "TEST_ROOT": self.root.as_posix(), "TEST_SIGNAL_FAIL": "1" if fail_signal else "0"}, timeout=30)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.glob("*.new-sandbox")))
return result
def test_install_twice_then_rollback_preserves_first_backup(self):
for attempt in range(2):
result = self.deploy("Install")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("WAN_RECOVERY_ROUTER_Install_OK", result.stdout)
for item, path in self.targets.items():
self.assertNotEqual(path.read_bytes(), self.original[item])
self.assertEqual(self.backup.joinpath(item).read_bytes(), self.original[item])
result = self.deploy("Rollback")
self.assertEqual(result.returncode, 0, result.stderr)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
def test_bad_candidate_and_missing_backup_leave_paired_files_unchanged(self):
for item in ("agent", "wan.sh"):
result = self.deploy("Install", bad_item=item)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(self.backup.exists())
for name, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[name])
result = self.deploy("Rollback")
self.assertNotEqual(result.returncode, 0)
self.assertIn("No complete backup", result.stderr)
for name, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[name])
def test_reloads_only_heartbeat_without_touching_xray(self):
result = self.deploy("Install")
self.assertEqual(result.returncode, 0, result.stderr)
calls = self.root.joinpath("ubus-calls").read_text()
self.assertIn('service signal {"name":"owrt-remote","instance":"heartbeat","signal":9}', calls)
self.assertNotIn('"instance":"xray"', calls)
self.assertIn('Heartbeat loop reloaded', result.stdout)
def test_stopped_loop_stays_stopped(self):
result = self.deploy("Install", stopped=True)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertNotIn('service signal', self.root.joinpath("ubus-calls").read_text())
self.assertIn('Heartbeat loop is stopped', result.stdout)
def test_failed_loop_reload_restores_paired_files_and_reports_failure(self):
result = self.deploy("Install", fail_signal=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn('Failed during heartbeat-reload', result.stderr)
self.assertNotIn('WAN_RECOVERY_ROUTER_Install_OK', result.stdout)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
def test_verify_failure_restores_paired_files(self):
result = self.deploy("Install", fail_verify=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn("Restoring the agent and WAN helper", result.stderr)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -31,7 +31,7 @@ class WanStabilityRollbackTests(unittest.TestCase):
source = (ROOT / "vps/deploy-wan-stability-fix.ps1").read_text(encoding="utf-8")
script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
script = script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(self.stage.as_posix()))
script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v116', 'backup=' + shlex.quote(self.backup.as_posix()))
script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v117', 'backup=' + shlex.quote(self.backup.as_posix()))
script = script.replace('config=/etc/config/owrtremote', 'config=' + shlex.quote(self.files["config"].as_posix()))
for name, original in (("agent", "/usr/sbin/owrt-remote"), ("wan.sh", "/usr/lib/owrt-remote/wan.sh")):
script = script.replace('printf ' + original + ' ;;', "printf '%s' " + shlex.quote(self.files[name].as_posix()) + ' ;;')
+1 -1
View File
@@ -29,7 +29,7 @@ class XrayMobileDeploymentTests(unittest.TestCase):
source = (ROOT / "vps/deploy-xray-mobile-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("target=/www/cgi-bin/owrt-remote", "target=" + shlex.quote(self.target.as_posix()))
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v116",
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v117",
"backup_dir=" + shlex.quote(self.backup_dir.as_posix()))
def deploy(self, action, content=None, fail_verify=False):
+1 -1
View File
@@ -25,7 +25,7 @@ class XrayValidationDeploymentTests(unittest.TestCase):
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-xray-validation-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v116",
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /www/cgi-bin/owrt-remote", "printf %s " + shlex.quote(self.targets["cgi"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))