Release v117 with persistent logs and WAN tunnel recovery
This commit is contained in:
1 parent
5f65a3d811
commit
b62db40daf
61 files changed
+3080
-438
No files matched your search
@@ -1,4 +1,4 @@
|
||||
# Heartbeat: HTTPS, авторизация и диагностика (v116)
|
||||
# Heartbeat: HTTPS, авторизация и диагностика (v117)
|
||||
|
||||
## Разбор сбоя 15.10.2026
|
||||
|
||||
@@ -82,7 +82,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
|
||||
Обновляются агент и WAN helper. Перезапускается только экземпляр Heartbeat
|
||||
под procd; Xray и удалённый туннель скрипт не перезапускает. Остановленный
|
||||
экземпляр остаётся остановленным. Резервная копия первых файлов сохраняется
|
||||
в `/root/owrt-remote-backup-before-heartbeat-transport-v116`.
|
||||
в `/root/owrt-remote-backup-before-heartbeat-transport-v117`.
|
||||
Маркер `HEARTBEAT_TRANSPORT_ROUTER_Install_OK` подтверждает установку кода,
|
||||
а доступность Hub проверяется отдельно:
|
||||
|
||||
|
||||
+12
-12
@@ -1,4 +1,4 @@
|
||||
# Фиксация VPS host и независимое соединение Remote через WAN (v116)
|
||||
# Фиксация VPS host и независимое соединение Remote через WAN (v117)
|
||||
|
||||
`vps_host_mode=manual` сохраняет заданный на роутере адрес. Это значение
|
||||
используется и когда опция отсутствует у старой установки. Только явно
|
||||
@@ -42,8 +42,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-hub-url-fix.
|
||||
```
|
||||
|
||||
Откат: те же команды с `-Action Rollback`, без `-HubUrl`. Резервные копии:
|
||||
`/root/owrt-remote-backup-before-hub-url-v116` на роутере и
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v116` на VPS.
|
||||
`/root/owrt-remote-backup-before-hub-url-v117` на роутере и
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v117` на VPS.
|
||||
`HEARTBEAT_OK` подтверждает ответ Hub. Если файлы установлены, но Hub недоступен,
|
||||
скрипт отдельно сообщает `HEARTBEAT_FAILED`; это не подтверждение связи.
|
||||
|
||||
@@ -125,21 +125,21 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.
|
||||
|
||||
Установка загружает локальные файлы, без Git. До остановки службы проверяются
|
||||
доступность WAN и поддержка таблицы реальной командой ip на роутере.
|
||||
На роутер загружается и файл LuCI: версия v116 появляется и в основной
|
||||
На роутер загружается и файл LuCI: версия v117 появляется и в основной
|
||||
CGI-панели, и в странице входа в неё через LuCI.
|
||||
Сначала сохраняются файлы
|
||||
и конфигурация; после замены проверяется heartbeat/health. При ошибке
|
||||
автоматически восстанавливается состояние перед текущей попыткой.
|
||||
Сохранённая резервная копия для ручного отката не перезаписывается.
|
||||
|
||||
* Роутер: `/root/owrt-remote-backup-before-host-policy-v116` — агент,
|
||||
* Роутер: `/root/owrt-remote-backup-before-host-policy-v117` — агент,
|
||||
helper, init, CGI, UCI-конфигурация и Xray-конфиг (в закрытом каталоге).
|
||||
* VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v116`.
|
||||
* VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v117`.
|
||||
База роутеров и настройки авторизации не заменяются.
|
||||
|
||||
Успешная установка заканчивается `HOST_POLICY_ROUTER_V116_OK` на роутере
|
||||
и `HOST_POLICY_VPS_V116_Install_OK` на VPS. Старые резервные копии предыдущей
|
||||
версии сохраняются; новые команды отката используют отдельные копии для v116.
|
||||
Успешная установка заканчивается `HOST_POLICY_ROUTER_V117_OK` на роутере
|
||||
и `HOST_POLICY_VPS_V117_Install_OK` на VPS. Старые резервные копии предыдущей
|
||||
версии сохраняются; новые команды отката используют отдельные копии для v117.
|
||||
|
||||
### Переход с nftables-маркировки на адресную маршрутизацию
|
||||
|
||||
@@ -148,11 +148,11 @@ CGI-панели, и в странице входа в неё через LuCI.
|
||||
Обновление VPS для этого изменения не требуется.
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v116" -Action Install
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v116" -Action Rollback
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v117" -Action Install
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v117" -Action Rollback
|
||||
```
|
||||
|
||||
Копия: `/root/owrt-remote-backup-before-podkop-compat-v116`.
|
||||
Копия: `/root/owrt-remote-backup-before-podkop-compat-v117`.
|
||||
После установки повторите диагностику Podkop. Предупреждение от прежних правил
|
||||
Remote должно исчезнуть. Настоящие дополнительные правила других приложений
|
||||
по-прежнему обнаруживаются штатной диагностикой.
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
# Persistent client journal (v117 local fix)
|
||||
|
||||
The helper is `/usr/lib/owrt-remote/logs.sh`. It changes only Remote's private
|
||||
journal, not logd, network configuration, policy routing or Xray configuration.
|
||||
|
||||
## Storage and limits
|
||||
|
||||
- Persistent: normally `/overlay/owrt-remote/logs`, only when `/overlay` is an
|
||||
actual writable JFFS2/UBIFS/ext4/f2fs/ext2/ext3/btrfs mount. A directly mounted
|
||||
persistent root uses `/etc/owrt-remote/logs`. Root OverlayFS and tmpfs are
|
||||
rejected. Existing ancestors/symlinks are resolved against `/proc/mounts`;
|
||||
the longest containing mount wins. An explicit `OWRT_REMOTE_LOG_DIR` is
|
||||
validated the same way.
|
||||
- Runtime: `/tmp/owrt-remote-log-runtime`, which must be on tmpfs. Contains a
|
||||
buffer <=16 KiB, <=64 hashed dedup entries, a heartbeat fingerprint, PID locks,
|
||||
one export <=16 KiB, one HTTP body about 22 KiB, one viewer snapshot <=16 KiB
|
||||
plus a bounded filename list. No whole-journal shell variable or RAM archive.
|
||||
HTTP response files are limited by a sender-only RLIMIT_FSIZE (32 KiB in
|
||||
BusyBox ash); this does not alter heartbeat or Xray process limits.
|
||||
- Flash data: <=2 MiB across <=256 immutable chunks, each <=16 KiB. This is an
|
||||
upper bound, not guaranteed capacity: many small chunks reach the file-count
|
||||
bound first. Metadata, filesystem allocation units and the one quarantined
|
||||
damaged chunk add small overhead beyond the raw-data bound.
|
||||
- Rotation runs each minute. Acknowledged chunks expire after 24 hours or
|
||||
earlier to admit new chunks. Unacknowledged chunks do not expire. If the
|
||||
bounded queue/RAM buffer fills, **new** events are skipped; existing
|
||||
unacknowledged chunks remain intact. A warning and dropped-event count make
|
||||
that loss visible. A bounded queue cannot preserve infinitely many events
|
||||
during an infinitely long outage.
|
||||
- Default batch flush: once per 60 seconds, at most 16 KiB of new data; explicit
|
||||
`flush force`, normal service stop and shutdown can flush sooner. There is no
|
||||
fsync per line. Idle polls do not rewrite persistent state. Metadata and
|
||||
filesystem write amplification are additional to the data rate; no universal
|
||||
Flash-lifetime guarantee is possible across devices/filesystems.
|
||||
- Less than 2 MiB free space or write/mount failure stops Flash writes and
|
||||
switches this boot to a warning-marked RAM fallback: <=64 KiB data in <=8
|
||||
chunks +16 KiB buffer. The old Flash queue is left intact. Restore storage
|
||||
and reboot to resume that queue. No automatic mid-boot merging of two journal
|
||||
identities, which could acknowledge the wrong records.
|
||||
|
||||
## Crash and synchronization rules
|
||||
|
||||
Each chunk filename includes journal identity, start/end byte offsets and a validated SHA-256/MD5/POSIX cksum digest. Data is written to `chunk.new` on the persistent filesystem and renamed;
|
||||
state is written to `state.new` on that filesystem and atomically replaced.
|
||||
Batch sync (`sync -f`, or BusyBox `sync` fallback) completes durability. A small
|
||||
RAM in-flight marker reconciles a killed flusher before another append. At
|
||||
boot, checksums reject partial/corrupt chunks, incomplete temporary files are
|
||||
removed, and a completed rename beyond the saved cursor is recovered. One
|
||||
corrupt chunk can be retained as `damaged.log` for inspection. A reboot never
|
||||
clears a valid journal.
|
||||
|
||||
Last-minute events still in RAM can be lost on sudden power removal. This is
|
||||
the explicit buffering tradeoff, not a claim of per-line durable logging.
|
||||
|
||||
Only a successful authenticated Hub reply advances the ack. The recent ack is
|
||||
checkpointed with the next periodic batch before pruning; if power is lost
|
||||
first, records may replay with the same journal/offset. Existing Hub ingestion
|
||||
suppresses these duplicates. Packets are <=16 KiB raw and transport them from
|
||||
files without an entire packet in a shell variable. One `logs-sync` sender is
|
||||
allowed; stale locks and stale payloads recover on the next attempt. Explicit
|
||||
local/Hub clear intentionally discards both confirmed and unconfirmed data.
|
||||
Log-sender curl temporaries use a private runtime subdirectory, reclaimed on
|
||||
the next attempt after SIGKILL; heartbeat's temporary directory is untouched.
|
||||
|
||||
The local installer replaces the helper, log service, agent, CGI and LuCI view. It
|
||||
does not restart the tunnel, heartbeat or network. With the previous collector
|
||||
stopped, it streams the old bounded RAM journal into Flash chunks, preserving
|
||||
its journal ID, total cursor, ack and clear token. Already delivered bytes are
|
||||
not resent; undelivered records remain eligible for delivery. No FIFO, lock or
|
||||
temporary request is migrated. A same-filesystem staging manifest allows a
|
||||
power-interrupted migration to finish even after the old /tmp has disappeared.
|
||||
Migration needs 6 MiB free Flash; if it cannot safely migrate legacy data, the
|
||||
local installer fails and restores code instead of silently deleting history.
|
||||
With no usable Flash, only a bounded legacy tail is retained in RAM with an
|
||||
explicit loss warning, and the former large RAM files are removed. An ordinary
|
||||
upgrade stops the previous collector before replacing files and invokes this
|
||||
migration on log-service startup. Rollback restores code/service state and
|
||||
leaves Flash logs intact for recovery; the old helper starts a fresh RAM log.
|
||||
|
||||
## Automated checks
|
||||
|
||||
```powershell
|
||||
$env:OWRT_LOG_TEST_SHELL = "$PWD\.test-output\busybox\busybox64.exe"
|
||||
python -m unittest discover -s tests -p test_persistent_logs.py -v
|
||||
Remove-Item Env:OWRT_LOG_TEST_SHELL
|
||||
python -m unittest discover -s tests -p test_global_logs.py -v
|
||||
python -m unittest discover -s tests -p test_router_log_transport.py -v
|
||||
```
|
||||
|
||||
Tests use real shell functions and disposable mount/filesystem fixtures. The
|
||||
BusyBox Windows build exercises ash and its applets; it is **not** a claim of
|
||||
testing OpenWrt 24.10.3's exact BusyBox/kernel/UBIFS or physical power loss.
|
||||
|
||||
## Device checks before release
|
||||
|
||||
Install locally, then inspect from Windows PowerShell:
|
||||
|
||||
```powershell
|
||||
ssh root@192.168.2.1 'sh /usr/lib/owrt-remote/logs.sh status; df -h /overlay /tmp; free; ubus call service list; owrt-remote heartbeat-diagnostics'
|
||||
```
|
||||
|
||||
Expect `persistent=1`, a persistent storage path and tmpfs runtime. Confirm the
|
||||
mount explicitly using `mount` and verify `state` and `part.*` appear on Flash.
|
||||
Append a marker and flush before a user-planned reboot:
|
||||
|
||||
```powershell
|
||||
ssh root@192.168.2.1 'printf "persistent-check\n" | sh /usr/lib/owrt-remote/logs.sh append; sh /usr/lib/owrt-remote/logs.sh flush force; sh /usr/lib/owrt-remote/logs.sh status'
|
||||
```
|
||||
|
||||
After reboot, `read` must still show the marker and the same journal ID. Test an
|
||||
outage only on a controlled router with a separate management path; do not
|
||||
change firewall/WAN settings on a remote production router merely for this
|
||||
test. During outage, watch data/chunk/runtime bounds; after reconnect, watch
|
||||
`ack_bytes` advance without duplicate Hub entries. Launch concurrent
|
||||
`owrt-remote logs-sync` calls and inspect service/process counts. Confirm UI,
|
||||
heartbeat, Xray and WAN direct remain functional. Actual sustained RAM,
|
||||
Flash-write and reboot observations must be collected on the device.
|
||||
|
||||
## VPS archive and malformed-fragment recovery
|
||||
|
||||
The Hub retains its existing persistent SQLite archive, normally at
|
||||
`/var/lib/owrt-remote/hub.db`, <=2 MiB of content per router. No new RAM archive
|
||||
or whole-journal queue is introduced on the VPS. The live view still transfers
|
||||
only the last 64 KiB. A storage diagnostic resolves DB symlinks and the longest
|
||||
containing Linux mount, marking tmpfs/ramfs as nonpersistent. The dedicated
|
||||
`deploy-persistent-logs-vps.ps1` verifies the running Hub's actual DB path and
|
||||
writable disk before installing code. It preserves the database and all settings;
|
||||
rollback replaces code only. It refuses installation over an unconfirmed/RAM DB
|
||||
rather than automatically relocating a live database.
|
||||
|
||||
Some OpenWrt builds lack the optional `cksum` applet. The previous local helper
|
||||
failed to check its exit status and could publish names ending in an empty digest.
|
||||
Export skipped these files, but the empty status packet claimed the full cursor;
|
||||
the previous Hub accepted it and falsely acknowledged missing data. Regression
|
||||
checks now simulate missing cksum and failed checksum tools. New chunks use an
|
||||
available SHA-256/MD5/POSIX checksum with validated output. An empty status packet
|
||||
cannot advance the delivered cursor on either side.
|
||||
|
||||
On upgrade, a versioned recovery marker forces inspection even during the same
|
||||
boot. Retained malformed fragments of the expected byte length are copied into a
|
||||
same-filesystem recovery generation with real digests and a committed migration
|
||||
manifest. Old falsely acknowledged offsets cannot suppress that generation.
|
||||
Recovery requires enough free disk for the bounded copy and retains originals
|
||||
until the new manifest is committed. This recovers retained bytes, not already
|
||||
removed files, and cannot retroactively verify their former integrity. A warning
|
||||
records that limitation. A new generation can repeat already delivered history
|
||||
if mixed with affected fragments; normal retries remain deduplicated by offsets.
|
||||
|
||||
Run the real client spool-to-Hub ingestion regression as well as storage/deploy
|
||||
checks:
|
||||
|
||||
```powershell
|
||||
python -m unittest discover -s tests -p test_router_log_archive.py -v
|
||||
python -m unittest discover -s tests -p test_persistent_logs_vps_deploy.py -v
|
||||
```
|
||||
@@ -1,4 +1,4 @@
|
||||
# reCAPTCHA v3 в OpenWrt Remote Hub v116
|
||||
# reCAPTCHA v3 в OpenWrt Remote Hub v117
|
||||
|
||||
В меню владельца, в блоке «Капча на экране входа», доступны цифры,
|
||||
Google reCAPTCHA v2 Checkbox и Google reCAPTCHA v3. Установка обновления
|
||||
@@ -74,8 +74,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-recaptcha-v3
|
||||
непосредственно перед попыткой. Резервные копии при повторном запуске
|
||||
не перезаписываются:
|
||||
|
||||
* `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v116`
|
||||
* `/opt/owrt-remote/captcha-before-recaptcha-v3-v116.json` (режим 0600).
|
||||
* `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v117`
|
||||
* `/opt/owrt-remote/captcha-before-recaptcha-v3-v117.json` (режим 0600).
|
||||
|
||||
Скрипт рассчитан на стандартные пути установленного Hub:
|
||||
`/opt/owrt-remote/owrt-remote-hub.py`, `/var/lib/owrt-remote/hub-auth.json`
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Аудит ресурсов OpenWrt Remote Hub v116
|
||||
# Аудит ресурсов OpenWrt Remote Hub v117
|
||||
|
||||
Исправления внесены поверх `7f809fd`. Версия панели остаётся v116.
|
||||
Исправления внесены поверх `7f809fd`. Версия панели остаётся v117.
|
||||
|
||||
## SQLite
|
||||
|
||||
@@ -115,7 +115,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix
|
||||
|
||||
Установщик загружает локальный файл без Git, запускает изолированные тесты,
|
||||
сохраняет исходный серверный файл:
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v116`,
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v117`,
|
||||
атомарно заменяет Python файл, перезапускает `owrt-remote` и проверяет health.
|
||||
Существующая резервная копия не перезаписывается. При ошибке установки
|
||||
восстанавливается файл, работавший непосредственно перед текущей попыткой.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Виджеты ресурсов VPS (v116)
|
||||
# Виджеты ресурсов VPS (v117)
|
||||
|
||||
Три карточки ЦП, ОЗУ и ДИСК находятся над кнопками «Рестарт Xray VPS»,
|
||||
профилем и выходом. На ПК ширина и высота совпадают с кнопками под ними.
|
||||
@@ -23,7 +23,7 @@ vCPU и объёмы занятой/общей памяти и диска.
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-vps-widgets.ps1" -Vps "root@193.233.82.38" -Action Install
|
||||
```
|
||||
|
||||
После `VPS_WIDGETS_V116_Install_OK` обновите страницу через Ctrl+F5.
|
||||
После `VPS_WIDGETS_V117_Install_OK` обновите страницу через Ctrl+F5.
|
||||
Скрипт загружает локальный файл, проверяет его до замены, перезапускает
|
||||
службу и проверяет health. При ошибке после замены восстанавливает файл
|
||||
перед текущей попыткой. База данных и настройки не заменяются.
|
||||
@@ -33,8 +33,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
|
||||
```
|
||||
|
||||
Откат использует неизменяемую копию первого состояния до установки:
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v116`.
|
||||
Успешное завершение: `VPS_WIDGETS_V116_Rollback_OK`.
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v117`.
|
||||
Успешное завершение: `VPS_WIDGETS_V117_Rollback_OK`.
|
||||
|
||||
## Локальные проверки
|
||||
|
||||
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
# WAN IPv4 DNS fallback (local v116 fix)
|
||||
# WAN IPv4 DNS fallback (local v117 fix)
|
||||
|
||||
`files/usr/lib/owrt-remote/wan.sh` now explicitly queries
|
||||
`nslookup -type=A -timeout=3 -retry=1 "$host" "$dns"`.
|
||||
@@ -58,7 +58,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
|
||||
```
|
||||
|
||||
The installer saves the first backup at
|
||||
`/root/owrt-remote-backup-before-wan-dns-v116/wan.sh`, checks shell syntax and
|
||||
`/root/owrt-remote-backup-before-wan-dns-v117/wan.sh`, checks shell syntax and
|
||||
reloads only the supervised heartbeat instance so it reads the new helper.
|
||||
It does not restart Xray or regenerate configuration/routing. Installation
|
||||
verification/reload failures restore the file from that attempt.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Несколько WAN интерфейсов и резервирование (v116)
|
||||
# Несколько WAN интерфейсов и резервирование (v117)
|
||||
|
||||
Кнопка **Выбрать интерфейсы** раскрывает список с галочками. Можно отметить
|
||||
один, два, три и больше интерфейсов. Список содержит имена
|
||||
@@ -46,7 +46,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
|
||||
|
||||
Скрипт устанавливает CGI, LuCI, агент, WAN helper и hotplug вместе. До замены
|
||||
сохраняются прежние файлы, UCI-настройки и Xray config в
|
||||
`/root/owrt-remote-backup-before-wan-multi-v116`.
|
||||
`/root/owrt-remote-backup-before-wan-multi-v117`.
|
||||
Успех подтверждает `WAN_MULTI_ROUTER_Install_OK`. После установки обновить
|
||||
панель роутера через Ctrl+F5, отметить интерфейсы и сохранить выбор.
|
||||
VPS обновлять не требуется. Старый `deploy-wan-interface-picker.ps1`
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# Восстановление Xray после пропажи выбранного WAN (v117)
|
||||
|
||||
Сообщения `No selected WAN has a physical IPv4 default route`, ошибки
|
||||
генерации конфигурации и `crash loop 6 crashes` описывают последовательность:
|
||||
временно исчез маршрут на выбранном uplink, каждый запуск агента завершался,
|
||||
а procd прекратил повторные попытки запуска Xray. Затем Heartbeat смог
|
||||
восстановиться самостоятельно. Он делает HTTP-запрос, поэтому его успех
|
||||
не подтверждает работу двух обратных соединений Xray для Админки и SSH.
|
||||
|
||||
В прежнем `wan_reconcile_xray` остановленный Xray игнорировался; при возврате
|
||||
того же интерфейса также ничего не происходило. Это подтверждённый дефект
|
||||
восстановления. Причину исчезновения исходного маршрута и фактическое
|
||||
текущее состояние Xray на удалённом устройстве только эти логи не определяют.
|
||||
|
||||
## Исправление
|
||||
|
||||
- Перед генерацией JSON служебный процесс ожидает выбранный WAN с физическим
|
||||
IPv4 default route. Проверка каждые 5 секунд, одна запись о начале ожидания
|
||||
и одна о возвращении WAN. Конфигурация в ожидании не заменяется.
|
||||
- Ошибки UUID, JSON или проверки Xray при доступном WAN не маскируются
|
||||
ожиданием сети: генерация по-прежнему завершается с настоящей ошибкой.
|
||||
- Heartbeat не перезапускает всю службу, пока все выбранные WAN недоступны.
|
||||
- После подготовки WAN проверяются процесс Xray и состояние procd. Если
|
||||
Xray отсутствует и его instance не работает, но служба включена и Heartbeat
|
||||
работает, повторно регистрируется только instance `xray` через `service add`.
|
||||
Работающий процесс запуска, который ожидает WAN или готовит Xray, не трогается.
|
||||
- Живой Heartbeat и остальные instance не удаляются. При прежнем здоровом
|
||||
туннеле на том же интерфейсе перезапуск не запрашивается.
|
||||
- Блокировка в RAM исключает одновременные запросы восстановления; PID
|
||||
завершившегося worker освобождается. Повторные запросы ограничены интервалом
|
||||
300 секунд, в том числе при ошибке ubus. Успешный ответ ubus означает принятие
|
||||
запроса запуска, а не доказанную доступность Админки.
|
||||
- Явная остановка через панель (`enabled=0`) или остановка всей службы
|
||||
не отменяется автоматикой. Невыбранные интерфейсы не используются.
|
||||
- Источники установки, настройки LAN, DNS, Podkop и логика WAN policy routing
|
||||
не меняются этим исправлением.
|
||||
|
||||
Формат `service add` и строковых параметров respawn сверялся с исходниками:
|
||||
|
||||
- https://github.com/openwrt/procd/blob/master/service/service.c
|
||||
- https://github.com/openwrt/procd/blob/master/service/instance.c
|
||||
- https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/procd/files/procd.sh
|
||||
|
||||
## Установка и откат
|
||||
|
||||
Windows PowerShell 5.1, роутер пользователя:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Install
|
||||
```
|
||||
|
||||
Скрипт обновляет агент и WAN helper парой, сохраняет первую копию в
|
||||
`/root/owrt-remote-backup-before-wan-recovery-v117`, проверяет shell-синтаксис
|
||||
и побайтовое совпадение. Затем перезагружает только работающий Heartbeat,
|
||||
чтобы в RAM не остался прежний shell с уже загруженными функциями.
|
||||
Туннель при исправном Xray не перезапускается. Остановленный Heartbeat
|
||||
не запускается принудительно. Ошибка установки восстанавливает файлы
|
||||
до текущей попытки. UCI и рабочий JSON скрипт установки не заменяет;
|
||||
обычный запуск Xray выполняет штатную проверку и генерацию конфигурации.
|
||||
|
||||
Установка подтверждается `WAN_RECOVERY_ROUTER_Install_OK`, это подтверждение
|
||||
установки кода. Проверка состояния:
|
||||
|
||||
```powershell
|
||||
ssh root@192.168.2.1 'owrt-remote status; owrt-remote heartbeat-diagnostics; logread -e owrt-remote | tail -n 60'
|
||||
```
|
||||
|
||||
Откат к коду до первого применения:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
|
||||
```
|
||||
|
||||
Для роутера друга нужен его собственный доступный SSH-адрес, VPN или человек
|
||||
в локальной сети. Через оборванные обратные SSH/HTTP-туннели установить код
|
||||
невозможно. Если WAN уже вернулся, для разового восстановления на самом
|
||||
роутере можно выполнить `/etc/init.d/owrt-remote restart`, затем проверить
|
||||
`owrt-remote status`. Сам по себе этот перезапуск не устанавливает исправление.
|
||||
|
||||
## Локальная проверка
|
||||
|
||||
`test_wan_recovery.py` моделирует длительное отсутствие маршрута, возврат
|
||||
того же интерфейса, остановленный Xray при работающем Heartbeat, начальную
|
||||
подготовку, исправный туннель, ручную остановку, параллельные запросы,
|
||||
оставшуюся блокировку, отказ ubus и настоящую ошибку генерации JSON.
|
||||
Поддерживает `OWRT_RECOVERY_TEST_SHELL` для BusyBox ash.
|
||||
|
||||
`test_wan_recovery_deploy.py` проверяет установку, повторную установку,
|
||||
сохранение первой резервной копии, откат, отсутствие копии, повреждённый
|
||||
payload, отказ проверки и отказ перезапуска Heartbeat без реального SSH.
|
||||
|
||||
Дополнительно запускаются регрессии WAN failover, host policy, HTTP/HTTPS
|
||||
авторизации и перенаправлений. Проверка BusyBox ash выполняется локальной
|
||||
Windows-сборкой BusyBox; OpenWrt 24.10.3 и настоящий procd на устройстве
|
||||
требуют проверки пользователем. Прямой WAN на удалённом устройстве не
|
||||
отключать для искусственного теста без независимого канала доступа.
|
||||
@@ -1,4 +1,4 @@
|
||||
# Исправление запуска прямого WAN в v116
|
||||
# Исправление запуска прямого WAN в v117
|
||||
|
||||
На проблемном роутере обычный `wan` был поднят, имел IPv4-адрес,
|
||||
маршрут и рабочий DNS. Проверка `ip -4 route show table 210` возвращала
|
||||
@@ -16,7 +16,7 @@ instance после шести падений. Из-за остановленн
|
||||
конфигурации останавливают агент до запуска Xray.
|
||||
|
||||
Изменены только `files/usr/lib/owrt-remote/wan.sh` и
|
||||
`files/usr/sbin/owrt-remote`. Версия остаётся v116. Настройки Podkop,
|
||||
`files/usr/sbin/owrt-remote`. Версия остаётся v117. Настройки Podkop,
|
||||
LAN и источники установки GitHub/GitLab не меняются. Обновление VPS
|
||||
для этого исправления не требуется.
|
||||
|
||||
@@ -28,7 +28,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
|
||||
|
||||
Скрипт перед заменой проверяет синтаксис и подготовку WAN новым helper,
|
||||
сохраняет агент, helper, UCI и Xray-конфигурацию в
|
||||
`/root/owrt-remote-backup-before-wan-stability-v116`. Сохранённая копия
|
||||
`/root/owrt-remote-backup-before-wan-stability-v117`. Сохранённая копия
|
||||
не перезаписывается при повторной установке. После замены выполняются
|
||||
перезапуск, проверка процесса Xray и настоящий heartbeat до Hub.
|
||||
При ошибке восстанавливается состояние до текущей попытки.
|
||||
|
||||
+6
-6
@@ -1,4 +1,4 @@
|
||||
# Применение OpenWrt config через панель роутера (v116)
|
||||
# Применение OpenWrt config через панель роутера (v117)
|
||||
|
||||
При отправке textarea браузер кодирует переносы строк как CRLF. CGI
|
||||
декодировал их без нормализации и записывал CR в исполняемый shell-скрипт.
|
||||
@@ -22,7 +22,7 @@ CRLF и LF, heredoc, пустые строки, кодированные сим
|
||||
|
||||
Скрипт обновляет только CGI на роутере. VPS обновлять для этого фикса
|
||||
не требуется. До замены сохраняется первая резервная копия CGI:
|
||||
`/root/owrt-remote-backup-before-xray-apply-v116/owrt-remote`.
|
||||
`/root/owrt-remote-backup-before-xray-apply-v117/owrt-remote`.
|
||||
|
||||
Из PowerShell Windows 11; замените IP, если адрес роутера отличается:
|
||||
|
||||
@@ -53,7 +53,7 @@ python tests/test_xray_apply.py -v
|
||||
|
||||
Переносы строк в формах описаны в [стандарте HTML](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html).
|
||||
|
||||
## Мобильная вставка команд (локальный фикс v116)
|
||||
## Мобильная вставка команд (локальный фикс v117)
|
||||
|
||||
Воспроизведён отдельный случай `rc=255`: в строке
|
||||
`uci set owrtremote.main.router_name=‘Дом у моря’` типографские кавычки
|
||||
@@ -77,7 +77,7 @@ python tests/test_xray_apply.py -v
|
||||
|
||||
Обновляется только `/www/cgi-bin/owrt-remote` на роутере, без перезапуска
|
||||
служб и изменений настроек при установке. Резервная копия отдельная:
|
||||
`/root/owrt-remote-backup-before-xray-mobile-v116/owrt-remote`.
|
||||
`/root/owrt-remote-backup-before-xray-mobile-v117/owrt-remote`.
|
||||
|
||||
Установка из PowerShell Windows 11:
|
||||
|
||||
@@ -101,7 +101,7 @@ python tests/test_xray_mobile_deploy.py -v
|
||||
```
|
||||
|
||||
|
||||
## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v116)
|
||||
## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v117)
|
||||
|
||||
В предоставленном журнале Xray 26.3.27 запускается успешно, а завершающая
|
||||
команда `owrt-remote heartbeat` возвращает `rc=1` с `Failed to allocate
|
||||
@@ -153,7 +153,7 @@ CGI получает этап выполнения от агента. Ошибк
|
||||
Новый скрипт `vps/deploy-xray-validation-fix.ps1` обновляет CGI и агент,
|
||||
не пересобирает настройки и не перезапускает службы при установке/откате.
|
||||
Первая резервная копия обоих файлов:
|
||||
`/root/owrt-remote-backup-before-xray-validation-v116`. Если установка
|
||||
`/root/owrt-remote-backup-before-xray-validation-v117`. Если установка
|
||||
оборвалась или проверка файлов не прошла, возвращаются файлы до этой попытки.
|
||||
На роутере друга нужно указать его доступный SSH-адрес вместо локального IP.
|
||||
|
||||
|
||||
+12
-90
@@ -1,4 +1,4 @@
|
||||
"""Exercise the real RAM journal and authorized router CGI with temporary files."""
|
||||
"""Exercise the bounded persistent journal and authorized router CGI with temporary files."""
|
||||
import concurrent.futures
|
||||
import base64
|
||||
import json
|
||||
@@ -22,12 +22,14 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.log_dir = self.root / 'journal'
|
||||
self.run_dir = self.root / 'runtime'
|
||||
self.helper = self.root / 'logs.sh'
|
||||
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
|
||||
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
|
||||
'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
|
||||
self.helper_source = source
|
||||
source = source.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
|
||||
source = source.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
self.cgi = self.root / 'cgi'
|
||||
self.cgi.write_text((ROOT / 'files/www/cgi-bin/owrt-remote').read_text(encoding='utf-8'),
|
||||
@@ -35,12 +37,15 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.key = self.root / 'key'
|
||||
self.key.write_text('test-key', encoding='utf-8')
|
||||
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': self.log_dir.as_posix(),
|
||||
'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
|
||||
'OWRT_REMOTE_LOG_HELPER': self.helper.as_posix(), 'TMPDIR': self.root.as_posix(),
|
||||
'KEY_FILE': self.key.as_posix(), 'TEST_UPTIME': '1000'}
|
||||
if os.name == 'nt':
|
||||
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
|
||||
|
||||
def helper_call(self, command, text='', *args, env=None):
|
||||
if command == 'export':
|
||||
self.helper_call('flush', '', 'force', env=env)
|
||||
result = subprocess.run([SH, self.helper.as_posix(), command, *args], input=text,
|
||||
encoding='utf-8', capture_output=True, env=env or self.env, timeout=20)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
@@ -61,8 +66,9 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.assertIn('Xray connected', downloaded)
|
||||
self.assertIn('Cache-Control: no-store', downloaded)
|
||||
# The bounded read is independent of append filtering; supply an existing large journal.
|
||||
with (self.log_dir / 'current.log').open('a', encoding='utf-8') as journal:
|
||||
journal.write(''.join(f'{i}: ' + 'y' * 1024 + '\n' for i in range(70)))
|
||||
state = (self.log_dir / 'state').read_text().split()
|
||||
for i in range(5):
|
||||
(self.log_dir / f'part.{state[0]}.{i:018d}.{i+1:018d}.1').write_text('y' * 15000 + '\n')
|
||||
live = self.cgi_call('logs_live').split('\n\n', 1)[1]
|
||||
# Git for Windows can translate CRLF inside native pipes.
|
||||
self.assertLessEqual(len(live.encode()), 65536)
|
||||
@@ -141,32 +147,6 @@ class GlobalLogTests(unittest.TestCase):
|
||||
lines = self.helper_call('read').splitlines()
|
||||
self.assertEqual([line.split(']', 1)[0] for line in lines], ['[ERROR', '[ERROR', '[ERROR', '[WARN'])
|
||||
|
||||
def test_persistent_storage_is_rejected_before_any_file_is_written(self):
|
||||
mounts = self.root / 'mounts'
|
||||
mounts.write_text('/dev/root / overlay rw 0 0\n', encoding='utf-8')
|
||||
source = self.helper_source.replace('/proc/mounts', mounts.as_posix())
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
env = self.env
|
||||
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not be written\n',
|
||||
encoding='utf-8', capture_output=True, env=env, timeout=20)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('refusing persistent storage', result.stderr)
|
||||
self.assertFalse(self.log_dir.exists())
|
||||
self.env = env
|
||||
self.assertIn('503 Service Unavailable', self.cgi_call('logs_live'))
|
||||
self.assertFalse(self.log_dir.exists())
|
||||
# The same location is permitted only when its containing mount is tmpfs.
|
||||
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix() + ' tmpfs rw 0 0\n', encoding='utf-8')
|
||||
self.helper_call('append', 'RAM fact\n')
|
||||
self.assertEqual(self.helper_call('read'), '[OK] RAM fact\n')
|
||||
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix()
|
||||
+ ' tmpfs rw 0 0\n/dev/flash ' + self.log_dir.as_posix()
|
||||
+ ' overlay rw 0 0\n', encoding='utf-8')
|
||||
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not enter nested Flash mount\n',
|
||||
encoding='utf-8', capture_output=True, env=env, timeout=20)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual((self.log_dir / 'current.log').read_text(), '[OK] RAM fact\n')
|
||||
|
||||
def test_clear_requires_post_and_correct_key(self):
|
||||
self.helper_call('append', 'retain this event\n')
|
||||
self.assertIn('Доступ запрещен', self.cgi_call('logs_download', key='wrong'))
|
||||
@@ -178,13 +158,6 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.helper_call('append', 'new event\n')
|
||||
self.assertEqual(self.helper_call('read'), '[OK] new event\n')
|
||||
|
||||
def test_retention_uses_uptime_and_clears_on_reboot(self):
|
||||
self.helper_call('append', 'old event\n')
|
||||
self.assertIn('old event', self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87399'}))
|
||||
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87400'}), '')
|
||||
self.helper_call('append', 'after daily clear\n', env={**self.env, 'TEST_UPTIME': '87401'})
|
||||
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '20'}), '')
|
||||
|
||||
def test_export_acknowledges_only_received_bytes_and_remote_clear_is_idempotent(self):
|
||||
self.helper_call('append', 'First event\n')
|
||||
first = json.loads(self.helper_call('export'))
|
||||
@@ -207,39 +180,6 @@ class GlobalLogTests(unittest.TestCase):
|
||||
self.helper_call('ack', '', second['journal_id'], str(second['to']), 'clear-123')
|
||||
self.assertIn('After clear', self.helper_call('read'), 'retrying clear must not clear new events')
|
||||
|
||||
def test_export_catches_up_in_bounded_batches_after_rotation(self):
|
||||
self.helper_call('clear')
|
||||
raw = b'[OK] ' + b'x' * 100000 + b'\n'
|
||||
(self.log_dir / 'current.log').write_bytes(raw)
|
||||
first = json.loads(self.helper_call('export'))
|
||||
first_raw = base64.b64decode(first['data'])
|
||||
self.assertLessEqual(len(first_raw), 65536)
|
||||
self.helper_call('ack', '', first['journal_id'], str(first['to']))
|
||||
second = json.loads(self.helper_call('export'))
|
||||
self.assertEqual(first_raw + base64.b64decode(second['data']), raw)
|
||||
# Bytes dropped by local rotation are detected by the server from the gap.
|
||||
(self.log_dir / 'current.log').write_bytes(b'[OK] Latest\n')
|
||||
(self.log_dir / 'total').write_text('200000')
|
||||
late = json.loads(self.helper_call('export'))
|
||||
self.assertEqual(late['from'], 200000 - len(b'[OK] Latest\n'))
|
||||
self.assertEqual(base64.b64decode(late['data']), b'[OK] Latest\n')
|
||||
|
||||
def test_rotation_retains_recent_log_and_clear_removes_both(self):
|
||||
self.helper_call('clear')
|
||||
current = self.log_dir / 'current.log'
|
||||
current.write_text('A' * 1048576, encoding='utf-8')
|
||||
self.helper_call('append', 'second file\n')
|
||||
self.assertTrue((self.log_dir / 'previous.log').exists())
|
||||
self.assertEqual(current.read_text(), '[OK] second file\n')
|
||||
current.write_text('B' * 1048576, encoding='utf-8')
|
||||
self.helper_call('append', 'third file\n')
|
||||
output = self.helper_call('read')
|
||||
self.assertNotIn('A', output)
|
||||
self.assertTrue(output.endswith('third file\n'))
|
||||
self.helper_call('clear')
|
||||
self.assertFalse((self.log_dir / 'previous.log').exists())
|
||||
self.assertEqual(current.read_text(), '')
|
||||
|
||||
def test_secrets_are_redacted_before_storage_and_html_is_literal(self):
|
||||
text = ('Heartbeat OK\nuci set owrtremote.main.hub_token=SECRET_TOKEN\n'
|
||||
'Authorization: Bearer SECRET_AUTH\nhttps://hub.test/?key=SECRET_KEY\n'
|
||||
@@ -256,11 +196,11 @@ class GlobalLogTests(unittest.TestCase):
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
|
||||
list(pool.map(lambda i: self.helper_call('append', f'event-{i}\n'), range(12)))
|
||||
self.assertEqual(set(self.helper_call('read').splitlines()), {f'[OK] event-{i}' for i in range(12)})
|
||||
self.assertFalse((self.log_dir / 'lock').exists())
|
||||
self.assertFalse((self.run_dir / 'lock').exists())
|
||||
|
||||
def test_stale_lock_recovers(self):
|
||||
self.helper_call('clear')
|
||||
lock = self.log_dir / 'lock'
|
||||
lock = self.run_dir / 'lock'
|
||||
lock.mkdir()
|
||||
(lock / 'pid').write_text('99999999')
|
||||
self.helper_call('append', 'recovered\n')
|
||||
@@ -272,28 +212,10 @@ class GlobalLogTests(unittest.TestCase):
|
||||
|
||||
def test_empty_lock_left_by_killed_writer_recovers(self):
|
||||
self.helper_call('clear')
|
||||
(self.log_dir / 'lock').mkdir()
|
||||
(self.run_dir / 'lock').mkdir()
|
||||
self.helper_call('append', 'after empty lock\n')
|
||||
self.assertEqual(self.helper_call('read'), '[OK] after empty lock\n')
|
||||
|
||||
def test_collector_failure_cleans_both_children_and_fifo(self):
|
||||
source = self.helper.read_text(encoding='utf-8')
|
||||
mocks = '''
|
||||
log_init() { return 0; }
|
||||
mkfifo() { return 0; }
|
||||
logread() { return 0; }
|
||||
log_lock() { return 1; }
|
||||
kill() { printf 'KILL=%s\\n' "$*"; }
|
||||
wait() { return 0; }
|
||||
rm() { printf 'REMOVE=%s\\n' "$*"; }
|
||||
'''
|
||||
self.helper.write_text(source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1),
|
||||
encoding='utf-8', newline='\n')
|
||||
result = subprocess.run([SH, self.helper.as_posix(), 'collect'], capture_output=True,
|
||||
encoding='utf-8', env=self.env, timeout=10)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(result.stdout.count('KILL='), 2)
|
||||
self.assertIn('REMOVE=-f ' + self.log_dir.as_posix() + '/stream.', result.stdout)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -43,7 +43,7 @@ class GlobalLogsDeploymentTests(unittest.TestCase):
|
||||
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
|
||||
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
|
||||
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
|
||||
mappings['/root/owrt-remote-backup-before-global-logs-stop-v116'] = self.backup.as_posix()
|
||||
mappings['/root/owrt-remote-backup-before-global-logs-stop-v117'] = self.backup.as_posix()
|
||||
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
|
||||
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
|
||||
# Existing sandbox service scripts only print markers. Mock readiness.
|
||||
@@ -55,7 +55,7 @@ jsonfilter() { return 0; }
|
||||
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
|
||||
'''
|
||||
self.script = self.script.replace('for item in cgi agent init;', mocks + '\nfor item in cgi agent init;', 1)
|
||||
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix()}
|
||||
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
|
||||
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
|
||||
if os.name == 'nt':
|
||||
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
|
||||
@@ -75,6 +75,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
|
||||
if item == 'logs.sh':
|
||||
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
|
||||
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
|
||||
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
|
||||
if item == 'logs-init':
|
||||
text = '#!/bin/sh\necho logs-service-$1\n'
|
||||
if bad and item == 'cgi':
|
||||
@@ -103,7 +104,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
|
||||
self.assertIn('GLOBAL_LOGS_ROUTER_Install_OK', result.stdout)
|
||||
for item in ('cgi', 'luci'):
|
||||
installed = self.root / self.targets[item].lstrip('/')
|
||||
self.assertIn('v116', installed.read_text(encoding='utf-8'))
|
||||
self.assertIn('v117', installed.read_text(encoding='utf-8'))
|
||||
for item, data in self.initial.items():
|
||||
self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
|
||||
result = self.deploy('Rollback')
|
||||
@@ -130,7 +131,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
|
||||
self.assert_initial()
|
||||
|
||||
def test_rollback_restores_existing_log_service_and_helper(self):
|
||||
# This fix is normally applied on top of v116's already installed journal.
|
||||
# This fix is normally applied on top of v117's already installed journal.
|
||||
for item in ('logs.sh', 'logs-init'):
|
||||
target = self.root / self.targets[item].lstrip('/')
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -20,7 +20,7 @@ class VpsLogDeployTests(unittest.TestCase):
|
||||
self.target = self.root / 'hub.py'
|
||||
self.previous = b'print("previous Hub")\n'
|
||||
self.target.write_bytes(self.previous)
|
||||
self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v116'
|
||||
self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v117'
|
||||
source = (ROOT / 'vps/deploy-global-logs-vps.ps1').read_text(encoding='utf-8')
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
|
||||
@@ -55,7 +55,7 @@ sleep() { return 0; }
|
||||
for _ in range(2):
|
||||
result = self.deploy('Install')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('GLOBAL_LOGS_VPS_V116_Install_OK', result.stdout)
|
||||
self.assertIn('GLOBAL_LOGS_VPS_V117_Install_OK', result.stdout)
|
||||
self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
|
||||
self.assertEqual(self.backup.read_bytes(), self.previous)
|
||||
result = self.deploy('Rollback')
|
||||
|
||||
@@ -25,7 +25,7 @@ class HeartbeatTransportDeploymentTests(unittest.TestCase):
|
||||
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
|
||||
source = (ROOT / "vps/deploy-heartbeat-transport-fix.ps1").read_text(encoding="utf-8")
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v116",
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v117",
|
||||
"backup_dir=" + shlex.quote(self.backup.as_posix()))
|
||||
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
|
||||
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
|
||||
|
||||
@@ -0,0 +1,423 @@
|
||||
"""Persistent spool recovery/resource limits using the real shell helper.
|
||||
|
||||
Mount fixtures replace /proc/mounts only in disposable copies. Set
|
||||
OWRT_LOG_TEST_SHELL to a BusyBox executable to exercise its actual ash/applets.
|
||||
"""
|
||||
import base64
|
||||
import concurrent.futures
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
|
||||
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
|
||||
BUSYBOX = os.environ.get('OWRT_LOG_TEST_SHELL', '')
|
||||
|
||||
|
||||
class PersistentLogsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(prefix='persistent-logs-')
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name).resolve()
|
||||
self.run_dir = self.root / 'runtime'
|
||||
self.flash = self.root / 'flash'
|
||||
self.flash.mkdir()
|
||||
self.helper = self.root / 'logs.sh'
|
||||
self.mounts = self.root / 'mounts'
|
||||
self.root_mount = self.root.as_posix()
|
||||
self.flash_mount = self.flash.as_posix()
|
||||
if os.name == 'nt' and not BUSYBOX:
|
||||
# Git's bundled readlink may emit native C:/ or MSYS /c paths.
|
||||
# The fixture must use what the actual applet returns, not assume it.
|
||||
env = {**os.environ, 'PATH': str(GIT_BIN) + os.pathsep + os.environ['PATH']}
|
||||
def resolved_mount(path):
|
||||
return subprocess.check_output([SH, '-c', 'readlink -f "$1"', '--', path.as_posix()],
|
||||
env=env, text=True).strip()
|
||||
self.root_mount = resolved_mount(self.root)
|
||||
self.flash_mount = resolved_mount(self.flash)
|
||||
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
|
||||
f'/dev/ubi0_1 {self.flash_mount} ubifs rw 0 0\n')
|
||||
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
|
||||
source = source.replace('/proc/mounts', self.mounts.as_posix())
|
||||
if os.name == 'nt':
|
||||
source = source.replace('case "$path" in /*)', 'case "$path" in [a-zA-Z]:/*|/*)')
|
||||
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
|
||||
'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
|
||||
source = source.replace('log_epoch() { date +%s; }',
|
||||
'log_epoch() { echo "${TEST_EPOCH:-' + str(int(time.time())) + '}"; }')
|
||||
# Real df remains covered in integration tests. Fixtures emulate free space.
|
||||
source = source.replace('free="$(df -Pk "$LOG_DIR" | awk \'NR==2 {print $4}\')"',
|
||||
'free="${TEST_FREE_KB:-8192}"')
|
||||
self.source = source
|
||||
self.source = self.source.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + (self.root / 'legacy').as_posix())
|
||||
self.helper.write_text(self.source, encoding='utf-8', newline='\n')
|
||||
self.env = {**os.environ, 'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
|
||||
'OWRT_REMOTE_LOG_DIR': (self.flash / 'journal').as_posix()}
|
||||
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
|
||||
self.shell = [BUSYBOX, 'ash'] if BUSYBOX else [SH]
|
||||
if BUSYBOX:
|
||||
self.env['BB_OVERRIDE_APPLETS'] = '1'
|
||||
|
||||
def call(self, command, text='', *args, check=True, **flags):
|
||||
result = subprocess.run([*self.shell, self.helper.as_posix(), command, *args],
|
||||
input=text, encoding='utf-8', capture_output=True,
|
||||
env={**self.env, **flags}, timeout=25)
|
||||
if check:
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result
|
||||
|
||||
def status(self):
|
||||
return dict(line.split('=', 1) for line in self.call('status').stdout.splitlines())
|
||||
|
||||
def flush(self, **flags):
|
||||
return self.call('flush', '', 'force', **flags)
|
||||
|
||||
def export(self):
|
||||
return json.loads(self.call('export').stdout)
|
||||
|
||||
def append(self, text, **flags):
|
||||
return self.call('append', text, **flags)
|
||||
|
||||
def acknowledge(self, batch):
|
||||
self.call('ack', '', batch['journal_id'], str(batch['to']))
|
||||
|
||||
@property
|
||||
def journal(self):
|
||||
return self.flash / 'journal'
|
||||
|
||||
def reboot(self):
|
||||
shutil.rmtree(self.run_dir)
|
||||
|
||||
def test_persistent_storage_selected_and_survives_runtime_loss(self):
|
||||
self.append('before reboot\n')
|
||||
self.flush()
|
||||
first = self.export()
|
||||
self.acknowledge(first)
|
||||
self.flush()
|
||||
self.reboot()
|
||||
self.assertEqual(self.status()['persistent'], '1')
|
||||
self.assertIn('before reboot', self.call('read').stdout)
|
||||
after = self.export()
|
||||
self.assertEqual(after['journal_id'], first['journal_id'])
|
||||
self.assertEqual(after['from'], first['to'])
|
||||
self.assertEqual(after['data'], '')
|
||||
self.assertFalse(any(self.journal.glob('stream.*')))
|
||||
self.assertFalse((self.journal / 'lock').exists())
|
||||
|
||||
def test_missing_optional_cksum_does_not_create_empty_digest_or_drop_facts(self):
|
||||
source = self.source.replace('case "${1:-}" in', 'cksum() { return 127; }\ncase "${1:-}" in', 1)
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
self.append('first event\nsecond event\n')
|
||||
self.flush()
|
||||
part = next(self.journal.glob('part.*'))
|
||||
self.assertIn('.s256-', part.name)
|
||||
self.assertIn(b'second event', base64.b64decode(self.export()['data']))
|
||||
self.reboot()
|
||||
self.assertIn(b'first event', base64.b64decode(self.export()['data']))
|
||||
|
||||
def test_digest_fingerprint_cache_stays_small_at_64_entries(self):
|
||||
self.call('check')
|
||||
# Seed 63 existing fingerprints; exercise eviction with two real events
|
||||
# instead of starting thousands of Windows utility processes for a fill.
|
||||
(self.run_dir / 'facts').write_text(''.join('1000\ts256-%035d\n' % i for i in range(63)))
|
||||
self.append('distinct event 63\ndistinct event 64\n')
|
||||
self.assertEqual(len((self.run_dir / 'facts').read_text().splitlines()), 64)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
|
||||
|
||||
def test_empty_digest_files_falsely_acked_by_old_hub_are_recovered(self):
|
||||
self.append('retained restart event\n'); self.flush()
|
||||
before = self.export()
|
||||
self.acknowledge(before); self.flush()
|
||||
part = next(self.journal.glob('part.*'))
|
||||
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
|
||||
# Upgrade during the same boot must override the former recovery marker.
|
||||
(self.run_dir / 'recovered-v2').unlink()
|
||||
(self.run_dir / 'recovered').touch()
|
||||
repaired = self.export()
|
||||
self.assertNotEqual(repaired['journal_id'], before['journal_id'])
|
||||
self.assertTrue(repaired['journal_id'].startswith('recovered-'))
|
||||
self.assertEqual(repaired['from'], 0)
|
||||
self.assertIn(b'retained restart event', base64.b64decode(repaired['data']))
|
||||
self.assertFalse(any(p.name.endswith('.') for p in self.journal.glob('part.*')))
|
||||
self.acknowledge(repaired)
|
||||
self.assertEqual(self.export()['data'], '')
|
||||
self.flush()
|
||||
# Recovery's warning was buffered after the replay chunk; deliver it too.
|
||||
self.acknowledge(self.export())
|
||||
self.flush()
|
||||
self.reboot()
|
||||
self.assertEqual(self.export()['data'], '')
|
||||
|
||||
def test_checksum_failure_keeps_pending_and_does_not_advance_total(self):
|
||||
self.append('keep until checksum works\n')
|
||||
before = self.status()['committed_bytes']
|
||||
source = self.source.replace('log_checksum() {', 'log_checksum() { return 1;', 1)
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
self.call('flush', '', 'force', check=False)
|
||||
self.assertFalse(list(self.journal.glob('part.*')))
|
||||
self.assertEqual(self.status()['committed_bytes'], before)
|
||||
self.assertIn('keep until checksum works', (self.run_dir / 'pending').read_text())
|
||||
|
||||
def test_md5_and_legacy_numeric_checksum_formats_remain_readable(self):
|
||||
for digest in ('md5', 'crc'):
|
||||
with self.subTest(digest=digest):
|
||||
self.call('clear')
|
||||
source = self.source.replace('if command -v sha256sum >/dev/null 2>&1; then method=s256',
|
||||
'if false; then method=s256')
|
||||
if digest == 'crc':
|
||||
source = source.replace('elif command -v md5sum >/dev/null 2>&1; then method=md5',
|
||||
'elif false; then method=md5')
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
self.append('digest compatibility ' + digest + '\n'); self.flush()
|
||||
value = self.export()
|
||||
self.reboot()
|
||||
self.assertEqual(self.export(), value)
|
||||
self.assertIn(digest.encode(), base64.b64decode(value['data']))
|
||||
name = next(self.journal.glob('part.*')).name.rsplit('.', 1)[1]
|
||||
self.assertEqual(name.startswith('md5-'), digest == 'md5')
|
||||
|
||||
def test_empty_export_never_claims_unreadable_chunks_as_delivered(self):
|
||||
self.append('not yet delivered\n'); self.flush()
|
||||
part = next(self.journal.glob('part.*'))
|
||||
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.invalid'))
|
||||
empty = self.export()
|
||||
self.assertEqual((empty['from'], empty['to'], empty['data']), (0, 0, ''))
|
||||
|
||||
def test_failed_or_unmounted_overlay_does_not_receive_journal(self):
|
||||
for kind, flags in [('overlay', 'rw'), ('tmpfs', 'rw'), ('ubifs', 'ro')]:
|
||||
with self.subTest(kind=kind, flags=flags):
|
||||
if self.run_dir.exists(): self.reboot()
|
||||
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
|
||||
f'fake {self.flash_mount} {kind} {flags} 0 0\n')
|
||||
self.append('RAM fallback\n')
|
||||
self.assertEqual(self.status()['persistent'], '0')
|
||||
self.assertIn('Flash недоступна', self.call('read').stdout)
|
||||
self.assertFalse(self.journal.exists())
|
||||
|
||||
def test_nested_tmpfs_and_symlink_do_not_pass_flash_check(self):
|
||||
nested = self.flash / 'ram'
|
||||
nested.mkdir()
|
||||
self.mounts.write_text(self.mounts.read_text() + f'tmpfs {self.flash_mount}/ram tmpfs rw 0 0\n')
|
||||
self.env['OWRT_REMOTE_LOG_DIR'] = (nested / 'logs').as_posix()
|
||||
self.append('nested tmpfs\n')
|
||||
self.assertEqual(self.status()['persistent'], '0')
|
||||
self.assertFalse((nested / 'logs').exists())
|
||||
|
||||
def test_periodic_flush_and_idle_polling_do_not_rewrite_flash(self):
|
||||
self.call('check')
|
||||
state = self.journal / 'state'
|
||||
before = state.stat().st_mtime_ns
|
||||
self.append('buffered record\n')
|
||||
self.assertEqual(list(self.journal.glob('part.*')), [])
|
||||
self.assertEqual(self.export()['data'], '')
|
||||
self.call('flush', TEST_UPTIME='1059')
|
||||
self.assertEqual(list(self.journal.glob('part.*')), [])
|
||||
self.call('flush', TEST_UPTIME='1060')
|
||||
self.assertEqual(len(list(self.journal.glob('part.*'))), 1)
|
||||
self.assertNotEqual(state.stat().st_mtime_ns, before)
|
||||
stable = {p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}
|
||||
for _ in range(3):
|
||||
self.call('flush', TEST_UPTIME='1120')
|
||||
self.export()
|
||||
self.assertEqual({p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}, stable)
|
||||
|
||||
def test_buffer_and_number_of_archives_are_bounded_without_dropping_unacked(self):
|
||||
self.helper.write_text(self.source.replace('LOG_LIMIT=2097152', 'LOG_LIMIT=32768')
|
||||
.replace('LOG_PART_LIMIT=256', 'LOG_PART_LIMIT=2'),
|
||||
encoding='utf-8', newline='\n')
|
||||
for i in range(4):
|
||||
self.append(f'event {i} ' + 'x' * 4000 + '\n')
|
||||
self.flush()
|
||||
parts = sorted(self.journal.glob('part.*'))
|
||||
self.assertEqual(len(parts), 2)
|
||||
original = {p.name: p.read_bytes() for p in parts}
|
||||
for i in range(5):
|
||||
self.append(f'overflow {i} ' + 'y' * 4000 + '\n')
|
||||
self.flush()
|
||||
self.assertEqual({p.name: p.read_bytes() for p in parts}, original)
|
||||
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
|
||||
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
|
||||
first = self.export()
|
||||
self.acknowledge(first)
|
||||
self.flush()
|
||||
self.assertFalse(parts[0].exists(), 'only acknowledged chunks make room')
|
||||
self.assertIn('пропущено новых событий', self.call('read').stdout)
|
||||
|
||||
def test_ack_is_monotonic_and_does_not_delete_unconfirmed_after_one_day(self):
|
||||
self.append('not delivered\n')
|
||||
self.flush()
|
||||
batch = self.export()
|
||||
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
|
||||
self.assertEqual(self.export(), batch)
|
||||
invalid = self.call('ack', '', batch['journal_id'], str(batch['to'] + 1), check=False)
|
||||
self.assertNotEqual(invalid.returncode, 0)
|
||||
self.acknowledge(batch)
|
||||
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
|
||||
self.assertEqual(list(self.journal.glob('part.*')), [])
|
||||
invalid = self.call('ack', '', batch['journal_id'], '0', check=False)
|
||||
self.assertNotEqual(invalid.returncode, 0)
|
||||
|
||||
def test_crash_after_chunk_rename_recovers_original_id_and_offsets(self):
|
||||
self.append('committed first\n')
|
||||
self.flush()
|
||||
first = self.export()
|
||||
previous_state = (self.journal / 'state').read_bytes()
|
||||
self.append('second chunk before state checkpoint\n')
|
||||
self.flush()
|
||||
(self.journal / 'state').write_bytes(previous_state)
|
||||
(self.journal / 'chunk.new').write_text('interrupted write')
|
||||
(self.journal / 'state.new').write_text('partial state')
|
||||
self.reboot()
|
||||
self.assertEqual(self.export()['journal_id'], first['journal_id'])
|
||||
self.acknowledge(first)
|
||||
second = self.export()
|
||||
self.assertEqual(second['from'], first['to'])
|
||||
self.assertIn(b'second chunk', base64.b64decode(second['data']))
|
||||
self.assertFalse((self.journal / 'chunk.new').exists())
|
||||
self.assertFalse((self.journal / 'state.new').exists())
|
||||
|
||||
def test_corrupt_chunk_is_quarantined_and_later_chunks_still_deliver(self):
|
||||
self.append('first\n'); self.flush()
|
||||
self.append('second\n'); self.flush()
|
||||
first = sorted(self.journal.glob('part.*'))[0]
|
||||
first.write_bytes(b'partial')
|
||||
self.reboot()
|
||||
batch = self.export()
|
||||
self.assertIn(b'second', base64.b64decode(batch['data']))
|
||||
self.assertTrue((self.journal / 'damaged.log').exists())
|
||||
self.assertIn('Повреждённый фрагмент', self.call('read').stdout)
|
||||
|
||||
def test_disk_full_falls_back_without_touching_router_configuration(self):
|
||||
self.append('retained on flash\n'); self.flush()
|
||||
self.append('buffered before ENOSPC\n')
|
||||
self.call('flush', '', 'force', TEST_FREE_KB='100', check=False)
|
||||
self.append('after disk full\n')
|
||||
self.assertEqual(self.status()['persistent'], '0')
|
||||
self.assertIn('after disk full', self.call('read').stdout)
|
||||
self.assertTrue(list(self.journal.glob('part.*')))
|
||||
|
||||
def test_confirmed_ack_can_replay_after_crash_without_changing_cursor_identity(self):
|
||||
self.append('duplicate safe\n'); self.flush()
|
||||
batch = self.export()
|
||||
self.acknowledge(batch)
|
||||
self.assertEqual(self.export()['data'], '')
|
||||
self.reboot() # ack not yet checkpointed: safe replay, Hub suppresses it.
|
||||
self.assertEqual(self.export(), batch)
|
||||
|
||||
def test_killed_flusher_reconciles_inflight_without_duplicate_data(self):
|
||||
self.append('already renamed\n')
|
||||
self.flush()
|
||||
batch = self.export()
|
||||
part = next(self.journal.glob('part.*'))
|
||||
fields = part.name.rsplit('.', 3)
|
||||
(self.run_dir / 'pending').write_bytes(base64.b64decode(batch['data']))
|
||||
(self.run_dir / 'inflight').write_text(f'{batch["journal_id"]} 0 {batch["to"]} {fields[-1]}\n')
|
||||
self.append('after killed writer\n')
|
||||
text = self.call('read').stdout
|
||||
self.assertEqual(text.count('already renamed'), 1)
|
||||
self.assertIn('after killed writer', text)
|
||||
self.assertFalse((self.run_dir / 'inflight').exists())
|
||||
|
||||
def test_corrupt_state_recovers_committed_generation_and_stale_temporaries(self):
|
||||
self.append('state recovery\n'); self.flush()
|
||||
batch = self.export()
|
||||
(self.journal / 'state').write_text('partial garbage')
|
||||
self.reboot()
|
||||
self.assertEqual(self.export(), batch)
|
||||
(self.run_dir / 'export.12345').write_text('orphan')
|
||||
self.export()
|
||||
self.assertFalse(list(self.run_dir.glob('export.*')))
|
||||
|
||||
def test_parallel_writers_and_stale_lock_recovery(self):
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
|
||||
list(pool.map(lambda i: self.append(f'concurrent {i}\n'), range(6)))
|
||||
self.assertEqual(len(self.call('read').stdout.splitlines()), 6)
|
||||
(self.run_dir / 'lock').mkdir()
|
||||
(self.run_dir / 'lock/pid').write_text('99999999')
|
||||
self.append('recovered lock\n')
|
||||
self.assertIn('recovered lock', self.call('read').stdout)
|
||||
self.assertFalse((self.run_dir / 'lock').exists())
|
||||
|
||||
def test_stale_lock_reaper_itself_recovers_after_process_crash(self):
|
||||
self.call('check')
|
||||
lock = self.run_dir / 'lock'; lock.mkdir()
|
||||
(lock / 'pid').write_text('99999999')
|
||||
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
|
||||
self.append('after killed reaper\n')
|
||||
self.assertIn('after killed reaper', self.call('read').stdout)
|
||||
self.assertFalse(lock.exists())
|
||||
|
||||
def test_collector_failure_reaps_children_and_removes_fifo(self):
|
||||
# Actual FIFO integration is tested on POSIX; Windows BusyBox has no
|
||||
# FIFO support. Exercise its traps with mocked process/FIFO operations.
|
||||
mocks = r'''
|
||||
mkfifo() { : >"$1"; }
|
||||
logread() { return 0; }
|
||||
sleep() { return 0; }
|
||||
kill() {
|
||||
printf '%s\n' "$*" >>"$RUN_DIR/kills"
|
||||
[ "$1" != -0 ]
|
||||
}
|
||||
wait() { return 0; }
|
||||
'''
|
||||
source = self.source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1)
|
||||
self.helper.write_text(source, encoding='utf-8', newline='\n')
|
||||
result = self.call('collect', check=False)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(list(self.run_dir.glob('stream.*')))
|
||||
self.assertFalse((self.run_dir / 'collector-lock').exists())
|
||||
lines = (self.run_dir / 'kills').read_text().splitlines()
|
||||
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 2)
|
||||
|
||||
def test_legacy_migration_preserves_identity_ack_and_only_undelivered_bytes(self):
|
||||
legacy = self.root / 'legacy'
|
||||
legacy.mkdir()
|
||||
(legacy / 'current.log').write_bytes(b'legacy current\n')
|
||||
(legacy / 'previous.log').write_bytes(b'legacy previous\n')
|
||||
raw = b'legacy previous\nlegacy current\n'
|
||||
(legacy / 'total').write_text(str(len(raw)))
|
||||
(legacy / 'journal-id').write_text('legacy-journal')
|
||||
(legacy / 'sync-ack').write_text('legacy-journal 16')
|
||||
(legacy / 'stream.dead').write_text('must stay transient')
|
||||
self.call('migrate')
|
||||
self.assertEqual(self.call('read').stdout, raw.decode())
|
||||
self.assertFalse((self.journal / 'stream.dead').exists())
|
||||
self.assertFalse((legacy / 'current.log').exists())
|
||||
self.assertFalse((legacy / 'total').exists())
|
||||
self.assertTrue((legacy / 'stream.dead').exists())
|
||||
batch = self.export()
|
||||
self.assertEqual(batch['journal_id'], 'legacy-journal')
|
||||
self.assertEqual(batch['from'], 16)
|
||||
self.assertEqual(base64.b64decode(batch['data']), b'legacy current\n')
|
||||
|
||||
def test_unavailable_flash_legacy_migration_does_not_keep_large_ram_archive(self):
|
||||
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n')
|
||||
legacy = self.root / 'legacy'; legacy.mkdir()
|
||||
(legacy / 'current.log').write_text('old data\n' * 10000)
|
||||
self.call('migrate')
|
||||
self.assertFalse((legacy / 'current.log').exists())
|
||||
self.assertEqual(self.status()['persistent'], '0')
|
||||
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
|
||||
self.assertIn('ограниченный хвост', self.call('read').stdout)
|
||||
|
||||
def test_interrupted_legacy_migration_finishes_after_old_ram_is_gone(self):
|
||||
self.append('staged persistent data\n'); self.flush()
|
||||
batch = self.export()
|
||||
staging = self.journal / 'migrating'; staging.mkdir()
|
||||
for part in self.journal.glob('part.*'):
|
||||
part.rename(staging / part.name)
|
||||
shutil.copyfile(self.journal / 'state', staging / 'state')
|
||||
(self.journal / 'state').write_text('interrupted state replacement')
|
||||
self.reboot()
|
||||
self.assertEqual(self.export(), batch)
|
||||
self.assertFalse(staging.exists())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Install/rollback journal files against a sandbox, never a real router."""
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
|
||||
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
|
||||
|
||||
|
||||
@unittest.skipUnless(SH, 'POSIX shell required')
|
||||
class PersistentLogsDeploymentTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(prefix='persistent-log-deploy-')
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.targets = {
|
||||
'cgi': '/www/cgi-bin/owrt-remote', 'agent': '/usr/sbin/owrt-remote',
|
||||
'init': '/etc/init.d/owrt-remote', 'logs.sh': '/usr/lib/owrt-remote/logs.sh',
|
||||
'logs-init': '/etc/init.d/owrt-remote-logs',
|
||||
'luci': '/www/luci-static/resources/view/owrt_remote.js',
|
||||
}
|
||||
self.initial = {}
|
||||
for item in ('cgi', 'agent', 'logs.sh', 'logs-init', 'luci'):
|
||||
path = self.root / self.targets[item].lstrip('/')
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text('#!/bin/sh\necho previous-' + item + '\n', encoding='utf-8', newline='\n')
|
||||
self.initial[item] = path.read_bytes()
|
||||
self.backup = self.root / 'backup'
|
||||
source = (ROOT / 'vps/deploy-persistent-logs.ps1').read_text(encoding='utf-8')
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
# Absolute paths are replaced only in this disposable test copy.
|
||||
mappings = {target: (self.root / target.lstrip('/')).as_posix() for target in self.targets.values()}
|
||||
mappings['/usr/lib/owrt-remote'] = (self.root / 'usr/lib/owrt-remote').as_posix()
|
||||
mappings['/www/luci-static/resources/view'] = (self.root / 'www/luci-static/resources/view').as_posix()
|
||||
mappings['/tmp/luci-indexcache'] = (self.root / 'luci-indexcache').as_posix()
|
||||
mappings['/tmp/luci-modulecache'] = (self.root / 'luci-modulecache').as_posix()
|
||||
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
|
||||
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
|
||||
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
|
||||
mappings['/root/owrt-remote-backup-before-persistent-logs-v117'] = self.backup.as_posix()
|
||||
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
|
||||
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
|
||||
# Existing sandbox service scripts only print markers. Mock readiness.
|
||||
mocks = '''
|
||||
logread() { return 0; }
|
||||
ubus() { printf '{"running": true}\\n'; }
|
||||
mkfifo() { return 0; }
|
||||
jsonfilter() { return 0; }
|
||||
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
|
||||
'''
|
||||
self.script = self.script.replace('for item in cgi agent logs.sh logs-init;', mocks + '\nfor item in cgi agent logs.sh logs-init;', 1)
|
||||
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
|
||||
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
|
||||
if os.name == 'nt':
|
||||
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
|
||||
|
||||
def deploy(self, action, bad=False, fail_verify=False):
|
||||
stage = self.root / 'stage'
|
||||
stage.mkdir()
|
||||
candidates = {
|
||||
'cgi': 'files/www/cgi-bin/owrt-remote', 'agent': 'files/usr/sbin/owrt-remote',
|
||||
'init': 'files/etc/init.d/owrt-remote', 'logs.sh': 'files/usr/lib/owrt-remote/logs.sh',
|
||||
'logs-init': 'files/etc/init.d/owrt-remote-logs',
|
||||
'luci': 'files/www/luci-static/resources/view/owrt_remote.js',
|
||||
}
|
||||
if action == 'Install':
|
||||
for item, source in candidates.items():
|
||||
text = (ROOT / source).read_text(encoding='utf-8')
|
||||
if item == 'logs.sh':
|
||||
text = text.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + shlex.quote((self.root / 'legacy').as_posix()))
|
||||
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
|
||||
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
|
||||
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
|
||||
if item == 'logs-init':
|
||||
text = '#!/bin/sh\necho logs-service-$1\n'
|
||||
if bad and item == 'cgi':
|
||||
text = '#!/bin/sh\nif true; then\n'
|
||||
stage.joinpath(item).write_text(text, encoding='utf-8', newline='\n')
|
||||
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(stage.as_posix()))
|
||||
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
|
||||
if fail_verify:
|
||||
script = script.replace('cmp "$stage/$item" "$path"', 'false')
|
||||
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
|
||||
env=self.env, timeout=30)
|
||||
self.assertFalse(stage.exists())
|
||||
self.assertFalse(list(self.root.rglob('*.new-sandbox')))
|
||||
return result
|
||||
|
||||
def assert_initial(self):
|
||||
for item, data in self.initial.items():
|
||||
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
|
||||
|
||||
def test_install_then_rollback_retains_first_backup(self):
|
||||
for _ in range(2):
|
||||
result = self.deploy('Install')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('PERSISTENT_LOGS_ROUTER_Install_OK', result.stdout)
|
||||
for item in ('cgi', 'luci'):
|
||||
installed = self.root / self.targets[item].lstrip('/')
|
||||
self.assertIn('v117', installed.read_text(encoding='utf-8'))
|
||||
for item, data in self.initial.items():
|
||||
self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
|
||||
result = self.deploy('Rollback')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('PERSISTENT_LOGS_ROUTER_Rollback_OK', result.stdout)
|
||||
self.assert_initial()
|
||||
|
||||
def test_missing_backup_changes_nothing(self):
|
||||
result = self.deploy('Rollback')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('No complete backup:', result.stderr)
|
||||
self.assert_initial()
|
||||
|
||||
def test_invalid_candidate_rejected_before_backup(self):
|
||||
result = self.deploy('Install', bad=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(self.backup.exists())
|
||||
self.assert_initial()
|
||||
|
||||
def test_failed_verification_restores_all_previous_files(self):
|
||||
result = self.deploy('Install', fail_verify=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('Restoring files from this attempt.', result.stderr)
|
||||
self.assert_initial()
|
||||
|
||||
def test_rollback_restores_existing_log_service_and_helper(self):
|
||||
# This fix is normally applied on top of v117's already installed journal.
|
||||
for item in ('logs.sh', 'logs-init'):
|
||||
target = self.root / self.targets[item].lstrip('/')
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text('#!/bin/sh\necho previous-' + item + '-$1\n', encoding='utf-8', newline='\n')
|
||||
self.initial[item] = target.read_bytes()
|
||||
result = self.deploy('Install')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
result = self.deploy('Rollback')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
for item, data in self.initial.items():
|
||||
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
|
||||
self.assertIn('previous-logs-init-start', result.stdout)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Validate VPS install, rollback and automatic recovery in a disposable tree."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
|
||||
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
|
||||
|
||||
|
||||
@unittest.skipUnless(SH, 'POSIX shell required')
|
||||
class VpsLogDeployTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(prefix='vps-log-deploy-')
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.target = self.root / 'hub.py'
|
||||
self.previous = b'print("previous Hub")\n'
|
||||
self.target.write_bytes(self.previous)
|
||||
self.backup = self.root / 'hub.py.bak-before-persistent-logs-sync-v117'
|
||||
source = (ROOT / 'vps/deploy-persistent-logs-vps.ps1').read_text(encoding='utf-8')
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
|
||||
self.script = self.script.replace('python_bin=/opt/owrt-remote/venv/bin/python', 'python_bin=python')
|
||||
self.script = self.script.replace('[ -x "$python_bin" ] || python_bin=python3', '')
|
||||
mocks = '''
|
||||
systemctl() { [ "${TEST_RESTART_FAIL:-0}" != 1 ] || [ "$1" != restart ]; }
|
||||
journalctl() { echo sandbox-journal; }
|
||||
curl() { [ "${TEST_HEALTH_FAIL:-0}" != 1 ] && printf '{"ok":true}'; }
|
||||
sleep() { return 0; }
|
||||
'''
|
||||
self.script = self.script.replace('test -s "$target"', mocks + '\ntest -s "$target"', 1)
|
||||
# Exercise the actual mount/path validation with an isolated process proc
|
||||
# fixture. Storage classification is covered independently on real paths.
|
||||
original = self.root / 'running.py'
|
||||
original.write_text('print("running")')
|
||||
proc = self.root / 'proc' / '123'
|
||||
proc.mkdir(parents=True)
|
||||
(proc / 'environ').write_bytes(('OWRT_REMOTE_STATE_DIR=' + self.root.as_posix()).encode() + b'\0')
|
||||
(proc / 'cmdline').write_bytes(b'python\0' + original.as_posix().encode() + b'\0')
|
||||
database = self.root / 'hub.db'
|
||||
database.write_bytes(b'existing archive must not change')
|
||||
self.database = database
|
||||
self.script = self.script.replace("Path('/proc/' + pid", "Path('" + (self.root / 'proc').as_posix() + "/' + pid")
|
||||
self.script = self.script.replace("storage = hub.router_log_storage(db_path)",
|
||||
"storage = hub.router_log_storage(db_path)\n" +
|
||||
"storage.update(persistent=os.environ.get('TEST_RAM_DISK') != '1', writable=True)")
|
||||
self.script = self.script.replace('systemctl() { ',
|
||||
'systemctl() { if [ "$1" = show ]; then echo 123; return 0; fi; ')
|
||||
self.env = dict(os.environ)
|
||||
if os.name == 'nt':
|
||||
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
|
||||
|
||||
def deploy(self, action, invalid=False, **flags):
|
||||
stage = self.root / 'stage'
|
||||
stage.mkdir()
|
||||
if action == 'Install':
|
||||
data = 'if invalid:\n' if invalid else (ROOT / 'vps/owrt-remote-hub.py').read_text(encoding='utf-8')
|
||||
stage.joinpath('hub.py').write_text(data, encoding='utf-8', newline='\n')
|
||||
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage="' + stage.as_posix() + '"')
|
||||
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
|
||||
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
|
||||
env={**self.env, **flags}, timeout=25)
|
||||
self.assertFalse(stage.exists())
|
||||
self.assertFalse(list(self.root.glob('*.new-sandbox')))
|
||||
self.assertEqual(self.database.read_bytes(), b'existing archive must not change')
|
||||
return result
|
||||
|
||||
def test_install_and_rollback_preserve_original_backup(self):
|
||||
for _ in range(2):
|
||||
result = self.deploy('Install')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn('PERSISTENT_LOGS_VPS_V117_Install_OK', result.stdout)
|
||||
self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
|
||||
self.assertEqual(self.backup.read_bytes(), self.previous)
|
||||
result = self.deploy('Rollback')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(self.target.read_bytes(), self.previous)
|
||||
|
||||
def test_failed_health_restores_file_from_this_attempt(self):
|
||||
self.assertEqual(self.deploy('Install').returncode, 0)
|
||||
installed = self.target.read_bytes()
|
||||
result = self.deploy('Rollback', TEST_HEALTH_FAIL='1')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('Restoring the previous Hub file', result.stderr)
|
||||
self.assertEqual(self.target.read_bytes(), installed)
|
||||
|
||||
def test_ram_database_preflight_does_not_install_or_modify_archive(self):
|
||||
result = self.deploy('Install', TEST_RAM_DISK='1')
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('not confirmed on writable persistent disk', result.stderr)
|
||||
self.assertEqual(self.target.read_bytes(), self.previous)
|
||||
self.assertFalse(self.backup.exists())
|
||||
|
||||
def test_invalid_candidate_and_missing_backup_change_nothing(self):
|
||||
self.assertNotEqual(self.deploy('Install', invalid=True).returncode, 0)
|
||||
self.assertFalse(self.backup.exists())
|
||||
self.assertNotEqual(self.deploy('Rollback').returncode, 0)
|
||||
self.assertEqual(self.target.read_bytes(), self.previous)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -131,7 +131,7 @@ class HubFixture:
|
||||
assert "groups" in json.loads(self.request("/api/router-groups"))
|
||||
assert "notifications" in json.loads(self.request("/api/notifications"))
|
||||
assert self.request("/access/test-router/") == b"router proxy fixture"
|
||||
assert b"v116" in self.request("/")
|
||||
assert b"v117" in self.request("/")
|
||||
self.app.snapshot_router_states()
|
||||
|
||||
def __exit__(self, *args):
|
||||
|
||||
@@ -113,6 +113,81 @@ class ArchiveTests(unittest.TestCase):
|
||||
self.assertIn('новый журнал', self.snapshot()['text'])
|
||||
self.assertIn('first', self.snapshot()['text'])
|
||||
|
||||
def test_empty_status_cannot_confirm_missing_records_or_invent_rotation_gap(self):
|
||||
reply = self.ingest(batch(b'', start=298))
|
||||
self.assertEqual(reply['offset'], 0)
|
||||
self.assertEqual(self.snapshot()['text'], '')
|
||||
raw = b'[OK] retained records arrive after status\n'
|
||||
self.ingest(batch(raw))
|
||||
reply = self.ingest(batch(b'', start=1000))
|
||||
self.assertEqual(reply['offset'], len(raw))
|
||||
self.assertEqual(self.snapshot()['text'].encode(), raw)
|
||||
|
||||
def test_actual_gap_with_data_is_reported_once_but_retries_keep_data(self):
|
||||
value = batch(b'[OK] surviving records\n', start=50)
|
||||
self.ingest(value)
|
||||
self.ingest(value)
|
||||
text = self.snapshot()['text']
|
||||
self.assertEqual(text.count('не доставлена'), 1)
|
||||
self.assertEqual(text.count('surviving records'), 1)
|
||||
|
||||
def test_empty_new_generation_cannot_erase_current_journal_cursor(self):
|
||||
raw = b'[OK] previous generation\n'
|
||||
self.ingest(batch(raw))
|
||||
self.ingest(batch(b'', start=2000, journal='new-journal'))
|
||||
self.ingest(batch(raw))
|
||||
self.assertEqual(self.snapshot()['text'].encode(), raw)
|
||||
|
||||
def test_recovery_generation_replays_previously_falsely_confirmed_chunks(self):
|
||||
# Reproduce the cursor left by the former empty-batch ingestion bug.
|
||||
with self.app.conn() as conn:
|
||||
conn.execute("insert into router_logs(router_id,journal_id,cursor) values ('main','broken',298)")
|
||||
self.ingest(batch(b'[OK] retained restart event\n', journal='recovered-old'))
|
||||
self.assertIn('retained restart event', self.snapshot()['text'])
|
||||
|
||||
def test_real_router_spool_recovers_bad_digest_and_delivers_after_restart(self):
|
||||
from test_persistent_logs import PersistentLogsTests
|
||||
client = PersistentLogsTests()
|
||||
client.setUp()
|
||||
self.addCleanup(client.doCleanups)
|
||||
client.append('Панель: действие=stop; начало\nСервис: остановка туннеля\n')
|
||||
client.flush()
|
||||
original = client.export()
|
||||
client.acknowledge(original); client.flush()
|
||||
with self.app.conn() as conn:
|
||||
conn.execute('insert into router_logs(router_id,journal_id,cursor) values (?,?,?)',
|
||||
('main', original['journal_id'], original['to']))
|
||||
part = next(client.journal.glob('part.*'))
|
||||
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
|
||||
(client.run_dir / 'recovered-v2').unlink()
|
||||
repaired = client.export()
|
||||
reply = self.ingest(repaired, {'enabled': False, 'xray': 'stopped'})
|
||||
client.call('ack', '', reply['journal_id'], str(reply['offset']))
|
||||
# A lost acknowledgment may replay, but the archive must stay unchanged.
|
||||
self.ingest(repaired)
|
||||
self.assertEqual(self.snapshot()['text'].count('Сервис: остановка туннеля'), 1)
|
||||
client.append('Сервис: запуск\n'); client.flush()
|
||||
second = client.export()
|
||||
reply = self.ingest(second, {'enabled': True, 'xray': 'running'})
|
||||
client.call('ack', '', reply['journal_id'], str(reply['offset']))
|
||||
text = self.snapshot()['text']
|
||||
self.assertIn('Сервис: запуск', text)
|
||||
self.assertNotIn('не доставлена до ротации', text)
|
||||
self.assertEqual(client.export()['data'], '')
|
||||
|
||||
def test_disk_mount_diagnostic_resolves_deepest_mount_and_escaped_paths(self):
|
||||
path = self.app.db_path.resolve()
|
||||
mount = str(path.parent).replace(' ', r'\040')
|
||||
root = str(Path(path.anchor)).replace(' ', r'\040')
|
||||
mountinfo = f'1 0 8:1 / {root} rw - ext4 /dev/vda1 rw\n'
|
||||
mountinfo += f'2 1 0:1 / {mount} rw - tmpfs tmpfs rw\n'
|
||||
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo)):
|
||||
result = hub.router_log_storage(path)
|
||||
self.assertIs(result['persistent'], False)
|
||||
self.assertEqual(result['filesystem'], 'tmpfs')
|
||||
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo.splitlines()[0])):
|
||||
self.assertIs(hub.router_log_storage(path)['persistent'], True)
|
||||
|
||||
def test_offline_clear_does_not_restore_old_logs_on_reconnection(self):
|
||||
old = b'[ERROR] old\n'
|
||||
self.ingest(batch(old))
|
||||
|
||||
@@ -35,6 +35,7 @@ const server = http.createServer((req, res) => {
|
||||
reads++;
|
||||
res.setHeader('Content-Type', 'application/json');
|
||||
return res.end(JSON.stringify({ok:true, online:false, text, bytes:Buffer.byteLength(text), last_sync:1791954000,
|
||||
storage:{path:'/var/lib/owrt-remote/hub.db',filesystem:'ext4',persistent:true},
|
||||
diagnosis:{level:'warn',text:'Remote выключен на роутере. Туннель и Heartbeat остановлены.',at:1791954000}}));
|
||||
}
|
||||
if (url.pathname === '/api/router/main/logs/clear') {
|
||||
@@ -95,6 +96,8 @@ const server = http.createServer((req, res) => {
|
||||
text=initialText();const before=reads;
|
||||
await page.goto(`http://127.0.0.1:${server.address().port}/router/main/logs`);
|
||||
await page.waitForFunction(()=>document.querySelector('#logsText').textContent.includes('Event 119'));
|
||||
assert((await page.locator('#logsStorage').textContent()).includes('Архив VPS: постоянный диск.'));
|
||||
assert((await page.locator('#logsStorage').textContent()).includes('/var/lib/owrt-remote/hub.db'));
|
||||
assert.deepEqual(await page.locator('body').evaluate(el=>{
|
||||
const s=getComputedStyle(el);return {image:s.backgroundImage,color:s.backgroundColor,attachment:s.backgroundAttachment};
|
||||
}),terminalBackground,'log page uses the VPS terminal background');
|
||||
|
||||
@@ -23,6 +23,8 @@ class RouterLogTransportTests(unittest.TestCase):
|
||||
self.env = {**self.journal.env, 'OWRT_REMOTE_WAN_HELPER': (ROOT / 'files/usr/lib/owrt-remote/wan.sh').as_posix(),
|
||||
'TEST_ROOT': self.journal.root.as_posix(), 'TEST_PYTHON': Path(sys.executable).as_posix()}
|
||||
self.mocks = r'''
|
||||
# Windows has no RLIMIT_FSIZE; production ash enforces the actual file limit.
|
||||
ulimit() { return 0; }
|
||||
uci_get() {
|
||||
case "$1" in
|
||||
enabled) printf '%s' "${TEST_ENABLED:-1}" ;; hub_token) printf test-token ;; router_id) printf main ;;
|
||||
@@ -60,6 +62,7 @@ jsonfilter() {
|
||||
'''
|
||||
|
||||
def sync(self, **flags):
|
||||
self.journal.helper_call('flush', '', 'force')
|
||||
script = self.source + '\n' + self.mocks + '\nsync_global_logs\n'
|
||||
return subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
|
||||
env={**self.env, **flags}, timeout=25)
|
||||
@@ -80,14 +83,34 @@ jsonfilter() {
|
||||
self.assertEqual('direct' in requests, direct == '1')
|
||||
self.assertEqual('prepare' in requests, direct == '1')
|
||||
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
|
||||
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
|
||||
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
|
||||
|
||||
def test_failed_delivery_preserves_unsent_records_and_removes_payload_files(self):
|
||||
before = json.loads(self.journal.helper_call('export'))
|
||||
result = self.sync(TEST_HTTP_FAIL='1')
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
self.assertEqual(json.loads(self.journal.helper_call('export')), before)
|
||||
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
|
||||
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
|
||||
|
||||
def test_agent_update_delivers_with_older_helper_without_its_own_runtime_directory(self):
|
||||
# The pre-persistent helper creates only /tmp/owrt-remote-logs. A WAN
|
||||
# agent update must not depend on that helper creating the new runtime.
|
||||
helper = self.journal.root / 'old-helper'
|
||||
record = base64.b64encode(b'[OK] older helper record\n').decode()
|
||||
batch = json.dumps({'journal_id': 'older', 'from': 0, 'to': 25, 'data': record})
|
||||
helper.write_text('#!/bin/sh\ncase "$1" in\ncheck) exit 0 ;;\nexport) printf \'%s\\n\' \'' + batch +
|
||||
'\' ;;\nack) printf \'%s\\n\' "$2 $3" >"$TEST_ROOT/older-ack" ;;\nesac\n',
|
||||
encoding='utf-8', newline='\n')
|
||||
runtime = self.journal.root / 'new-runtime'
|
||||
result = subprocess.run([SH], input=self.source + '\n' + self.mocks + '\nsync_global_logs\n',
|
||||
encoding='utf-8', capture_output=True,
|
||||
env={**self.env, 'OWRT_REMOTE_LOG_HELPER': helper.as_posix(),
|
||||
'OWRT_REMOTE_LOG_RUNTIME': runtime.as_posix()}, timeout=25)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertTrue(runtime.is_dir())
|
||||
self.assertTrue((self.journal.root / 'older-ack').is_file())
|
||||
self.assertFalse((runtime / 'sync-lock').exists())
|
||||
self.assertFalse(list(runtime.glob('sync.*')))
|
||||
|
||||
def test_disabled_remote_still_delivers_stop_reason_and_state_without_restart(self):
|
||||
self.journal.helper_call('append', 'Сервис: остановка туннеля\n')
|
||||
@@ -97,14 +120,14 @@ jsonfilter() {
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
payload = json.loads((self.journal.root / 'delivered.json').read_text())
|
||||
self.assertEqual(payload['state'], {'enabled': False, 'xray': 'stopped'})
|
||||
self.assertFalse(list(self.journal.log_dir.glob('sync.*')))
|
||||
self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
|
||||
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
|
||||
|
||||
def test_overlapping_delivery_does_not_start_second_sender(self):
|
||||
lock = self.journal.log_dir / 'sync-lock'
|
||||
lock = self.journal.run_dir / 'sync-lock'
|
||||
lock.mkdir()
|
||||
# The shell's own PID is alive for the duration of sync_global_logs.
|
||||
script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_DIR/sync-lock/pid"\nsync_global_logs\n'
|
||||
script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_RUNTIME/sync-lock/pid"\nsync_global_logs\n'
|
||||
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True, env=self.env, timeout=20)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse((self.journal.root / 'requests').exists())
|
||||
@@ -113,15 +136,25 @@ jsonfilter() {
|
||||
def test_wan_failure_releases_delivery_lock_for_next_attempt(self):
|
||||
result = self.sync(TEST_DIRECT='1', TEST_WAN_FAIL='1', TEST_ENABLED='0')
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
self.assertFalse((self.journal.log_dir / 'sync-lock').exists())
|
||||
self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
|
||||
self.assertNotEqual(json.loads(self.journal.helper_call('export'))['data'], '')
|
||||
self.assertEqual(self.sync(TEST_DIRECT='1', TEST_ENABLED='0').returncode, 0)
|
||||
|
||||
def test_abandoned_empty_delivery_lock_is_recovered(self):
|
||||
(self.journal.log_dir / 'sync-lock').mkdir()
|
||||
(self.journal.run_dir / 'sync-lock').mkdir()
|
||||
result = self.sync(TEST_ENABLED='0')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse((self.journal.log_dir / 'sync-lock').exists())
|
||||
self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
|
||||
self.assertTrue((self.journal.root / 'delivered.json').exists())
|
||||
self.assertFalse(list(self.journal.run_dir.glob('http/owrt-remote-http.*')))
|
||||
|
||||
def test_abandoned_delivery_lock_reaper_is_cleaned_for_next_attempt(self):
|
||||
lock = self.journal.run_dir / 'sync-lock'
|
||||
lock.mkdir(); (lock / 'pid').write_text('99999999')
|
||||
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
|
||||
self.assertEqual(self.sync().returncode, 0)
|
||||
self.assertEqual(self.sync().returncode, 0)
|
||||
self.assertFalse(lock.exists())
|
||||
self.assertTrue((self.journal.root / 'delivered.json').exists())
|
||||
|
||||
def test_panel_stop_sends_completed_result_without_waiting_for_restart(self):
|
||||
|
||||
@@ -35,7 +35,7 @@ class WanDnsDeploymentTests(unittest.TestCase):
|
||||
source = (ROOT / "vps/deploy-wan-dns-fix.ps1").read_text(encoding="utf-8")
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace("target=/usr/lib/owrt-remote/wan.sh", "target=" + shlex.quote(self.target.as_posix()))
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v116",
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v117",
|
||||
"backup_dir=" + shlex.quote(self.backup.as_posix()))
|
||||
|
||||
def deploy(self, action="Install", bad=False, fail_verify=False, fail_signal=False):
|
||||
|
||||
@@ -129,7 +129,7 @@ class WanFailoverTests(unittest.TestCase):
|
||||
self.prepare()
|
||||
config = self.directory / "xray.json"
|
||||
config.write_text(json.dumps({"outbounds": [{"tag": "vps-interconn", "streamSettings": {"sockopt": {"interface": "eth0"}}}]}))
|
||||
self.config(wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix())
|
||||
self.config(enabled="1", wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix())
|
||||
result = self.shell('''
|
||||
xray_running() { return 0; }
|
||||
wan_reconnect_xray() { printf '%s -> %s\n' "$1" "$2" >>"$TEST_ROOT/reconnect"; }
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
"""WAN outage/recovery using the real agent shell and a simulated procd."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import unittest
|
||||
|
||||
import test_host_policy as base
|
||||
from test_wan_failover import UPLINK_MOCKS, JSONFILTER
|
||||
|
||||
|
||||
@unittest.skipUnless(base.SH, "POSIX shell required")
|
||||
class WanRecoveryTests(unittest.TestCase):
|
||||
setUp = base.RouterShellTests.setUp
|
||||
config = base.RouterShellTests.config
|
||||
|
||||
def shell(self, commands):
|
||||
shell = os.environ.get("OWRT_RECOVERY_TEST_SHELL", base.SH)
|
||||
args = [shell, "ash"] if "busybox" in shell.lower() else [shell]
|
||||
return subprocess.run(args, input=self.agent + "\n" + base.MOCKS + "\n" + UPLINK_MOCKS + "\n" + commands,
|
||||
encoding="utf-8", capture_output=True, cwd=base.ROOT, env=self.env, timeout=40)
|
||||
|
||||
def prepare(self, enabled="1"):
|
||||
self.config(enabled=enabled, wan_direct="1", wan_interface="wan", vps_host="192.0.2.10",
|
||||
hub_url="https://hub.example", hub_token="secret-test-token")
|
||||
self.directory.joinpath("jsonfilter.py").write_text(JSONFILTER, encoding="utf-8")
|
||||
self.directory.joinpath("service.json").write_text(json.dumps({"owrt-remote": {"instances": {
|
||||
"heartbeat": {"running": True, "pid": 123}, "xray": {"running": False}
|
||||
}}}), encoding="utf-8")
|
||||
|
||||
PROCD = r'''
|
||||
xray_running() { return 1; }
|
||||
ubus() {
|
||||
case "$1 $2 $3" in
|
||||
'call service list') cat "$TEST_ROOT/service.json" ;;
|
||||
'call service add')
|
||||
printf '%s\n' "$4" >>"$TEST_ROOT/add-calls"
|
||||
[ ! -e "$TEST_ROOT/add-fail" ] ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
jsonfilter() {
|
||||
"$TEST_PYTHON" -c 'import json,sys
|
||||
state=json.load(sys.stdin)
|
||||
instance="heartbeat" if ".heartbeat." in sys.argv[1] else "xray"
|
||||
value=state.get("owrt-remote",{}).get("instances",{}).get(instance,{}).get("running","")
|
||||
print(str(value).lower())' "$2"
|
||||
}
|
||||
wan_select_interface() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf wan; }
|
||||
wan_device() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf eth0; }
|
||||
'''
|
||||
|
||||
def test_online_heartbeat_after_outage_recovers_dead_tunnel_on_same_uplink_once(self):
|
||||
self.prepare()
|
||||
result = self.shell(self.PROCD + r'''
|
||||
touch "$TEST_ROOT/wan-down"
|
||||
wan_reconcile_xray || true
|
||||
test ! -e "$TEST_ROOT/add-calls" || exit 1
|
||||
rm "$TEST_ROOT/wan-down"
|
||||
wan_reconcile_xray || exit
|
||||
wan_reconcile_xray
|
||||
''')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
requests = self.directory.joinpath("add-calls").read_text().splitlines()
|
||||
self.assertEqual(len(requests), 1)
|
||||
request = json.loads(requests[0])
|
||||
self.assertEqual(set(request["instances"]), {"xray"})
|
||||
self.assertEqual(request["instances"]["xray"]["command"], ["/usr/sbin/owrt-remote", "run"])
|
||||
self.assertEqual(request["instances"]["xray"]["respawn"], ["3600", "5", "5"])
|
||||
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
|
||||
self.assertNotIn("secret-test-token", result.stderr)
|
||||
|
||||
def test_initial_start_waiting_for_wan_is_left_to_procd(self):
|
||||
self.prepare()
|
||||
state = self.directory / "service.json"
|
||||
data = json.loads(state.read_text())
|
||||
data["owrt-remote"]["instances"]["xray"]["running"] = True
|
||||
state.write_text(json.dumps(data))
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("add-calls").exists())
|
||||
|
||||
def test_successful_heartbeat_recovers_dead_xray_and_still_returns_hub_response(self):
|
||||
self.prepare()
|
||||
result = self.shell(self.PROCD + r'''
|
||||
wan_prepare() { return 0; }
|
||||
heartbeat_json() { printf '{}'; }
|
||||
heartbeat_candidate_endpoints() { printf 'https://hub.example/api/heartbeat'; }
|
||||
post_json() { printf '{"ok":true}'; }
|
||||
heartbeat_apply_hub_update() { return 0; }
|
||||
send_heartbeat
|
||||
''')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(json.loads(result.stdout), {"ok": True})
|
||||
request = json.loads(self.directory.joinpath("add-calls").read_text())
|
||||
self.assertEqual(set(request["instances"]), {"xray"})
|
||||
self.assertIn("Heartbeat: OK", result.stderr)
|
||||
|
||||
def test_healthy_xray_is_not_restarted_when_wan_returns(self):
|
||||
self.prepare()
|
||||
result = self.shell(self.PROCD + "xray_running() { return 0; }; wan_reconcile_xray")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("add-calls").exists())
|
||||
|
||||
def test_panel_stop_and_external_service_stop_are_respected(self):
|
||||
self.prepare(enabled="0")
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("add-calls").exists())
|
||||
self.prepare()
|
||||
self.directory.joinpath("service.json").write_text('{}')
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("add-calls").exists())
|
||||
|
||||
def test_parallel_requests_and_stale_lock_produce_one_recovery(self):
|
||||
self.prepare()
|
||||
lock = self.directory / "owrt-remote-xray-recovery"
|
||||
lock.mkdir()
|
||||
lock.joinpath("pid").write_text("2147483000")
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray & wan_reconcile_xray & wait")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
|
||||
self.assertFalse(lock.exists())
|
||||
|
||||
def test_ubus_failure_reports_error_cleans_lock_and_honors_backoff(self):
|
||||
self.prepare()
|
||||
self.directory.joinpath("add-fail").touch()
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray; rc=$?; wan_reconcile_xray; exit $rc")
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
self.assertIn("procd", result.stderr)
|
||||
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
|
||||
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
|
||||
|
||||
def test_empty_lock_and_dead_reaper_do_not_block_later_recovery(self):
|
||||
self.prepare()
|
||||
lock = self.directory / "owrt-remote-xray-recovery"
|
||||
lock.mkdir()
|
||||
(lock / "reap").mkdir()
|
||||
(lock / "reap/pid").write_text("2147483000")
|
||||
result = self.shell(self.PROCD + "wan_reconcile_xray")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
|
||||
self.assertFalse(lock.exists())
|
||||
|
||||
def test_missing_wan_does_not_cause_repeated_service_restarts(self):
|
||||
self.prepare()
|
||||
self.directory.joinpath("wan-down").touch()
|
||||
result = self.shell(self.PROCD + r'''
|
||||
active_ssh_session() { return 1; }
|
||||
heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal
|
||||
''')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("owrt-remote-heartbeat.restart").exists())
|
||||
self.assertIn("IPv4 default route", result.stderr)
|
||||
|
||||
def test_supervised_start_waits_beyond_six_retries_then_launches_with_valid_config(self):
|
||||
self.prepare()
|
||||
self.directory.joinpath("wan-down").touch()
|
||||
executable = self.directory / "fake-xray"
|
||||
executable.write_text('#!/bin/sh\n[ "$1" != help ] || { echo run; exit; }; printf "%s\\n" "$*" >"$TEST_ROOT/xray-called"\n', newline="\n")
|
||||
executable.chmod(0o755)
|
||||
config = self.directory / "valid-config.json"
|
||||
config.write_text('{"outbounds":[]}')
|
||||
result = self.shell(r'''
|
||||
sleep() {
|
||||
local n
|
||||
n=$(cat "$TEST_ROOT/waits" 2>/dev/null || echo 0)
|
||||
n=$((n+1)); printf '%s' "$n" >"$TEST_ROOT/waits"
|
||||
[ "$n" -lt 8 ] || rm -f "$TEST_ROOT/wan-down"
|
||||
}
|
||||
ensure_xray_available_with_retry() { return 0; }
|
||||
xray_bin_path() { printf '%s' "$TEST_ROOT/fake-xray"; }
|
||||
render_client_config() { printf '%s' "$TEST_ROOT/valid-config.json"; }
|
||||
run_xray
|
||||
''')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(self.directory.joinpath("waits").read_text(), "8")
|
||||
self.assertEqual(self.directory.joinpath("xray-called").read_text().strip(), 'run -config ' + config.as_posix())
|
||||
self.assertEqual(result.stderr.count("конфигурация сохранена"), 1)
|
||||
self.assertIn("продолжается запуск", result.stderr)
|
||||
|
||||
def test_disabling_service_cancels_wait_without_rewriting_configuration(self):
|
||||
self.prepare()
|
||||
self.directory.joinpath("wan-down").touch()
|
||||
result = self.shell(r'''
|
||||
sleep() { uci set owrtremote.main.enabled=0; }
|
||||
render_client_config() { touch "$TEST_ROOT/rendered"; }
|
||||
run_xray
|
||||
''')
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("rendered").exists())
|
||||
|
||||
def test_configuration_error_with_available_wan_is_not_retried_as_wan_outage(self):
|
||||
self.prepare()
|
||||
result = self.shell(r'''
|
||||
sleep() { touch "$TEST_ROOT/slept"; }
|
||||
ensure_xray_available_with_retry() { return 0; }
|
||||
xray_bin_path() { printf /usr/bin/xray; }
|
||||
render_client_config() { printf 'invalid configuration\n' >&2; return 7; }
|
||||
run_xray
|
||||
''')
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
self.assertIn("invalid configuration", result.stderr)
|
||||
self.assertFalse(self.directory.joinpath("slept").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,122 @@
|
||||
"""Run installation and rollback in a sandbox, without SSH or services."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from test_xray_mobile_deploy import SH, GIT_BIN, ROOT
|
||||
|
||||
|
||||
@unittest.skipUnless(SH, "POSIX shell required")
|
||||
class WanRecoveryDeploymentTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(prefix="wan-recovery-deploy-")
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.targets = {"wan.sh": self.root / "wan.sh", "agent": self.root / "agent"}
|
||||
for name, path in self.targets.items():
|
||||
path.write_text("#!/bin/sh\necho previous-" + name + "\n", encoding="utf-8", newline="\n")
|
||||
self.original = {name: path.read_bytes() for name, path in self.targets.items()}
|
||||
self.backup = self.root / "backup"
|
||||
self.env = dict(os.environ)
|
||||
if os.name == "nt":
|
||||
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
|
||||
source = (ROOT / "vps/deploy-wan-recovery-fix.ps1").read_text(encoding="utf-8")
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-recovery-v117",
|
||||
"backup_dir=" + shlex.quote(self.backup.as_posix()))
|
||||
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
|
||||
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
|
||||
|
||||
def deploy(self, action, bad_item=None, fail_verify=False, fail_signal=False, stopped=False):
|
||||
stage = self.root / "stage"
|
||||
stage.mkdir()
|
||||
for item, source in {"wan.sh": "files/usr/lib/owrt-remote/wan.sh", "agent": "files/usr/sbin/owrt-remote"}.items():
|
||||
text = (ROOT / source).read_text(encoding="utf-8")
|
||||
if bad_item == item:
|
||||
text = "#!/bin/sh\nif true; then\n"
|
||||
stage.joinpath(item).write_text(text, encoding="utf-8", newline="\n")
|
||||
script = self.script.replace("stage=/tmp/__UPLOAD_NAME__", "stage=" + shlex.quote(stage.as_posix()))
|
||||
script = script.replace("__ACTION__", action).replace("__UPLOAD_NAME__", "sandbox")
|
||||
if fail_verify:
|
||||
script = script.replace('cmp "$source_dir/$item" "$path"', "false")
|
||||
mocks = r'''
|
||||
|
||||
ubus() {
|
||||
printf '%s\n' "$*" >>"$TEST_ROOT/ubus-calls"
|
||||
if [ "$3" = signal ]; then
|
||||
[ "${TEST_SIGNAL_FAIL:-0}" = 0 ] || return 1
|
||||
printf 202 >"$TEST_ROOT/pid"
|
||||
fi
|
||||
printf '{}'
|
||||
}
|
||||
jsonfilter() { cat >/dev/null; cat "$TEST_ROOT/pid"; }
|
||||
sleep() { :; }
|
||||
'''
|
||||
self.root.joinpath("pid").write_text("" if stopped else "101")
|
||||
result = subprocess.run([SH], input=mocks + script, encoding="utf-8", capture_output=True,
|
||||
env={**self.env, "TEST_ROOT": self.root.as_posix(), "TEST_SIGNAL_FAIL": "1" if fail_signal else "0"}, timeout=30)
|
||||
self.assertFalse(stage.exists())
|
||||
self.assertFalse(list(self.root.glob("*.new-sandbox")))
|
||||
return result
|
||||
|
||||
def test_install_twice_then_rollback_preserves_first_backup(self):
|
||||
for attempt in range(2):
|
||||
result = self.deploy("Install")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("WAN_RECOVERY_ROUTER_Install_OK", result.stdout)
|
||||
for item, path in self.targets.items():
|
||||
self.assertNotEqual(path.read_bytes(), self.original[item])
|
||||
self.assertEqual(self.backup.joinpath(item).read_bytes(), self.original[item])
|
||||
result = self.deploy("Rollback")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
for item, path in self.targets.items():
|
||||
self.assertEqual(path.read_bytes(), self.original[item])
|
||||
|
||||
def test_bad_candidate_and_missing_backup_leave_paired_files_unchanged(self):
|
||||
for item in ("agent", "wan.sh"):
|
||||
result = self.deploy("Install", bad_item=item)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(self.backup.exists())
|
||||
for name, path in self.targets.items():
|
||||
self.assertEqual(path.read_bytes(), self.original[name])
|
||||
result = self.deploy("Rollback")
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("No complete backup", result.stderr)
|
||||
for name, path in self.targets.items():
|
||||
self.assertEqual(path.read_bytes(), self.original[name])
|
||||
|
||||
def test_reloads_only_heartbeat_without_touching_xray(self):
|
||||
result = self.deploy("Install")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
calls = self.root.joinpath("ubus-calls").read_text()
|
||||
self.assertIn('service signal {"name":"owrt-remote","instance":"heartbeat","signal":9}', calls)
|
||||
self.assertNotIn('"instance":"xray"', calls)
|
||||
self.assertIn('Heartbeat loop reloaded', result.stdout)
|
||||
|
||||
def test_stopped_loop_stays_stopped(self):
|
||||
result = self.deploy("Install", stopped=True)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertNotIn('service signal', self.root.joinpath("ubus-calls").read_text())
|
||||
self.assertIn('Heartbeat loop is stopped', result.stdout)
|
||||
|
||||
def test_failed_loop_reload_restores_paired_files_and_reports_failure(self):
|
||||
result = self.deploy("Install", fail_signal=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('Failed during heartbeat-reload', result.stderr)
|
||||
self.assertNotIn('WAN_RECOVERY_ROUTER_Install_OK', result.stdout)
|
||||
for item, path in self.targets.items():
|
||||
self.assertEqual(path.read_bytes(), self.original[item])
|
||||
|
||||
def test_verify_failure_restores_paired_files(self):
|
||||
result = self.deploy("Install", fail_verify=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("Restoring the agent and WAN helper", result.stderr)
|
||||
for item, path in self.targets.items():
|
||||
self.assertEqual(path.read_bytes(), self.original[item])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -31,7 +31,7 @@ class WanStabilityRollbackTests(unittest.TestCase):
|
||||
source = (ROOT / "vps/deploy-wan-stability-fix.ps1").read_text(encoding="utf-8")
|
||||
script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
script = script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(self.stage.as_posix()))
|
||||
script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v116', 'backup=' + shlex.quote(self.backup.as_posix()))
|
||||
script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v117', 'backup=' + shlex.quote(self.backup.as_posix()))
|
||||
script = script.replace('config=/etc/config/owrtremote', 'config=' + shlex.quote(self.files["config"].as_posix()))
|
||||
for name, original in (("agent", "/usr/sbin/owrt-remote"), ("wan.sh", "/usr/lib/owrt-remote/wan.sh")):
|
||||
script = script.replace('printf ' + original + ' ;;', "printf '%s' " + shlex.quote(self.files[name].as_posix()) + ' ;;')
|
||||
|
||||
@@ -29,7 +29,7 @@ class XrayMobileDeploymentTests(unittest.TestCase):
|
||||
source = (ROOT / "vps/deploy-xray-mobile-fix.ps1").read_text(encoding="utf-8")
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace("target=/www/cgi-bin/owrt-remote", "target=" + shlex.quote(self.target.as_posix()))
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v116",
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v117",
|
||||
"backup_dir=" + shlex.quote(self.backup_dir.as_posix()))
|
||||
|
||||
def deploy(self, action, content=None, fail_verify=False):
|
||||
|
||||
@@ -25,7 +25,7 @@ class XrayValidationDeploymentTests(unittest.TestCase):
|
||||
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
|
||||
source = (ROOT / "vps/deploy-xray-validation-fix.ps1").read_text(encoding="utf-8")
|
||||
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v116",
|
||||
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v117",
|
||||
"backup_dir=" + shlex.quote(self.backup.as_posix()))
|
||||
self.script = self.script.replace("printf /www/cgi-bin/owrt-remote", "printf %s " + shlex.quote(self.targets["cgi"].as_posix()))
|
||||
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
|
||||
|
||||
Reference in new issue
Block a user