Release v117 with persistent logs and WAN tunnel recovery

This commit is contained in:
kzolotarev95 committed 2026-10-15 23:08:19 +04:00
1 parent 5f65a3d811
commit b62db40daf
61 files changed
+3077 -435

No files matched your search

+44 -7
View File
@@ -175,16 +175,53 @@ wget -O - "https://raw.githubusercontent.com/kzolotarev95/luci-app-owrt-remote/m
и ошибки Xray. Успешные Heartbeat без изменения состояния и повторяющиеся и ошибки Xray. Успешные Heartbeat без изменения состояния и повторяющиеся
события не заполняют журнал; подробности каждого соединения Xray отключены. события не заполняют журнал; подробности каждого соединения Xray отключены.
В просмотре успех отмечается зелёным, предупреждение — жёлтым, ошибка — красным. В просмотре успех отмечается зелёным, предупреждение — жёлтым, ошибка — красным.
Журнал хранится в `/tmp/owrt-remote-logs/`, занимает около 2 МБ максимум, Постоянный журнал и курсоры доставки хранятся на проверенном доступном для записи
очищается каждые 24 часа и исчезает после перезагрузки. Сбор работает отдельно Flash-разделе: обычно `/overlay/owrt-remote/logs/`. `/overlay` используется только
от туннеля; остановка Xray не останавливает журнал. Ключи и пароли скрываются. если это настоящая точка монтирования JFFS2/UBIFS/ext4/f2fs; tmpfs и корневой
Перед записью проверяется tmpfs: журнал и его служебное состояние не размещаются OverlayFS не считаются постоянным хранилищем. На системах с непосредственно
на Flash. После очистки одинаковые факты могут быть записаны заново. записываемым постоянным корнем используется `/etc/owrt-remote/logs/`.
Предел — 2 МиБ данных и 256 неизменяемых фрагментов до 16 КиБ каждый.
Доставленные фрагменты удаляются через 24 часа или раньше при нехватке места.
Неподтверждённые записи не удаляются автоматически: при заполненной очереди
новые события сверх лимита пропускаются, после восстановления пишется их число.
В `/tmp/owrt-remote-log-runtime/` остаются только буфер записи до 16 КиБ,
небольшой кеш повторов, блокировки, FIFO и ограниченные пакеты HTTP.
Буфер сбрасывается раз в 60 секунд; обычная остановка сборщика сбрасывает его
перед выходом. После внезапного отключения питания возможна потеря последней
минуты в RAM; завершённые фрагменты имеют контрольную сумму и сохраняются
после перезагрузки. Состояние меняется атомарно на том же разделе, а sync
выполняется только при фиксации пакета, не после каждой строки.
Если Flash недоступна, используется ограниченный журнал в RAM до 64 КиБ
плюс буфер 16 КиБ с предупреждением. При ошибке записи Flash сохранённая
очередь остаётся на месте; возвращение к ней — после восстановления хранилища
и перезагрузки роутера. Менее 2 МиБ свободного Flash останавливает новые записи
на неё. Системный logd, WAN, DNS, Podkop, Xray и конфигурация сети не меняются.
Сбор работает отдельно от туннеля. Ключи и пароли скрываются.
Контрольная сумма фрагментов вычисляется доступным `sha256sum`, `md5sum`
или `cksum`; пустая/ошибочная сумма не позволяет зафиксировать новый фрагмент.
Фрагменты без суммы, оставленные ранним локальным исправлением, восстанавливаются
из сохранённых файлов в отдельный журнал и повторно отправляются на VPS.
Пустой пакет состояния больше не подтверждает неполученные байты журнала.
Старые предупреждения о ротации остаются историей; уже потерянные файлы
восстановить нельзя. Новая контрольная сумма проверяет дальнейшую целостность,
но не доказывает целостность фрагментов, у которых раньше не было суммы.
Подробности устройства хранения и проверки: [tests/PERSISTENT_LOGS.md](tests/PERSISTENT_LOGS.md).
В карточке роутера на VPS есть **Глобал логи**, открывающиеся в новой вкладке. В карточке роутера на VPS есть **Глобал логи**, открывающиеся в новой вкладке.
Архив хранится в постоянной SQLite-базе `/var/lib/owrt-remote/hub.db`
(или по явно настроенному пути БД), до 2 МиБ на роутер. Тип файловой системы
и путь показаны в окне логов; для tmpfs/ramfs выводится предупреждение.
Локальный установщик VPS проверяет постоянный записываемый диск перед заменой
Hub и не меняет путь БД или конфигурацию сервера.
Там доступны те же скачивание, просмотр с обновлением каждые 3 секунды и очистка, Там доступны те же скачивание, просмотр с обновлением каждые 3 секунды и очистка,
а также зелёные, жёлтые и красные записи. Роутер доставляет новые записи на VPS а также зелёные, жёлтые и красные записи. Роутер доставляет новые записи на VPS
отдельно от Heartbeat: повторная отправка после обрыва не создаёт дубликатов. отдельно от Heartbeat: повторная отправка после обрыва не создаёт дубликатов.
Новые события обычно поступают после пакетной записи на Flash — до 60 секунд;
проверка очереди и обновление просмотра выполняются каждые 3 секунды.
Остановка Remote кнопкой «Остановить» не прекращает доставку журнала: пока Остановка Remote кнопкой «Остановить» не прекращает доставку журнала: пока
работает интернет, записи об остановке и дальнейшие действия поступают на VPS. работает интернет, записи об остановке и дальнейшие действия поступают на VPS.
Над журналом Hub показывает полученное состояние: Remote выключен, Xray Над журналом Hub показывает полученное состояние: Remote выключен, Xray
@@ -195,9 +232,9 @@ Hub сообщает об этом прямо; последнее предупр
Кнопка доступна и при оффлайн: Hub хранит последние доставленные записи в своей Кнопка доступна и при оффлайн: Hub хранит последние доставленные записи в своей
базе, до 2 МБ на роутер. Архив сохраняется после перезагрузки роутера и Hub; базе, до 2 МБ на роутер. Архив сохраняется после перезагрузки роутера и Hub;
очистка ОЗУ роутера раз в сутки не удаляет копию на VPS. Hub записывает факт ротация локального журнала не удаляет копию на VPS. Hub записывает факт
истечения ожидания Heartbeat. Записи, созданные после полной потери связи, истечения ожидания Heartbeat. Записи, созданные после полной потери связи,
появятся только при её восстановлении, если ещё сохранены в ОЗУ роутера. появятся только при её восстановлении, если сохранены в ограниченной очереди роутера.
Очистка из Hub сразу удаляет архив VPS, а очистка журнала оффлайн-роутера Очистка из Hub сразу удаляет архив VPS, а очистка журнала оффлайн-роутера
применится при следующей доставке. Доступ к журналу требует полного доступа применится при следующей доставке. Доступ к журналу требует полного доступа
к этому роутеру; удаление роутера удаляет его архив. к этому роутеру; удаление роутера удаляет его архив.
+7
View File
@@ -5,6 +5,7 @@ START=98
STOP=11 STOP=11
start_service() { start_service() {
/bin/sh /usr/lib/owrt-remote/logs.sh migrate || logger -t owrt-remote 'WARN: старый RAM-журнал ещё не перенесён; см. свободное место Flash.'
procd_open_instance journal procd_open_instance journal
procd_set_param command /bin/sh /usr/lib/owrt-remote/logs.sh collect procd_set_param command /bin/sh /usr/lib/owrt-remote/logs.sh collect
procd_set_param respawn 3600 5 5 procd_set_param respawn 3600 5 5
@@ -18,3 +19,9 @@ start_service() {
procd_set_param respawn 3600 5 5 procd_set_param respawn 3600 5 5
procd_close_instance procd_close_instance
} }
stop_service() {
# Flush a last bounded batch on ordinary stop/shutdown. Sudden power loss can
# lose only the current RAM write buffer; committed immutable chunks survive.
/bin/sh /usr/lib/owrt-remote/logs.sh flush force >/dev/null 2>&1 || true
}
+621 -174
View File
@@ -1,89 +1,425 @@
#!/bin/sh #!/bin/sh
# The private Remote journal lives in RAM. It never changes the system logd. # Persistent, bounded spool; only the small write buffer and process state use RAM.
LOG_DIR="${OWRT_REMOTE_LOG_DIR:-/tmp/owrt-remote-logs}" RUN_DIR="${OWRT_REMOTE_LOG_RUNTIME:-/tmp/owrt-remote-log-runtime}"
LOG_LIMIT=1048576 LOG_LIMIT=2097152
LOG_KEEP_SECONDS=86400 LOG_KEEP_SECONDS=86400
LOG_PART_LIMIT=256
LOG_BUFFER_LIMIT=16384
LOG_BATCH_LIMIT=16384
LOG_FLUSH_SECONDS=60
LOG_FREE_KB=2048
log_clock() { cut -d. -f1 /proc/uptime; } log_clock() { cut -d. -f1 /proc/uptime; }
log_ram_storage() { log_epoch() { date +%s; }
local resolved parent log_self_pid() {
if [ -e "$LOG_DIR" ] || [ -L "$LOG_DIR" ]; then # $$ is the parent shell's PID inside ash subshells/pipelines. Reading proc
resolved="$(readlink -f "$LOG_DIR")" || return 1 # with the shell builtin yields the actual lock holder, including the writer.
else local self rest
# BusyBox realpath may require the last component to exist. if read -r self rest </proc/self/stat 2>/dev/null; then printf '%s\n' "$self"; else printf '%s\n' "$$"; fi
# Resolve the existing parent before creating the journal directory. }
parent="$(readlink -f "$(dirname "$LOG_DIR")")" || return 1 log_mount() {
resolved="$parent/$(basename "$LOG_DIR")" # Resolve symlinks, including the existing ancestor of a new directory.
fi local path suffix
awk -v path="$resolved" ' path="$1"; suffix=''
case "$path" in /*) ;; *) return 1 ;; esac
while [ ! -e "$path" ]; do
[ "$path" != / ] || return 1
suffix="/$(basename "$path")$suffix"; path="$(dirname "$path")"
done
path="$(readlink -f "$path")$suffix" || return 1
awk -v path="$path" '
(path == $2 || index(path, $2 "/") == 1 || $2 == "/") && length($2) > best { (path == $2 || index(path, $2 "/") == 1 || $2 == "/") && length($2) > best {
best = length($2); filesystem = $3 best=length($2); mount=$2; fs=$3; flags=$4
} }
END { exit(filesystem == "tmpfs" ? 0 : 1) } END { if (best) print mount, fs, flags; else exit 1 }
' /proc/mounts ' /proc/mounts
} }
log_ram_storage() {
local mount fs flags
read -r mount fs flags <<EOF
$(log_mount "$RUN_DIR")
EOF
[ "$fs" = tmpfs ]
}
log_flash_storage() {
local mount fs flags
read -r mount fs flags <<EOF
$(log_mount "$1")
EOF
case "$fs" in jffs2|ubifs|ext4|f2fs|btrfs|ext2|ext3) ;; *) return 1 ;; esac
case ",$flags," in *,rw,*) ;; *) return 1 ;; esac
# A tmpfs/root overlay and an unmounted /overlay are never persistent storage.
[ "$mount" != /overlay ] || [ -d /overlay ]
}
log_init() { log_init() {
local candidate selected mount fs flags free
umask 077 umask 077
log_ram_storage || { printf 'Remote journal requires RAM (tmpfs); refusing persistent storage.\n' >&2; return 1; } log_ram_storage || { printf 'Remote: runtime directory must be tmpfs.\n' >&2; return 1; }
mkdir -p "$RUN_DIR" || return 1
chmod 700 "$RUN_DIR" || return 1
# Pin one storage choice for this boot; never interleave two journal IDs.
selected="$(cat "$RUN_DIR/storage" 2>/dev/null)"
if [ -n "$selected" ]; then
LOG_DIR="$selected"
else
candidate="${OWRT_REMOTE_LOG_DIR:-}"
if [ -z "$candidate" ]; then
# Select /overlay only when it really is a writable persistent mount.
read -r mount fs flags <<EOF
$(log_mount /overlay)
EOF
if [ "$mount" = /overlay ] && log_flash_storage /overlay; then
candidate=/overlay/owrt-remote/logs
elif log_flash_storage /; then
# OpenWrt installations with a directly writable ext4/UBIFS root.
candidate=/etc/owrt-remote/logs
fi
fi
if [ -n "$candidate" ] && log_flash_storage "$candidate" &&
mkdir -p "$candidate" && [ -w "$candidate" ]; then
LOG_DIR="$candidate"
chmod 700 "$LOG_DIR" || return 1
free="$(df -Pk "$LOG_DIR" | awk 'NR==2 {print $4}')"
case "$free" in ''|*[!0-9]*) free=0 ;; esac
# Reading an existing queue is still useful on a full disk; starting
# a new one must not consume the last bytes needed for router settings.
if [ ! -s "$LOG_DIR/state" ] && [ "$free" -lt "$LOG_FREE_KB" ]; then LOG_DIR="$RUN_DIR/fallback"; fi
else
LOG_DIR="$RUN_DIR/fallback"
fi
fi
PERSISTENT=1
if [ "$LOG_DIR" = "$RUN_DIR/fallback" ]; then
PERSISTENT=0; LOG_LIMIT=65536; LOG_PART_LIMIT=8
mkdir -p "$LOG_DIR" || return 1 mkdir -p "$LOG_DIR" || return 1
chmod 700 "$LOG_DIR" fi
} }
log_lock() { log_lock() {
local attempt owner local attempt owner
log_init || return 1 log_init || return 1
attempt=0 attempt=0
while ! mkdir "$LOG_DIR/lock" 2>/dev/null; do while ! mkdir "$RUN_DIR/lock" 2>/dev/null; do
owner="$(cat "$LOG_DIR/lock/pid" 2>/dev/null)" owner="$(cat "$RUN_DIR/lock/pid" 2>/dev/null)"
case "$owner" in case "$owner" in
''|*[!0-9]*) ''|*[!0-9]*)
# Recover a writer killed between mkdir and writing its PID. [ "$attempt" -lt 20 ] || log_reap_lock "$RUN_DIR/lock" "$attempt" ;;
if [ "$attempt" -ge 20 ]; then *) kill -0 "$owner" 2>/dev/null || log_reap_lock "$RUN_DIR/lock" "$attempt" ;;
rm -f "$LOG_DIR/lock/pid"
rmdir "$LOG_DIR/lock" 2>/dev/null || true
fi ;;
*) if ! kill -0 "$owner" 2>/dev/null; then
rm -f "$LOG_DIR/lock/pid"
rmdir "$LOG_DIR/lock" 2>/dev/null || true
fi ;;
esac esac
attempt=$((attempt + 1)) attempt=$((attempt + 1)); [ "$attempt" -lt 100 ] || return 1
[ "$attempt" -lt 100 ] || return 1
sleep 0.1 sleep 0.1
done done
printf '%s\n' "$$" >"$LOG_DIR/lock/pid" log_self_pid >"$RUN_DIR/lock/pid"
} # Another first writer may have selected storage while we waited.
log_unlock() { if [ -s "$RUN_DIR/storage" ]; then
rm -f "$LOG_DIR/lock/pid" LOG_DIR="$(cat "$RUN_DIR/storage")"
rmdir "$LOG_DIR/lock" 2>/dev/null || true [ "$LOG_DIR" != "$RUN_DIR/fallback" ] || { PERSISTENT=0; LOG_LIMIT=65536; LOG_PART_LIMIT=8; }
} else
log_reset_locked() { printf '%s\n' "$LOG_DIR" >"$RUN_DIR/storage"
: >"$LOG_DIR/current.log"
rm -f "$LOG_DIR/previous.log"
rm -f "$LOG_DIR/facts" "$LOG_DIR/facts.new" "$LOG_DIR/heartbeat-ok"
rm -f "$LOG_DIR/journal-id" "$LOG_DIR/total" "$LOG_DIR/sync-ack"
log_clock >"$LOG_DIR/age"
}
log_cursor_locked() {
local total
if [ ! -s "$LOG_DIR/journal-id" ]; then
cat /proc/sys/kernel/random/uuid >"$LOG_DIR/journal-id" 2>/dev/null ||
printf '%s-%s-%s\n' "$(date +%s)" "$(log_clock)" "$$" >"$LOG_DIR/journal-id"
fi fi
if [ ! -s "$LOG_DIR/total" ]; then # Change storage only under the same append/export lock. On write/mount
total="$(wc -c <"$LOG_DIR/current.log")" # failure preserve the Flash queue and switch this boot to a bounded fallback.
[ ! -f "$LOG_DIR/previous.log" ] || total=$((total + $(wc -c <"$LOG_DIR/previous.log"))) if [ "$LOG_DIR" != "$RUN_DIR/fallback" ] && { ! log_flash_storage "$LOG_DIR" || [ ! -w "$LOG_DIR" ] || [ -f "$RUN_DIR/write-warning" ]; }; then
printf '%s\n' "$total" >"$LOG_DIR/total" log_fallback_locked || { log_unlock; return 1; }
fi fi
} }
log_maintain_locked() { log_reap_lock() {
local now age local owner reap_owner
now="$(log_clock)" # Serialize stale-lock removal itself, so two contenders cannot remove a
age="$(cat "$LOG_DIR/age" 2>/dev/null)" # freshly acquired lock after observing the same previous dead owner.
case "$age" in ''|*[!0-9]*) log_reset_locked; return ;; esac if ! mkdir "$1/reap" 2>/dev/null; then
if [ "$now" -lt "$age" ] || [ "$((now - age))" -ge "$LOG_KEEP_SECONDS" ]; then reap_owner="$(cat "$1/reap/pid" 2>/dev/null)"
log_reset_locked case "$reap_owner" in
''|*[!0-9]*) [ "${2:-0}" -ge 20 ] || return 0 ;;
*) kill -0 "$reap_owner" 2>/dev/null && return 0 ;;
esac
rm -f "$1/reap/pid"; rmdir "$1/reap" 2>/dev/null || return 0
return 0
fi
log_self_pid >"$1/reap/pid"
owner="$(cat "$1/pid" 2>/dev/null)"
case "$owner" in
''|*[!0-9]*) rm -f "$1/pid" ;;
*) if kill -0 "$owner" 2>/dev/null; then rm -f "$1/reap/pid"; rmdir "$1/reap"; return 0; else rm -f "$1/pid"; fi ;;
esac
rm -f "$1/reap/pid"; rmdir "$1/reap"; rmdir "$1" 2>/dev/null || true
}
log_fallback_locked() {
LOG_DIR="$RUN_DIR/fallback"; PERSISTENT=0; LOG_LIMIT=65536; LOG_PART_LIMIT=8
mkdir -p "$LOG_DIR" || return 1
printf '%s\n' "$LOG_DIR" >"$RUN_DIR/storage"
rm -f "$RUN_DIR/recovered" "$RUN_DIR/recovered-v2" "$RUN_DIR/ack"
}
log_unlock() { rm -f "$RUN_DIR/lock/pid"; rmdir "$RUN_DIR/lock" 2>/dev/null || true; }
log_warn_locked() {
# One warning until successful recovery; never feed our own collector a loop.
if [ ! -f "$RUN_DIR/$1" ]; then
local warning size
warning="$(printf '[WARN] %s owrt-remote: %s' "$(date '+%Y-%m-%d %H:%M:%S')" "$2")"
size=0; [ ! -f "$RUN_DIR/pending" ] || size="$(wc -c <"$RUN_DIR/pending")"
if [ "$((size + $(printf '%s\n' "$warning" | wc -c)))" -le "$LOG_BUFFER_LIMIT" ]; then printf '%s\n' "$warning" >>"$RUN_DIR/pending"; fi
: >"$RUN_DIR/$1"
printf 'Remote: %s\n' "$2" >&2
fi fi
} }
log_new_id() {
cat /proc/sys/kernel/random/uuid 2>/dev/null || printf '%s-%s-%s\n' "$(log_epoch)" "$(log_clock)" "$$"
}
log_checksum() {
# OpenWrt does not necessarily ship the optional BusyBox cksum applet.
# Validate command output before publishing a chunk or advancing its cursor.
local method value
case "${2:-}" in
s256-*) method=s256 ;;
md5-*) method=md5 ;;
*[!0-9]*|'')
if command -v sha256sum >/dev/null 2>&1; then method=s256
elif command -v md5sum >/dev/null 2>&1; then method=md5
else method=crc; fi ;;
*) method=crc ;;
esac
case "$method" in
s256) value="$(sha256sum "$1" 2>/dev/null)" || return 1 ;;
md5) value="$(md5sum "$1" 2>/dev/null)" || return 1 ;;
crc) value="$(cksum <"$1" 2>/dev/null)" || return 1 ;;
esac
# Both BusyBox and GNU utilities separate the digest from the file by space.
value="${value%% *}"
case "$method" in
s256|md5)
case "$value" in ''|*[!0-9a-f]*) return 1 ;; esac
if [ "$method" = s256 ]; then [ "${#value}" = 64 ] || return 1; else [ "${#value}" = 32 ] || return 1; fi
printf '%s-%s\n' "$method" "$value" ;;
crc) case "$value" in ''|*[!0-9]*) return 1 ;; esac; printf '%s\n' "$value" ;;
esac
}
log_checkpoint_locked() {
STATE_CHANGED=0
# Same filesystem + rename: no partially rewritten cursor is ever observed.
# Unchanged state causes no Flash write, including idle sync polls.
if [ "$(cat "$LOG_DIR/state" 2>/dev/null)" = "$JOURNAL $ACK $TOTAL ${CLEAR:--}" ]; then return 0; fi
printf '%s %s %s %s\n' "$JOURNAL" "$ACK" "$TOTAL" "${CLEAR:--}" >"$LOG_DIR/state.new" &&
mv -f "$LOG_DIR/state.new" "$LOG_DIR/state" || return 1
STATE_CHANGED=1
}
log_commit_locked() {
# Durability is a batch operation, never per log line. BusyBox builds without
# sync -f use ordinary sync; no extra package or fsync-per-line is required.
[ "$PERSISTENT" = 0 ] || [ "$STATE_CHANGED" = 0 ] || { sync -f "$LOG_DIR" 2>/dev/null || sync; }
}
log_load_locked() {
local extra file start end crc actual
log_finish_migration_locked || return 1
JOURNAL=''; ACK=0; TOTAL=0; CLEAR='-'
if [ -s "$LOG_DIR/state" ]; then
read -r JOURNAL ACK TOTAL CLEAR extra <"$LOG_DIR/state"
case "$JOURNAL" in ''|*[!a-zA-Z0-9._:-]*) JOURNAL='' ;; esac
case "$ACK:$TOTAL" in *[!0-9:]*|:*|*:) JOURNAL='' ;; esac
case "$CLEAR" in ''|*[!a-zA-Z0-9._:-]*) JOURNAL='' ;; esac
[ "${#JOURNAL}" -le 80 ] && [ "${#CLEAR}" -le 80 ] && [ "${#ACK}" -le 16 ] && [ "${#TOTAL}" -le 16 ] || JOURNAL=''
[ -z "$extra" ] || JOURNAL=''
fi
if [ -z "$JOURNAL" ]; then
# If a state sector was lost, recover the newest committed generation.
# Reset only the ack: Hub offsets make re-delivery harmless.
file="$(ls -t "$LOG_DIR"/part.* 2>/dev/null | head -n 1)"
if [ -n "$file" ]; then
JOURNAL="${file##*/part.}"; JOURNAL="${JOURNAL%.*}"; JOURNAL="${JOURNAL%.*}"; JOURNAL="${JOURNAL%.*}"
case "$JOURNAL" in ''|*[!a-zA-Z0-9._:-]*) JOURNAL='' ;; esac
fi
[ -n "$JOURNAL" ] || JOURNAL="$(log_new_id)"
ACK=0; TOTAL=0; CLEAR='-'
log_checkpoint_locked && log_commit_locked || return 1
fi
# Validate immutable chunks once per collector boot, never load their contents.
if [ ! -f "$RUN_DIR/recovered-v2" ]; then
log_repair_legacy_locked || return 1
# A completed explicit clear may have left old chunks after a power cut.
for file in "$LOG_DIR"/part.*; do
[ -f "$file" ] || continue
case "$file" in "$LOG_DIR/part.$JOURNAL."*) ;; *) rm -f "$file" ;; esac
done
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
log_part_fields "$file" || continue
actual="$(log_checksum "$file" "$crc")" || return 1
if [ "$actual" != "$crc" ] || [ "$(wc -c <"$file")" -ne "$((end - start))" ]; then
# Keep one damaged chunk on Flash for inspection; never export it.
mv -f "$file" "$LOG_DIR/damaged.log" || return 1
log_warn_locked damaged-warning 'Повреждённый фрагмент журнала изолирован; возможна потеря последнего пакета после сбоя питания.'
fi
done
rm -f "$LOG_DIR/chunk.new" "$LOG_DIR/state.new"
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
log_part_fields "$file" || continue
[ "$end" -le "$TOTAL" ] || TOTAL="$end"
done
log_checkpoint_locked && log_commit_locked || return 1
: >"$RUN_DIR/recovered-v2"
fi
# A process can die after durable rename/checkpoint but before clearing its
# RAM buffer. Reconcile that one in-flight batch before accepting another line.
if [ -s "$RUN_DIR/inflight" ]; then
read -r extra start end crc <"$RUN_DIR/inflight"
case "$start:$end" in *[!0-9:]*|:*|*:) return 1 ;; esac
case "$crc" in ''|*[!0-9a-fsmd-]*) return 1 ;; esac
file="$LOG_DIR/$(printf 'part.%s.%018d.%018d.%s' "$extra" "$start" "$end" "$crc")"
if [ "$extra" = "$JOURNAL" ] && [ -f "$file" ]; then
actual="$(log_checksum "$file" "$crc")" || return 1
if [ "$actual" = "$crc" ] && [ "$(wc -c <"$file")" -eq "$((end - start))" ]; then
[ "$TOTAL" -ge "$end" ] || TOTAL="$end"
log_checkpoint_locked && log_commit_locked || return 1
: >"$RUN_DIR/pending"
fi
fi
rm -f "$RUN_DIR/inflight"
fi
[ "$ACK" -le "$TOTAL" ] || ACK=0
if [ -s "$RUN_DIR/ack" ]; then
read -r extra actual <"$RUN_DIR/ack"
if [ "$extra" = "$JOURNAL" ]; then
case "$actual" in ''|*[!0-9]*) ;; *) if [ "$actual" -le "$TOTAL" ] && [ "$actual" -gt "$ACK" ]; then ACK="$actual"; fi ;; esac
fi
fi
if [ ! -f "$RUN_DIR/last-flush" ]; then log_clock >"$RUN_DIR/last-flush"; fi
[ -f "$RUN_DIR/pending" ] || : >"$RUN_DIR/pending"
if [ "$PERSISTENT" = 0 ]; then log_warn_locked fallback-warning 'Flash недоступна: используется ограниченный журнал в ОЗУ (64 КиБ + буфер 16 КиБ); после перезагрузки он исчезнет.'; fi
}
log_part_fields() {
local fields
fields="${1##*/part.$JOURNAL.}"
start="${fields%%.*}"; fields="${fields#*.}"; end="${fields%%.*}"; crc="${fields#*.}"
# Windows fixtures strip trailing dots; also recover a legacy missing suffix.
if [ "$fields" = "$end" ] && [ "${2:-}" = repair ]; then crc=''; fi
case "$start:$end" in *[!0-9:]*|:*|*:) return 1 ;; esac
[ "${#start}" -le 18 ] && [ "${#end}" -le 18 ] || return 1
case "$crc" in
'') [ "${2:-}" = repair ] || return 1 ;;
s256-*) fields="${crc#s256-}"; case "$fields" in *[!0-9a-f]*) return 1 ;; esac; [ "${#fields}" = 64 ] || return 1 ;;
md5-*) fields="${crc#md5-}"; case "$fields" in *[!0-9a-f]*) return 1 ;; esac; [ "${#fields}" = 32 ] || return 1 ;;
*[!0-9]*) return 1 ;;
*) [ "${#crc}" -le 10 ] || return 1 ;;
esac
# Avoid octal arithmetic for zero-padded names.
while [ "${start#0}" != "$start" ]; do start="${start#0}"; done; start="${start:-0}"
while [ "${end#0}" != "$end" ]; do end="${end#0}"; done; end="${end:-0}"
[ "$end" -gt "$start" ] && [ "$((end - start))" -le "$LOG_BUFFER_LIMIT" ]
}
log_repair_legacy_locked() {
local file start end crc found base maximum bytes free new_id dest actual
found=0; base="$TOTAL"; maximum="$TOTAL"; bytes=0
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
log_part_fields "$file" repair || continue
[ -n "$crc" ] || found=1
[ "$start" -ge "$base" ] || base="$start"
[ "$end" -le "$maximum" ] || maximum="$end"
bytes=$((bytes + end - start))
done
[ "$found" = 1 ] || return 0
# The previous bug confirmed empty exports without delivering these chunks.
# Publish a new generation so that the Hub accepts their replay. Only missing
# digest files of the expected length can be recovered; no invented checksum.
if [ "$PERSISTENT" = 1 ]; then
free="$(df -Pk "$LOG_DIR" | awk 'NR==2 {print $4}')"
case "$free" in ''|*[!0-9]*) return 1 ;; esac
[ "$free" -ge "$((LOG_FREE_KB + (bytes + 1023) / 1024))" ] || return 1
fi
new_id="recovered-$(log_new_id)"
mkdir -p "$LOG_DIR/migrating" || return 1
# An interrupted, uncommitted attempt is safe to rebuild from the originals.
rm -f "$LOG_DIR"/migrating/part.* "$LOG_DIR/migrating/chunk.new"
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
log_part_fields "$file" repair || return 1
[ "$(wc -c <"$file")" -eq "$((end - start))" ] || return 1
if [ -n "$crc" ]; then actual="$(log_checksum "$file" "$crc")" && [ "$actual" = "$crc" ] || return 1; fi
cp "$file" "$LOG_DIR/migrating/chunk.new" || return 1
crc="$(log_checksum "$LOG_DIR/migrating/chunk.new")" || return 1
dest="$(printf 'part.%s.%018d.%018d.%s' "$new_id" "$((start - base))" "$((end - base))" "$crc")"
mv -f "$LOG_DIR/migrating/chunk.new" "$LOG_DIR/migrating/$dest" || return 1
done
printf '%s 0 %s %s\n' "$new_id" "$((maximum - base))" "$CLEAR" >"$LOG_DIR/migrating/state.new" &&
mv -f "$LOG_DIR/migrating/state.new" "$LOG_DIR/migrating/state" || return 1
sync -f "$LOG_DIR" 2>/dev/null || sync
log_finish_migration_locked || return 1
JOURNAL="$new_id"; ACK=0; TOTAL=$((maximum - base))
rm -f "$RUN_DIR/ack" "$RUN_DIR/inflight"
log_warn_locked repaired-warning 'Восстановлены фрагменты без контрольной суммы. Сохранённая история повторно отправляется на VPS; ранее целостность этих фрагментов не проверялась.'
}
log_prune_locked() {
local file start end crc age now bytes count
now="$(log_epoch)"; bytes=0; count=0
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
bytes=$((bytes + $(wc -c <"$file"))); count=$((count + 1))
done
for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ -f "$file" ] || continue
log_part_fields "$file" || continue
[ "$end" -le "$ACK" ] || continue
age="$(stat -c %Y "$file" 2>/dev/null)"; age="${age:-$now}"
# Only confirmed chunks may expire or make room. Backwards wall-clock is safe.
if [ "$((now - age))" -ge "$LOG_KEEP_SECONDS" ] || [ "$bytes" -gt "$((LOG_LIMIT - LOG_BUFFER_LIMIT))" ] || [ "$count" -ge "$LOG_PART_LIMIT" ]; then
rm -f "$file" || return 1
bytes=$((bytes - (end - start))); count=$((count - 1))
fi
done
STORED_BYTES="$bytes"; STORED_PARTS="$count"
}
log_flush_locked() {
local force now last bytes free crc dest dropped
force="${1:-}"; now="$(log_clock)"; last="$(cat "$RUN_DIR/last-flush")"
[ "$force" = force ] || [ "$now" -lt "$last" ] || [ "$((now - last))" -ge "$LOG_FLUSH_SECONDS" ] || return 0
# A successful Hub ack is checkpointed before confirmed chunks can be removed.
log_checkpoint_locked || { log_warn_locked write-warning 'Flash: ошибка записи состояния; временный буфер ограничен, рабочие настройки не изменены.'; return 1; }
log_commit_locked || return 1
log_prune_locked || return 1
bytes="$(wc -c <"$RUN_DIR/pending")"
if [ "$bytes" -gt 0 ]; then
if [ "$STORED_BYTES" -gt "$((LOG_LIMIT - bytes))" ] || [ "$STORED_PARTS" -ge "$LOG_PART_LIMIT" ]; then
log_warn_locked queue-warning 'Очередь логов заполнена: неподтверждённые записи сохранены; новые события сверх лимита пропускаются до освобождения места.'
log_clock >"$RUN_DIR/last-flush"
return 0
fi
if [ "$PERSISTENT" = 1 ]; then
free="$(df -Pk "$LOG_DIR" | awk 'NR==2 {print $4}')"
case "$free" in ''|*[!0-9]*) free=0 ;; esac
if [ "$free" -lt "$LOG_FREE_KB" ]; then log_warn_locked write-warning 'На Flash менее 2 МиБ свободного места; запись логов приостановлена, буфер в ОЗУ ограничен.'; return 1; fi
fi
# An immutable chunk is either fully committed by rename, or ignored on recovery.
# A fixed-size RAM buffer is the only source; never copy the entire journal.
cp "$RUN_DIR/pending" "$LOG_DIR/chunk.new" || { log_warn_locked write-warning 'Flash: ошибка записи журнала; данные остаются в ограниченном буфере ОЗУ.'; return 1; }
crc="$(log_checksum "$LOG_DIR/chunk.new")" || { log_warn_locked checksum-warning 'Не удалось вычислить контрольную сумму; журнал не подтверждён, записи остаются в буфере.'; return 1; }
printf '%s %s %s %s\n' "$JOURNAL" "$TOTAL" "$((TOTAL + bytes))" "$crc" >"$RUN_DIR/inflight"
dest="$(printf 'part.%s.%018d.%018d.%s' "$JOURNAL" "$TOTAL" "$((TOTAL + bytes))" "$crc")"
mv -f "$LOG_DIR/chunk.new" "$LOG_DIR/$dest" || { log_warn_locked write-warning 'Flash: не удалось зафиксировать фрагмент журнала.'; return 1; }
TOTAL=$((TOTAL + bytes))
log_checkpoint_locked && log_commit_locked || { log_warn_locked write-warning 'Flash: ошибка фиксации состояния журнала.'; return 1; }
: >"$RUN_DIR/pending"
rm -f "$RUN_DIR/inflight"
rm -f "$RUN_DIR/write-warning" "$RUN_DIR/queue-warning" "$RUN_DIR/buffer-warning"
dropped="$(cat "$RUN_DIR/dropped" 2>/dev/null)"; dropped="${dropped:-0}"
if [ "$dropped" -gt 0 ]; then
printf '[WARN] %s owrt-remote: Восстановлена запись журнала; пропущено новых событий сверх лимита: %s.\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$dropped" >"$RUN_DIR/pending"
rm -f "$RUN_DIR/dropped"
fi
fi
log_clock >"$RUN_DIR/last-flush"
}
log_open() {
log_lock || return 1
trap 'log_unlock' EXIT
trap 'exit 1' INT TERM
if ! log_load_locked; then
if [ "$PERSISTENT" = 1 ]; then
log_fallback_locked && log_load_locked || { log_close; return 1; }
else
log_close; return 1
fi
fi
}
log_close() { log_unlock; trap - EXIT INT TERM; }
log_redact() { log_redact() {
# Never retain command blocks containing credentials or authenticated URLs. # Never retain command blocks containing credentials or authenticated URLs.
LC_ALL=C awk -v fallback="${1:-OK}" ' LC_ALL=C awk -v fallback="${1:-OK}" '
@@ -110,16 +446,23 @@ log_new_fact_locked() {
case "$key" in case "$key" in
'Heartbeat: OK;'*) 'Heartbeat: OK;'*)
heartbeat=1 heartbeat=1
if [ "$(cat "$LOG_DIR/heartbeat-ok" 2>/dev/null)" = "$key" ]; then return 1; fi if [ "$(cat "$RUN_DIR/heartbeat-ok" 2>/dev/null)" = "$key" ]; then return 1; fi
printf '%s\n' "$key" >"$LOG_DIR/heartbeat-ok" printf '%s\n' "$key" >"$RUN_DIR/heartbeat-ok"
# A recovery lets a later failure with the same cause appear again. # A recovery lets the same failure appear again; keys are fixed-size hashes.
if [ -f "$LOG_DIR/facts" ]; then : >"$RUN_DIR/facts" ;;
awk -F '\t' '$2 !~ /Heartbeat:|heartbeat failed/ { print }' "$LOG_DIR/facts" >"$LOG_DIR/facts.new"
mv -f "$LOG_DIR/facts.new" "$LOG_DIR/facts"
fi ;;
esac esac
if [ "$heartbeat" = 1 ]; then return 0; fi if [ "$heartbeat" = 1 ]; then return 0; fi
key="${1%%]*}] $key" # Update heartbeat recovery state before replacing the full key with its hash.
case "$key" in
*Heartbeat:*|*'heartbeat failed'*)
case "$1" in '[ERROR]'*|'[WARN]'*) rm -f "$RUN_DIR/heartbeat-ok" ;; esac ;;
*'Сервис: запуск') rm -f "$RUN_DIR/heartbeat-ok" ;;
esac
printf '%s\n' "${1%%]*}] $key" >"$RUN_DIR/fact-key"
key="$(log_checksum "$RUN_DIR/fact-key")" || { rm -f "$RUN_DIR/fact-key"; return 0; }
rm -f "$RUN_DIR/fact-key"
# Dedup needs only a fingerprint, not the full SHA-256. Keep 64 entries <4 KiB.
key="$(printf '%.40s' "$key")"
now="$(log_clock)" now="$(log_clock)"
# A bounded cache suppresses repeated identical facts while they continue. # A bounded cache suppresses repeated identical facts while they continue.
# After a quiet period a new occurrence is recorded, rather than lost forever. # After a quiet period a new occurrence is recorded, rather than lost forever.
@@ -130,116 +473,213 @@ log_new_fact_locked() {
for (i = (n > 63 ? n - 62 : 1); i <= n; i++) print saved[i] for (i = (n > 63 ? n - 62 : 1); i <= n; i++) print saved[i]
print now "\t" key print now "\t" key
exit(repeated ? 1 : 0) exit(repeated ? 1 : 0)
}' "$LOG_DIR/facts" >"$LOG_DIR/facts.new" 2>/dev/null }' "$RUN_DIR/facts" >"$RUN_DIR/facts.new" 2>/dev/null
new_fact=$? new_fact=$?
mv -f "$LOG_DIR/facts.new" "$LOG_DIR/facts" mv -f "$RUN_DIR/facts.new" "$RUN_DIR/facts"
[ "$new_fact" = 0 ] || return 1 [ "$new_fact" = 0 ] || return 1
case "$key" in
*Heartbeat:*|*'heartbeat failed'*)
case "$1" in '[ERROR]'*|'[WARN]'*) rm -f "$LOG_DIR/heartbeat-ok" ;; esac ;;
*'Сервис: запуск') rm -f "$LOG_DIR/heartbeat-ok" ;;
esac
return 0 return 0
} }
log_append() { log_append() {
local line size local line size bytes dropped
# Process one bounded line at a time and release the lock between writes.
log_redact "${1:-OK}" | while IFS= read -r line; do log_redact "${1:-OK}" | while IFS= read -r line; do
log_lock || exit 1 log_open || exit 1
trap 'log_unlock' EXIT log_flush_locked || true
trap 'exit 0' INT TERM [ -f "$RUN_DIR/facts" ] || : >"$RUN_DIR/facts"
log_maintain_locked if log_new_fact_locked "$line"; then
log_cursor_locked size="$(wc -c <"$RUN_DIR/pending")"; bytes="$(printf '%s\n' "$line" | wc -c)"
[ -f "$LOG_DIR/facts" ] || : >"$LOG_DIR/facts" # Reserve room for a single overflow warning, too.
if ! log_new_fact_locked "$line"; then if [ "$((size + bytes))" -le "$((LOG_BUFFER_LIMIT - 1024))" ]; then
log_unlock printf '%s\n' "$line" >>"$RUN_DIR/pending"
trap - EXIT INT TERM else
continue dropped="$(cat "$RUN_DIR/dropped" 2>/dev/null)"; dropped="${dropped:-0}"
[ "$dropped" -ge 2147483647 ] || dropped=$((dropped + 1))
printf '%s\n' "$dropped" >"$RUN_DIR/dropped"
log_warn_locked buffer-warning 'Буфер логов заполнен: новые события сверх лимита пропускаются, неподтверждённый журнал сохранён.'
fi fi
size="$(wc -c <"$LOG_DIR/current.log")"
if [ "$size" -ge "$LOG_LIMIT" ]; then
mv -f "$LOG_DIR/current.log" "$LOG_DIR/previous.log"
: >"$LOG_DIR/current.log"
fi fi
printf '%s\n' "$line" >>"$LOG_DIR/current.log" log_close
printf '%s\n' "$(( $(cat "$LOG_DIR/total") + $(printf '%s\n' "$line" | wc -c) ))" >"$LOG_DIR/total"
log_unlock
trap - EXIT INT TERM
done done
} }
log_read() { log_read() {
local previous_size current_size local file start end crc snapshot mode owner
log_lock || return 1 log_open || return 1
log_maintain_locked # Capture only file names and the small pending buffer. Slow HTTP clients must
# Open the retained files under the lock, then release it before streaming # never hold the append lock; concurrent expiry may omit an already-acked chunk.
# to a slow phone. Downloads must not prevent the collector from writing. # One viewer snapshot at a time bounds temporary files even with slow clients.
previous_size=0 if ! mkdir "$RUN_DIR/view-lock" 2>/dev/null; then
if [ -f "$LOG_DIR/previous.log" ]; then owner="$(cat "$RUN_DIR/view-lock/pid" 2>/dev/null)"
previous_size="$(wc -c <"$LOG_DIR/previous.log")" case "$owner" in ''|*[!0-9]*) ;; *) kill -0 "$owner" 2>/dev/null && { log_close; return 1; } ;; esac
exec 3<"$LOG_DIR/previous.log" rm -f "$RUN_DIR/view-lock/pid"; rmdir "$RUN_DIR/view-lock" 2>/dev/null || { log_close; return 1; }
else mkdir "$RUN_DIR/view-lock" || { log_close; return 1; }
exec 3</dev/null
fi fi
current_size="$(wc -c <"$LOG_DIR/current.log")" printf '%s\n' "$$" >"$RUN_DIR/view-lock/pid"
exec 4<"$LOG_DIR/current.log" rm -f "$RUN_DIR"/view.*
log_unlock snapshot="$RUN_DIR/view.$$"; mode="${1:-}"
if [ "${1:-}" = live ]; then : >"$snapshot.list"
{ head -c "$previous_size" <&3; head -c "$current_size" <&4; } | tail -c 65536 for file in "$LOG_DIR"/part."$JOURNAL".*; do
[ ! -f "$file" ] || printf '%s\n' "$file" >>"$snapshot.list"
done
cp "$RUN_DIR/pending" "$snapshot.pending" || { rm -f "$RUN_DIR/view-lock/pid"; rmdir "$RUN_DIR/view-lock"; log_close; return 1; }
log_close
trap 'rm -f "$snapshot.list" "$snapshot.pending" "$RUN_DIR/view-lock/pid"; rmdir "$RUN_DIR/view-lock" 2>/dev/null || true' EXIT
trap 'exit 1' INT TERM
if [ "$mode" = live ]; then
{ while IFS= read -r file; do [ ! -f "$file" ] || cat "$file"; done <"$snapshot.list"; cat "$snapshot.pending"; } | tail -c 65536
else else
head -c "$previous_size" <&3 while IFS= read -r file; do [ ! -f "$file" ] || cat "$file"; done <"$snapshot.list"
head -c "$current_size" <&4 cat "$snapshot.pending"
fi fi
exec 3<&- 4<&- rm -f "$snapshot.list" "$snapshot.pending" "$RUN_DIR/view-lock/pid"; rmdir "$RUN_DIR/view-lock"; trap - EXIT INT TERM
} }
log_clear() { log_reset_locked() {
log_lock || return 1 local clear
log_reset_locked clear="${1:--}"
log_unlock # Commit a new generation before deleting its predecessor; a crash cannot
# attach old offsets to a new journal. Explicit clear also discards unacked data.
JOURNAL="$(log_new_id)"; ACK=0; TOTAL=0; CLEAR="$clear"
# Chunks include their generation ID: atomically switching state makes all
# previous chunks inactive, even if power fails during their subsequent removal.
log_checkpoint_locked && log_commit_locked || return 1
rm -f "$LOG_DIR"/part.* "$LOG_DIR/damaged.log"
: >"$RUN_DIR/pending"
rm -f "$RUN_DIR/ack" "$RUN_DIR/facts" "$RUN_DIR/heartbeat-ok" "$RUN_DIR/dropped" "$RUN_DIR/buffer-warning" "$RUN_DIR/inflight"
log_clock >"$RUN_DIR/last-flush"
} }
log_clear() { log_open || return 1; log_reset_locked; result=$?; log_close; return "$result"; }
log_export() { log_export() {
local journal total previous current ack_id ack start available count clear_ack chunk local file start end crc from to skip count chunk clear
log_lock || return 1 log_open || return 1
log_maintain_locked log_flush_locked || true
log_cursor_locked # Reap a temporary export left by SIGKILL. The journal lock excludes another
journal="$(cat "$LOG_DIR/journal-id")"; total="$(cat "$LOG_DIR/total")" # exporter; there is never more than one raw packet in RAM.
ack_id=''; ack=0 rm -f "$RUN_DIR"/export.*
if [ -s "$LOG_DIR/sync-ack" ]; then read -r ack_id ack <"$LOG_DIR/sync-ack"; fi # An empty/status packet proves no bytes beyond the last actual ack.
[ "$ack_id" = "$journal" ] || ack=0 from="$ACK"; to="$ACK"
case "$ack" in ''|*[!0-9]*) ack=0 ;; esac chunk="$RUN_DIR/export.$$"; : >"$chunk"
[ "$ack" -le "$total" ] || ack=0 trap 'rm -f "$chunk"; log_unlock' EXIT
previous=0 # One immutable chunk per packet, never more than 16 KiB before base64.
if [ -f "$LOG_DIR/previous.log" ]; then previous="$(wc -c <"$LOG_DIR/previous.log")"; exec 3<"$LOG_DIR/previous.log"; else exec 3</dev/null; fi for file in "$LOG_DIR"/part."$JOURNAL".*; do
current="$(wc -c <"$LOG_DIR/current.log")"; exec 4<"$LOG_DIR/current.log" [ -f "$file" ] || continue
start=$((total - previous - current)); [ "$ack" -le "$start" ] || start="$ack" log_part_fields "$file" || continue
available=$((total - start)); count="$available"; [ "$count" -le 65536 ] || count=65536 [ "$end" -gt "$ACK" ] || continue
clear_ack="$(cat "$LOG_DIR/sync-clear-ack" 2>/dev/null)" from="$start"; [ "$ACK" -le "$start" ] || from="$ACK"
log_unlock skip=$((from - start)); count=$((end - from)); [ "$count" -le "$LOG_BATCH_LIMIT" ] || count="$LOG_BATCH_LIMIT"
chunk="$LOG_DIR/export.$$" # dd streams a fixed-size segment; no large tail ring or shell string.
trap 'rm -f "$chunk"; exec 3<&- 4<&-' EXIT dd if="$file" bs=1 skip="$skip" count="$count" 2>/dev/null >"$chunk" || { rm -f "$chunk"; log_close; return 1; }
: >"$chunk" to=$((from + $(wc -c <"$chunk"))); break
if [ "$count" -gt 0 ]; then done
{ head -c "$previous" <&3; head -c "$current" <&4; } | tail -c "$available" | head -c "$count" >"$chunk" clear="$CLEAR"; [ "$clear" != - ] || clear=''
fi printf '{"journal_id":"%s","from":%s,"to":%s,"clear_ack":"%s","data":"' "$JOURNAL" "$from" "$to" "$clear"
count="$(wc -c <"$chunk")" base64 <"$chunk" | tr -d '\r\n'; printf '"}\n'
printf '{"journal_id":"%s","from":%s,"to":%s,"clear_ack":"%s","data":"' "$journal" "$start" "$((start + count))" "$clear_ack" rm -f "$chunk"
base64 <"$chunk" | tr -d '\r\n' log_close
printf '"}\n'
rm -f "$chunk"; exec 3<&- 4<&-; trap - EXIT
} }
log_ack() { log_ack() {
local journal offset clear_token local journal offset clear
journal="$1"; offset="$2"; clear_token="${3:-}" journal="$1"; offset="$2"; clear="${3:-}"
case "$journal$clear_token" in *[!a-zA-Z0-9._:-]*) return 1 ;; esac case "$journal$clear" in ''|*[!a-zA-Z0-9._:-]*) return 1 ;; esac
case "$offset" in ''|*[!0-9]*) return 1 ;; esac case "$offset" in ''|*[!0-9]*) return 1 ;; esac
log_lock || return 1 [ "${#journal}" -le 80 ] && [ "${#clear}" -le 80 ] && [ "${#offset}" -le 16 ] || return 1
log_maintain_locked log_open || return 1
if [ -n "$clear_token" ] && [ "$(cat "$LOG_DIR/sync-clear-ack" 2>/dev/null)" != "$clear_token" ]; then if [ -n "$clear" ] && [ "$CLEAR" != "$clear" ]; then
log_reset_locked log_reset_locked "$clear" || { log_close; return 1; }
printf '%s\n' "$clear_token" >"$LOG_DIR/sync-clear-ack" elif [ "$JOURNAL" = "$journal" ]; then
elif [ "$(cat "$LOG_DIR/journal-id" 2>/dev/null)" = "$journal" ]; then if [ "$offset" -gt "$TOTAL" ] || [ "$offset" -lt "$ACK" ]; then log_close; return 1; fi
printf '%s %s\n' "$journal" "$offset" >"$LOG_DIR/sync-ack" ACK="$offset"
# Checkpoint with the next periodic flush. A crash only replays confirmed
# bytes; the existing Hub journal-ID/offset protocol suppresses duplicates.
printf '%s %s\n' "$JOURNAL" "$ACK" >"$RUN_DIR/ack.new" && mv -f "$RUN_DIR/ack.new" "$RUN_DIR/ack"
fi fi
log_unlock log_close
}
log_flush() { log_open || return 1; log_flush_locked "${1:-}"; result=$?; log_close; return "$result"; }
log_status() {
log_open || return 1
printf 'storage=%s\npersistent=%s\nruntime=%s\nlimit_bytes=%s\nmax_parts=%s\nbuffer_limit_bytes=%s\nflush_seconds=%s\njournal_id=%s\ncommitted_bytes=%s\nack_bytes=%s\npending_bytes=%s\n' "$LOG_DIR" "$PERSISTENT" "$RUN_DIR" "$LOG_LIMIT" "$LOG_PART_LIMIT" "$LOG_BUFFER_LIMIT" "$LOG_FLUSH_SECONDS" "$JOURNAL" "$TOTAL" "$ACK" "$(wc -c <"$RUN_DIR/pending")"
log_close
}
log_migrate() {
local legacy name free mount fs flags old_id old_total old_ack ack_id previous current base file bytes index size crc target token
legacy=/tmp/owrt-remote-logs
[ -d "$legacy" ] || return 0
[ -f "$legacy/current.log" ] || [ -f "$legacy/previous.log" ] || return 0
log_open || return 1
read -r mount fs flags <<EOF
$(log_mount "$legacy")
EOF
if [ "$fs" != tmpfs ]; then log_close; return 0; fi
if [ "$PERSISTENT" != 1 ]; then
# Never leave the old 2 MiB RAM journal alongside a 64 KiB fallback. Keep
# one bounded tail, with an explicit warning about the discarded history.
# Legacy bytes were already redacted. Copy a bounded tail as a stream,
# avoiding a slow per-line replay or a second long-lived RAM archive.
free=$((LOG_BUFFER_LIMIT - 1024 - $(wc -c <"$RUN_DIR/pending")))
[ "$free" -le 8192 ] || free=8192
if [ "$free" -gt 0 ]; then
{ cat "$legacy/previous.log" "$legacy/current.log" 2>/dev/null; } | tail -c "$free" >>"$RUN_DIR/pending"
fi
log_warn_locked legacy-warning 'Flash недоступна: от старого RAM-журнала сохранён только ограниченный хвост; остальная история отброшена.'
else
# Called with the old collector stopped. Retain its journal identity and
# byte cursor, so undelivered history resumes without duplicating Hub data.
free="$(df -Pk "$LOG_DIR" | awk 'NR==2 {print $4}')"
case "$free" in ''|*[!0-9]*) free=0 ;; esac
if [ "$free" -lt 6144 ]; then log_close; printf 'Remote: legacy archive retained in RAM; migration needs 6 MiB free Flash.\n' >&2; return 1; fi
old_id="$(cat "$legacy/journal-id" 2>/dev/null)"
case "$old_id" in ''|*[!a-zA-Z0-9._:-]*) old_id="$JOURNAL" ;; esac
old_total="$(cat "$legacy/total" 2>/dev/null)"
previous=0; current=0
[ ! -f "$legacy/previous.log" ] || previous="$(wc -c <"$legacy/previous.log")"
[ ! -f "$legacy/current.log" ] || current="$(wc -c <"$legacy/current.log")"
bytes=$((previous + current))
if [ "$bytes" -gt "$LOG_LIMIT" ]; then log_close; printf 'Remote: legacy journal exceeds 2 MiB; export it before updating.\n' >&2; return 1; fi
case "$old_total" in ''|*[!0-9]*) old_total="$bytes" ;; esac
[ "$old_total" -ge "$bytes" ] || old_total="$bytes"
base=$((old_total - bytes)); old_ack=0; ack_id=''
if [ -f "$legacy/sync-ack" ]; then read -r ack_id old_ack <"$legacy/sync-ack"; fi
case "$old_ack" in ''|*[!0-9]*) old_ack=0 ;; esac
[ "$ack_id" = "$old_id" ] && [ "$old_ack" -le "$old_total" ] || old_ack=0
# Do not replace a nonempty new journal (e.g. after a failed retry).
if [ "$TOTAL" -gt 0 ] && [ "$JOURNAL" != "$old_id" ]; then log_close; printf 'Remote: legacy journal conflict; no history was removed.\n' >&2; return 1; fi
JOURNAL="$old_id"; TOTAL="$base"; ACK="$old_ack"
token="$(cat "$legacy/sync-clear-ack" 2>/dev/null)"
case "$token" in ''|*[!a-zA-Z0-9._:-]*) CLEAR='-' ;; *) CLEAR="$token" ;; esac
# A staging directory on the same Flash filesystem keeps restart recovery
# from importing only half a legacy journal after power loss.
mkdir -p "$LOG_DIR/migrating" || { log_close; return 1; }
for file in "$legacy/previous.log" "$legacy/current.log"; do
[ -f "$file" ] || continue
size="$(wc -c <"$file")"; index=0
while [ "$size" -gt 0 ]; do
dd if="$file" bs=16384 skip="$index" count=1 2>/dev/null >"$LOG_DIR/migrating/chunk.new" || { log_close; return 1; }
bytes="$(wc -c <"$LOG_DIR/migrating/chunk.new")"
[ "$bytes" -gt 0 ] || { log_close; return 1; }
crc="$(log_checksum "$LOG_DIR/migrating/chunk.new")" || { log_close; return 1; }
target="$(printf 'part.%s.%018d.%018d.%s' "$JOURNAL" "$TOTAL" "$((TOTAL + bytes))" "$crc")"
mv -f "$LOG_DIR/migrating/chunk.new" "$LOG_DIR/migrating/$target" || { log_close; return 1; }
TOTAL=$((TOTAL + bytes)); size=$((size - bytes)); index=$((index + 1))
done
done
# Commit a recovery manifest before publishing any fragments. On reboot,
# even when the old /tmp is gone, the entire staged queue can be finished.
printf '%s %s %s %s\n' "$JOURNAL" "$ACK" "$TOTAL" "$CLEAR" >"$LOG_DIR/migrating/state.new" &&
mv -f "$LOG_DIR/migrating/state.new" "$LOG_DIR/migrating/state" || { log_close; return 1; }
sync -f "$LOG_DIR" 2>/dev/null || sync
log_finish_migration_locked || { log_close; return 1; }
rm -f "$RUN_DIR/ack"
sync -f "$LOG_DIR" 2>/dev/null || sync
fi
rm -f "$legacy/current.log" "$legacy/previous.log" "$legacy/age" "$legacy/journal-id" "$legacy/total" "$legacy/sync-ack" "$legacy/sync-clear-ack" "$legacy/facts" "$legacy/facts.new" "$legacy/heartbeat-ok"
log_close
}
log_finish_migration_locked() {
local file
[ -s "$LOG_DIR/migrating/state" ] || return 0
for file in "$LOG_DIR"/migrating/part.*; do [ ! -f "$file" ] || mv -f "$file" "$LOG_DIR/" || return 1; done
mv -f "$LOG_DIR/migrating/state" "$LOG_DIR/state" || return 1
sync -f "$LOG_DIR" 2>/dev/null || sync
rmdir "$LOG_DIR/migrating" 2>/dev/null || true
rm -f "$RUN_DIR/recovered" "$RUN_DIR/recovered-v2" "$RUN_DIR/ack"
} }
log_sync_loop() { log_sync_loop() {
while :; do while :; do
@@ -248,30 +688,37 @@ log_sync_loop() {
done done
} }
log_collect() { log_collect() {
# These variables must remain visible to EXIT even after this function returns. local owner
log_init || return 1 log_open || return 1
fifo="$LOG_DIR/stream.$$" log_close
mkfifo "$fifo" || return 1 if ! mkdir "$RUN_DIR/collector-lock" 2>/dev/null; then
reader=''; writer=''; timer='' owner="$(cat "$RUN_DIR/collector-lock/pid" 2>/dev/null)"
trap 'trap - EXIT; for child in "$reader" "$writer" "$timer"; do [ -z "$child" ] || kill "$child" 2>/dev/null || true; done; wait; rm -f "$fifo"' EXIT case "$owner" in ''|*[!0-9]*) ;; *) kill -0 "$owner" 2>/dev/null && return 0 ;; esac
rm -f "$RUN_DIR/collector-lock/pid"; rmdir "$RUN_DIR/collector-lock" 2>/dev/null || return 1
mkdir "$RUN_DIR/collector-lock" || return 1
fi
log_self_pid >"$RUN_DIR/collector-lock/pid"
rm -f "$RUN_DIR"/stream.*
fifo="$RUN_DIR/stream.$$"; reader=''; writer=''; timer=''
trap 'trap - EXIT; for child in "$reader" "$writer" "$timer"; do [ -z "$child" ] || kill "$child" 2>/dev/null || true; done; wait; rm -f "$fifo" "$RUN_DIR/collector-lock/pid"; rmdir "$RUN_DIR/collector-lock" 2>/dev/null || true; (log_flush force) || true' EXIT
trap 'exit 0' INT TERM trap 'exit 0' INT TERM
# Follow only new Remote/Xray service records: a clear must not reload history. mkfifo "$fifo" || return 1
logread -f -e 'owrt-remote\(\[[0-9][0-9]*\]\)\{0,1\}:' >"$fifo" 2>/dev/null & logread -f -e 'owrt-remote\(\[[0-9][0-9]*\]\)\{0,1\}:' >"$fifo" 2>/dev/null & reader=$!
reader=$! while IFS= read -r line; do printf '%s\n' "$line" | log_append; done <"$fifo" & writer=$!
while IFS= read -r line; do printf '%s\n' "$line" | log_append; done <"$fifo" &
writer=$!
while :; do while :; do
log_lock || return 1 # Lock-operation traps must not replace the collector's child/FIFO cleanup.
log_maintain_locked (log_flush) || true
log_unlock sleep 5 & timer=$!
sleep 60 & timer=$!
wait "$timer" || return 0 wait "$timer" || return 0
kill -0 "$reader" 2>/dev/null && kill -0 "$writer" 2>/dev/null || return 1 kill -0 "$reader" 2>/dev/null && kill -0 "$writer" 2>/dev/null || return 1
done done
} }
case "${1:-}" in case "${1:-}" in
check) log_init ;; check) log_open && log_close ;;
status) log_status ;;
migrate) log_migrate ;;
flush) log_flush "${2:-}" ;;
collect) log_collect ;; collect) log_collect ;;
read) log_read "${2:-}" ;; read) log_read "${2:-}" ;;
clear) log_clear ;; clear) log_clear ;;
@@ -282,5 +729,5 @@ case "${1:-}" in
event) event)
case "${3:-ok}" in bad|error) level=ERROR ;; warn) level=WARN ;; *) level=OK ;; esac case "${3:-ok}" in bad|error) level=ERROR ;; warn) level=WARN ;; *) level=OK ;; esac
{ printf '%s owrt-remote: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$2"; cat; } | log_append "$level" ;; { printf '%s owrt-remote: %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$2"; cat; } | log_append "$level" ;;
*) printf 'Usage: logs.sh collect|read [live]|clear|append|event MESSAGE\n' >&2; exit 2 ;; *) printf 'Usage: logs.sh collect|read [live]|clear|append|event MESSAGE|status|flush [force]\n' >&2; exit 2 ;;
esac esac
+156 -12
View File
@@ -951,6 +951,13 @@ post_json() (
die "для heartbeat нужен uclient-fetch, wget или curl" die "для heartbeat нужен uclient-fetch, wget или curl"
) )
log_post_json_bounded() (
# Also bounds remote_curl_post's private response body. The subshell limit
# cannot affect heartbeat/Xray. POSIX -f uses 512-byte blocks (32 KiB here).
ulimit -f 64 || exit 1
post_json "$1" "$2" "$3"
)
global_logs_state() { global_logs_state() {
local enabled xray local enabled xray
enabled=false; xray=stopped enabled=false; xray=stopped
@@ -960,13 +967,17 @@ global_logs_state() {
} }
sync_global_logs() ( sync_global_logs() (
local helper router_id batch file response endpoint base token journal offset clear_token rc sync_lock owner attempt local helper router_id file response endpoint base token journal offset clear_token rc sync_lock owner attempt runtime TMPDIR stale
helper="${OWRT_REMOTE_LOG_HELPER:-/usr/lib/owrt-remote/logs.sh}" helper="${OWRT_REMOTE_LOG_HELPER:-/usr/lib/owrt-remote/logs.sh}"
[ -r "$helper" ] || return 0 [ -r "$helper" ] || return 0
token="$(uci_get hub_token '')"; [ -n "$token" ] || return 0 token="$(uci_get hub_token '')"; [ -n "$token" ] || return 0
# The journal is independent of Remote's enabled flag and Xray lifecycle. # The journal is independent of Remote's enabled flag and Xray lifecycle.
sh "$helper" check || return 1 sh "$helper" check || return 1
sync_lock="${OWRT_REMOTE_LOG_DIR:-/tmp/owrt-remote-logs}/sync-lock" runtime="${OWRT_REMOTE_LOG_RUNTIME:-/tmp/owrt-remote-log-runtime}"
# Older helpers create only their journal directory. An independent agent
# update still needs a private runtime directory for delivery and its lock.
(umask 077; mkdir -p "$runtime") || return 1
sync_lock="$runtime/sync-lock"
if ! mkdir "$sync_lock" 2>/dev/null; then if ! mkdir "$sync_lock" 2>/dev/null; then
# Give a sender time to write its PID, then recover an abandoned lock. # Give a sender time to write its PID, then recover an abandoned lock.
for attempt in 1 2 3 4 5 6 7 8 9 10; do for attempt in 1 2 3 4 5 6 7 8 9 10; do
@@ -975,23 +986,56 @@ sync_global_logs() (
done done
owner="$(cat "$sync_lock/pid" 2>/dev/null)" owner="$(cat "$sync_lock/pid" 2>/dev/null)"
case "$owner" in ''|*[!0-9]*) ;; *) kill -0 "$owner" 2>/dev/null && return 0 ;; esac case "$owner" in ''|*[!0-9]*) ;; *) kill -0 "$owner" 2>/dev/null && return 0 ;; esac
rm -f "$sync_lock/pid"; rmdir "$sync_lock" 2>/dev/null || return 0 if ! mkdir "$sync_lock/reap" 2>/dev/null; then
owner="$(cat "$sync_lock/reap/pid" 2>/dev/null)"
case "$owner" in ''|*[!0-9]*) sleep 1 ;; *) kill -0 "$owner" 2>/dev/null && return 0 ;; esac
rm -f "$sync_lock/reap/pid"; rmdir "$sync_lock/reap" 2>/dev/null || return 0
return 0
fi
printf '%s\n' "$$" >"$sync_lock/reap/pid"
owner="$(cat "$sync_lock/pid" 2>/dev/null)"
case "$owner" in ''|*[!0-9]*) ;; *) if kill -0 "$owner" 2>/dev/null; then rm -f "$sync_lock/reap/pid"; rmdir "$sync_lock/reap"; return 0; fi ;; esac
rm -f "$sync_lock/pid"
rm -f "$sync_lock/reap/pid"
rmdir "$sync_lock/reap" || return 0
rmdir "$sync_lock" 2>/dev/null || return 0
mkdir "$sync_lock" 2>/dev/null || return 0 mkdir "$sync_lock" 2>/dev/null || return 0
fi fi
# ash keeps $$ unchanged inside a subshell. Store the actual sender PID so
# a killed sender is recoverable while its parent service is still alive.
if read -r owner attempt </proc/self/stat 2>/dev/null; then
printf '%s\n' "$owner" >"$sync_lock/pid"
else
printf '%s\n' "$$" >"$sync_lock/pid" printf '%s\n' "$$" >"$sync_lock/pid"
fi
file=''; response='' file=''; response=''
trap 'rm -f "$file" "$response" "$sync_lock/pid"; rmdir "$sync_lock" 2>/dev/null || true' EXIT trap 'rm -f "$file" "$response" "$sync_lock/pid"; rmdir "$sync_lock" 2>/dev/null || true' EXIT
trap 'exit 1' INT TERM trap 'exit 1' INT TERM
# A killed sender can leave one bounded payload; reclaim it under the singleton.
rm -f "$runtime"/sync.*
# Give this sender its own temporary HTTP directory; never sweep heartbeat's
# files. Clean a curl subprocess killed before its EXIT trap could run.
TMPDIR="$runtime/http"; export TMPDIR
mkdir -p "$TMPDIR" || return 1
for stale in "$TMPDIR"/owrt-remote-http.*; do
[ -d "$stale" ] || continue
rm -f "$stale/body" "$stale/stats" "$stale/error" "$stale/dns"
rmdir "$stale" 2>/dev/null || true
done
if truthy "$(uci_get wan_direct 0)" && [ ! -s "$WAN_STATE/active-interface" ]; then wan_prepare || return 1; fi if truthy "$(uci_get wan_direct 0)" && [ ! -s "$WAN_STATE/active-interface" ]; then wan_prepare || return 1; fi
batch="$(sh "$helper" export)" || return 1
router_id="$(safe_id "$(uci_get router_id main)")" router_id="$(safe_id "$(uci_get router_id main)")"
# export verifies that this directory is on tmpfs; payloads also stay in RAM. # Export at most 16 KiB of raw records directly into a bounded request file.
# Neither the Flash journal nor a base64 packet is copied into a shell variable.
umask 077 umask 077
file="${OWRT_REMOTE_LOG_DIR:-/tmp/owrt-remote-logs}/sync.$$"; response="$file.response" file="$runtime/sync.$$"; response="$file.response"
printf '{"id":%s,"batch":%s,"state":%s}\n' "$(json_string "$router_id")" "$batch" "$(global_logs_state)" >"$file" {
printf '{"id":%s,"batch":' "$(json_string "$router_id")"
sh "$helper" export || exit 1
printf ',"state":%s}\n' "$(global_logs_state)"
} >"$file" || return 1
for endpoint in $(heartbeat_candidate_endpoints); do for endpoint in $(heartbeat_candidate_endpoints); do
base="$(heartbeat_base_from_endpoint "$endpoint")" base="$(heartbeat_base_from_endpoint "$endpoint")"
if post_json "$base/api/router-logs" "$token" "$file" >"$response" 2>/dev/null && if log_post_json_bounded "$base/api/router-logs" "$token" "$file" >"$response" 2>/dev/null &&
grep -Eq '"ok"[[:space:]]*:[[:space:]]*true' "$response"; then grep -Eq '"ok"[[:space:]]*:[[:space:]]*true' "$response"; then
journal="$(jsonfilter -i "$response" -e '@.journal_id' 2>/dev/null)" journal="$(jsonfilter -i "$response" -e '@.journal_id' 2>/dev/null)"
offset="$(jsonfilter -i "$response" -e '@.offset' 2>/dev/null)" offset="$(jsonfilter -i "$response" -e '@.offset' 2>/dev/null)"
@@ -1136,8 +1180,9 @@ heartbeat_diagnostics() {
if [ -r "$state" ]; then cat "$state"; else printf 'Замеров пока нет; выполни owrt-remote heartbeat.\n'; fi if [ -r "$state" ]; then cat "$state"; else printf 'Замеров пока нет; выполни owrt-remote heartbeat.\n'; fi
} }
send_heartbeat() ( send_heartbeat() {
local hub_url token endpoint tmp response event used fallback_base errors rc dir metrics http attempt max_attempts reason previous previous_reason safe_endpoint elapsed started diagnostic local hub_url token endpoint tmp response event used fallback_base errors rc dir metrics http attempt max_attempts reason previous previous_reason safe_endpoint elapsed started diagnostic
(
event="${1:-}" event="${1:-}"
hub_url="$(uci_get hub_url '')" hub_url="$(uci_get hub_url '')"
token="$(uci_get hub_token '')" token="$(uci_get hub_token '')"
@@ -1210,19 +1255,96 @@ send_heartbeat() (
[ "$used" = "1" ] || die "hub_url/public_url/vps_host пустые" [ "$used" = "1" ] || die "hub_url/public_url/vps_host пустые"
case "$hub_url" in https://*) log 'Heartbeat: fallback остановлен; reason=https_required; автоматический переход на HTTP отключён' ;; esac case "$hub_url" in https://*) log 'Heartbeat: fallback остановлен; reason=https_required; автоматический переход на HTTP отключён' ;; esac
return 1 return 1
) )
}
wan_reconcile_xray() { wan_reconcile_xray() {
local config bound device local config bound device
truthy "$(uci_get enabled 0)" || return 0
device="$(wan_device)" || return 1
if ! xray_running; then
wan_recover_xray
return $?
fi
config="$(uci_get xray_config /etc/xray/owrt-remote-client.json)" config="$(uci_get xray_config /etc/xray/owrt-remote-client.json)"
[ -r "$config" ] || return 0 [ -r "$config" ] || return 0
bound="$(jsonfilter -i "$config" -e '@.outbounds[@.tag="vps-interconn"].streamSettings.sockopt.interface' 2>/dev/null)" bound="$(jsonfilter -i "$config" -e '@.outbounds[@.tag="vps-interconn"].streamSettings.sockopt.interface' 2>/dev/null)"
device="$(wan_device)" || return 1
[ "$bound" != "$device" ] || return 0 [ "$bound" != "$device" ] || return 0
xray_running || return 0
wan_reconnect_xray "$bound" "$device" wan_reconnect_xray "$bound" "$device"
} }
wan_recover_xray() {
local lock owner rest state running heartbeat now last attempt
(
umask 077
lock="${TMPDIR:-/tmp}/owrt-remote-xray-recovery"
# A stopped procd instance cannot be revived by sending it a signal.
# Serialize re-registration; do not interrupt the healthy heartbeat instance.
attempt=0
while ! mkdir "$lock" 2>/dev/null; do
owner="$(cat "$lock/pid" 2>/dev/null)"
case "$owner" in
''|*[!0-9]*)
# Allow a new worker to record its PID before reaping an empty lock.
if [ "$attempt" -lt 10 ]; then attempt=$((attempt+1)); sleep 0.1; continue; fi ;;
*) kill -0 "$owner" 2>/dev/null && return 0 ;;
esac
wan_recovery_reap "$lock" "$attempt"
attempt=$((attempt+1)); [ "$attempt" -lt 20 ] || return 0
done
trap 'rm -f "$lock/pid"; rmdir "$lock" 2>/dev/null || true' EXIT
trap 'exit 1' INT TERM
owner=$$
# ash keeps $$ unchanged in a subshell; /proc/self identifies this worker.
if [ -r /proc/self/stat ]; then read -r owner rest </proc/self/stat || owner=$$; fi
printf '%s\n' "$owner" >"$lock/pid"
truthy "$(uci_get enabled 0)" || return 0
wan_select_interface >/dev/null || return 0
xray_running && return 0
state="$(ubus call service list '{"name":"owrt-remote"}' 2>/dev/null)" || return 1
running="$(printf '%s' "$state" | jsonfilter -e '@["owrt-remote"].instances.xray.running' 2>/dev/null)"
[ "$running" != true ] || return 0
heartbeat="$(printf '%s' "$state" | jsonfilter -e '@["owrt-remote"].instances.heartbeat.running' 2>/dev/null)"
# Respect an explicit service stop, including stops made outside the panel.
[ "$heartbeat" = true ] || return 0
now="$(uptime_seconds)"
now="$(safe_num "$now" 0)"
[ "$now" -gt 0 ] || now="$(safe_num "$(date +%s 2>/dev/null)" 0)"
last="$(safe_num "$(cat "${TMPDIR:-/tmp}/owrt-remote-xray-recovery.last" 2>/dev/null)" 0)"
if [ "$last" -gt 0 ] && [ "$now" -ge "$last" ] && [ $((now-last)) -lt 300 ]; then return 0; fi
printf '%s\n' "$now" >"${TMPDIR:-/tmp}/owrt-remote-xray-recovery.last"
log 'Xray: WAN доступен, туннель остановлен; запрошен повторный запуск отдельного instance'
ubus call service add '{"name":"owrt-remote","instances":{"xray":{"command":["/usr/sbin/owrt-remote","run"],"env":{"OWRT_REMOTE_SERVICE_LOG":"1"},"respawn":["3600","5","5"],"stdout":true,"stderr":true}}}' >/dev/null || {
log 'ОШИБКА: procd не принял запрос восстановления Xray; Heartbeat продолжает работать'
return 1
}
)
}
wan_recovery_reap() {
local owner reaper rest
# The reaper itself must be serialized and recoverable after SIGKILL.
if ! mkdir "$1/reap" 2>/dev/null; then
reaper="$(cat "$1/reap/pid" 2>/dev/null)"
case "$reaper" in
''|*[!0-9]*) [ "$2" -ge 15 ] || return 0 ;;
*) kill -0 "$reaper" 2>/dev/null && return 0 ;;
esac
rm -f "$1/reap/pid"; rmdir "$1/reap" 2>/dev/null || return 0
return 0
fi
reaper=$$
if [ -r /proc/self/stat ]; then read -r reaper rest </proc/self/stat || reaper=$$; fi
printf '%s\n' "$reaper" >"$1/reap/pid"
owner="$(cat "$1/pid" 2>/dev/null)"
case "$owner" in
''|*[!0-9]*) ;;
*) if kill -0 "$owner" 2>/dev/null; then rm -f "$1/reap/pid"; rmdir "$1/reap"; return 0; fi ;;
esac
rm -f "$1/pid" "$1/reap/pid"
rmdir "$1/reap" 2>/dev/null && rmdir "$1" 2>/dev/null || true
}
wan_reconnect_xray() { wan_reconnect_xray() {
local restart_lock owner local restart_lock owner
restart_lock="${TMPDIR:-/tmp}/owrt-remote-wan-restart" restart_lock="${TMPDIR:-/tmp}/owrt-remote-wan-restart"
@@ -1265,6 +1387,10 @@ heartbeat_self_heal() {
printf '%s\n' "$fails" >"$fail_file" printf '%s\n' "$fails" >"$fail_file"
[ "$fails" -lt "$threshold" ] && return 0 [ "$fails" -lt "$threshold" ] && return 0
if truthy "$(uci_get wan_direct 0)" && ! wan_select_interface >/dev/null; then
log 'Heartbeat: перезапуск отложен; ни один выбранный WAN не имеет физического IPv4 default route'
return 0
fi
if active_ssh_session; then if active_ssh_session; then
log "heartbeat failed ${threshold} times, restart postponed because SSH session is active" log "heartbeat failed ${threshold} times, restart postponed because SSH session is active"
return 0 return 0
@@ -1535,11 +1661,29 @@ ensure_xray_available_with_retry() {
return 1 return 1
} }
wait_for_selected_wan() {
local waiting
waiting=0
while truthy "$(uci_get wan_direct 0)"; do
truthy "$(uci_get enabled 0)" || return 1
if wan_select_interface >/dev/null; then
[ "$waiting" = 0 ] || log 'Xray: выбранный WAN снова доступен; продолжается запуск туннеля'
return 0
fi
[ "$waiting" = 1 ] || log 'Xray: ожидание выбранного WAN с физическим IPv4 default route; конфигурация сохранена'
waiting=1
sleep 5
done
}
run_xray() { run_xray() {
local enabled xray_bin xray_config local enabled xray_bin xray_config
enabled="$(uci_get enabled 0)" enabled="$(uci_get enabled 0)"
truthy "$enabled" || die "service disabled: set option enabled '1'" truthy "$enabled" || die "service disabled: set option enabled '1'"
# Missing WAN is a temporary condition, not a failed Xray configuration.
# Stay supervised instead of exhausting procd's six startup attempts.
wait_for_selected_wan || return 1
ensure_xray_available_with_retry || die "cannot prepare Xray for tunnel" ensure_xray_available_with_retry || die "cannot prepare Xray for tunnel"
xray_bin="$(xray_bin_path)" xray_bin="$(xray_bin_path)"
xray_config="$(render_client_config)" || die "cannot create Xray configuration" xray_config="$(render_client_config)" || die "cannot create Xray configuration"
+4 -4
View File
@@ -147,7 +147,7 @@ logs_response() {
fi fi
if ! sh "$LOG_HELPER" check >/dev/null 2>&1; then if ! sh "$LOG_HELPER" check >/dev/null 2>&1; then
printf 'Status: 503 Service Unavailable\r\nContent-Type: text/plain; charset=utf-8\r\nCache-Control: no-store\r\n\r\n' printf 'Status: 503 Service Unavailable\r\nContent-Type: text/plain; charset=utf-8\r\nCache-Control: no-store\r\n\r\n'
printf 'Журнал недоступен: для логов требуется ОЗУ (tmpfs). Запись на Flash запрещена.\n' printf 'Журнал временно недоступен: не удалось подготовить хранилище или получить блокировку. Настройки роутера не изменены.\n'
return return
fi fi
if [ "$mode" = clear ]; then if [ "$mode" = clear ]; then
@@ -505,7 +505,7 @@ render_page() {
<meta charset="utf-8"> <meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="owrt-remote-ui-version" content="$OWRT_REMOTE_UI_VERSION"> <meta name="owrt-remote-ui-version" content="$OWRT_REMOTE_UI_VERSION">
<title>OpenWrt Remote Hub v116</title> <title>OpenWrt Remote Hub v117</title>
<style> <style>
:root{color-scheme:dark;--bg:#07040f;--panel:rgba(19,14,32,.88);--text:#f7f2ff;--muted:#b9adc9;--line:rgba(169,126,255,.24);--blue:#7c3aed;--green:#22c55e;--red:#fb7185;--amber:#f59e0b;--teal:#a855f7;--grid:rgba(168,85,247,.16)} :root{color-scheme:dark;--bg:#07040f;--panel:rgba(19,14,32,.88);--text:#f7f2ff;--muted:#b9adc9;--line:rgba(169,126,255,.24);--blue:#7c3aed;--green:#22c55e;--red:#fb7185;--amber:#f59e0b;--teal:#a855f7;--grid:rgba(168,85,247,.16)}
*{box-sizing:border-box}body{position:relative;min-height:100vh;margin:0;overflow-x:hidden;background-color:var(--bg);background-image:radial-gradient(circle at 12% 8%,rgba(168,85,247,.46),transparent 31%),radial-gradient(circle at 82% 12%,rgba(79,70,229,.38),transparent 30%),radial-gradient(circle at 50% 105%,rgba(236,72,153,.26),transparent 35%),linear-gradient(145deg,#07040f,#120a24 48%,#05030a),repeating-linear-gradient(0deg,transparent 0 30px,var(--grid) 31px),repeating-linear-gradient(90deg,transparent 0 30px,var(--grid) 31px);background-size:130% 130%,140% 140%,135% 135%,100% 100%,31px 31px,31px 31px;background-attachment:fixed;color:var(--text);font:14px/1.45 system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;animation:bgFlow 22s ease-in-out infinite alternate} *{box-sizing:border-box}body{position:relative;min-height:100vh;margin:0;overflow-x:hidden;background-color:var(--bg);background-image:radial-gradient(circle at 12% 8%,rgba(168,85,247,.46),transparent 31%),radial-gradient(circle at 82% 12%,rgba(79,70,229,.38),transparent 30%),radial-gradient(circle at 50% 105%,rgba(236,72,153,.26),transparent 35%),linear-gradient(145deg,#07040f,#120a24 48%,#05030a),repeating-linear-gradient(0deg,transparent 0 30px,var(--grid) 31px),repeating-linear-gradient(90deg,transparent 0 30px,var(--grid) 31px);background-size:130% 130%,140% 140%,135% 135%,100% 100%,31px 31px,31px 31px;background-attachment:fixed;color:var(--text);font:14px/1.45 system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;animation:bgFlow 22s ease-in-out infinite alternate}
@@ -540,7 +540,7 @@ form{margin:0}.formgrid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr
<div class="brand"> <div class="brand">
<div class="brandPanel"> <div class="brandPanel">
<div class="titleRow"> <div class="titleRow">
<h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v116</span></span></h1> <h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v117</span></span></h1>
</div> </div>
<div class="links"> <div class="links">
<a class="uiPill" href="https://t.me/kzolotarev95" target="_blank" rel="noopener noreferrer">Telegram</a> <a class="uiPill" href="https://t.me/kzolotarev95" target="_blank" rel="noopener noreferrer">Telegram</a>
@@ -564,7 +564,7 @@ form{margin:0}.formgrid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr
<section class="panel globalLogsPanel" id="globalLogsPanel" hidden> <section class="panel globalLogsPanel" id="globalLogsPanel" hidden>
<div class="panelHead"><h2>Глобал логи</h2><button class="btn" id="globalLogsClose" type="button">Закрыть</button></div> <div class="panelHead"><h2>Глобал логи</h2><button class="btn" id="globalLogsClose" type="button">Закрыть</button></div>
<blockquote class="globalLogsNote">Действия панели, изменения связи с Hub и WAN, предупреждения и ошибки Xray. Одинаковые события не дублируются. Журнал хранится только в ОЗУ, очищается после перезагрузки и каждые 24 часа. Максимум около 2 МБ. Ключи и пароли скрываются.</blockquote> <blockquote class="globalLogsNote">Действия панели, Heartbeat, WAN и ошибки Xray. Журнал сохраняется на проверенном Flash-хранилище: до 2 МиБ, до 256 фрагментов; доставленные записи хранятся до 24 часов. Запись пакетами раз в минуту; последние 16 КиБ буфера могут потеряться при внезапном отключении питания. При недоступной Flash — ограниченный журнал в ОЗУ с предупреждением. Переполнение не удаляет неподтверждённые записи: новые события сверх лимита пропускаются. Ключи и пароли скрываются.</blockquote>
<div class="globalLogsLegend" aria-label="Цвета журнала"><span class="globalLogOk">● Успешно</span><span class="globalLogWarn">● Предупреждение</span><span class="globalLogError">● Ошибка</span></div> <div class="globalLogsLegend" aria-label="Цвета журнала"><span class="globalLogOk">● Успешно</span><span class="globalLogWarn">● Предупреждение</span><span class="globalLogError">● Ошибка</span></div>
<div class="actions"> <div class="actions">
<a class="btn" href="?key=$(h "$key")&amp;action=logs_download">Скачать логи</a> <a class="btn" href="?key=$(h "$key")&amp;action=logs_download">Скачать логи</a>
@@ -14,7 +14,7 @@ return view.extend({
if (!key) { if (!key) {
return E('div', { 'class': 'cbi-map' }, [ return E('div', { 'class': 'cbi-map' }, [
E('h2', {}, _('OpenWrt Remote Hub v116')), E('h2', {}, _('OpenWrt Remote Hub v117')),
E('div', { 'class': 'alert-message warning' }, [ E('div', { 'class': 'alert-message warning' }, [
_('Web key not found. Reinstall the module or create /etc/owrt-remote/web.key.') _('Web key not found. Reinstall the module or create /etc/owrt-remote/web.key.')
]) ])
@@ -34,7 +34,7 @@ return view.extend({
}, 100); }, 100);
return E('div', { 'class': 'cbi-map' }, [ return E('div', { 'class': 'cbi-map' }, [
E('h2', {}, _('OpenWrt Remote Hub v116')), E('h2', {}, _('OpenWrt Remote Hub v117')),
E('p', {}, _('Opening remote access panel...')), E('p', {}, _('Opening remote access panel...')),
E('p', {}, [ E('p', {}, [
E('a', { 'class': 'btn cbi-button cbi-button-apply', 'href': url }, _('Open panel')) E('a', { 'class': 'btn cbi-button cbi-button-apply', 'href': url }, _('Open panel'))
+6
View File
@@ -214,6 +214,12 @@ install_xray_runtime() {
"$remote_bin" install-xray-tmp || die "failed to install Xray to /tmp" "$remote_bin" install-xray-tmp || die "failed to install Xray to /tmp"
} }
# Stop the old log writer before replacing its storage code. Its bounded legacy
# archive is migrated by the new log service, without moving any other /tmp data.
if [ "${ROOT%/}" = "" ] && [ -x /etc/init.d/owrt-remote-logs ]; then
/etc/init.d/owrt-remote-logs stop || true
fi
install_file "usr/sbin/owrt-remote" 0755 install_file "usr/sbin/owrt-remote" 0755
install_file "usr/lib/owrt-remote/wan.sh" 0644 install_file "usr/lib/owrt-remote/wan.sh" 0644
install_file "usr/lib/owrt-remote/logs.sh" 0644 install_file "usr/lib/owrt-remote/logs.sh" 0644
+2 -2
View File
@@ -1,4 +1,4 @@
# Heartbeat: HTTPS, авторизация и диагностика (v116) # Heartbeat: HTTPS, авторизация и диагностика (v117)
## Разбор сбоя 15.10.2026 ## Разбор сбоя 15.10.2026
@@ -82,7 +82,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Обновляются агент и WAN helper. Перезапускается только экземпляр Heartbeat Обновляются агент и WAN helper. Перезапускается только экземпляр Heartbeat
под procd; Xray и удалённый туннель скрипт не перезапускает. Остановленный под procd; Xray и удалённый туннель скрипт не перезапускает. Остановленный
экземпляр остаётся остановленным. Резервная копия первых файлов сохраняется экземпляр остаётся остановленным. Резервная копия первых файлов сохраняется
в `/root/owrt-remote-backup-before-heartbeat-transport-v116`. в `/root/owrt-remote-backup-before-heartbeat-transport-v117`.
Маркер `HEARTBEAT_TRANSPORT_ROUTER_Install_OK` подтверждает установку кода, Маркер `HEARTBEAT_TRANSPORT_ROUTER_Install_OK` подтверждает установку кода,
а доступность Hub проверяется отдельно: а доступность Hub проверяется отдельно:
+12 -12
View File
@@ -1,4 +1,4 @@
# Фиксация VPS host и независимое соединение Remote через WAN (v116) # Фиксация VPS host и независимое соединение Remote через WAN (v117)
`vps_host_mode=manual` сохраняет заданный на роутере адрес. Это значение `vps_host_mode=manual` сохраняет заданный на роутере адрес. Это значение
используется и когда опция отсутствует у старой установки. Только явно используется и когда опция отсутствует у старой установки. Только явно
@@ -42,8 +42,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-hub-url-fix.
``` ```
Откат: те же команды с `-Action Rollback`, без `-HubUrl`. Резервные копии: Откат: те же команды с `-Action Rollback`, без `-HubUrl`. Резервные копии:
`/root/owrt-remote-backup-before-hub-url-v116` на роутере и `/root/owrt-remote-backup-before-hub-url-v117` на роутере и
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v116` на VPS. `/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v117` на VPS.
`HEARTBEAT_OK` подтверждает ответ Hub. Если файлы установлены, но Hub недоступен, `HEARTBEAT_OK` подтверждает ответ Hub. Если файлы установлены, но Hub недоступен,
скрипт отдельно сообщает `HEARTBEAT_FAILED`; это не подтверждение связи. скрипт отдельно сообщает `HEARTBEAT_FAILED`; это не подтверждение связи.
@@ -125,21 +125,21 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.
Установка загружает локальные файлы, без Git. До остановки службы проверяются Установка загружает локальные файлы, без Git. До остановки службы проверяются
доступность WAN и поддержка таблицы реальной командой ip на роутере. доступность WAN и поддержка таблицы реальной командой ip на роутере.
На роутер загружается и файл LuCI: версия v116 появляется и в основной На роутер загружается и файл LuCI: версия v117 появляется и в основной
CGI-панели, и в странице входа в неё через LuCI. CGI-панели, и в странице входа в неё через LuCI.
Сначала сохраняются файлы Сначала сохраняются файлы
и конфигурация; после замены проверяется heartbeat/health. При ошибке и конфигурация; после замены проверяется heartbeat/health. При ошибке
автоматически восстанавливается состояние перед текущей попыткой. автоматически восстанавливается состояние перед текущей попыткой.
Сохранённая резервная копия для ручного отката не перезаписывается. Сохранённая резервная копия для ручного отката не перезаписывается.
* Роутер: `/root/owrt-remote-backup-before-host-policy-v116` — агент, * Роутер: `/root/owrt-remote-backup-before-host-policy-v117` — агент,
helper, init, CGI, UCI-конфигурация и Xray-конфиг (в закрытом каталоге). helper, init, CGI, UCI-конфигурация и Xray-конфиг (в закрытом каталоге).
* VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v116`. * VPS: `/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v117`.
База роутеров и настройки авторизации не заменяются. База роутеров и настройки авторизации не заменяются.
Успешная установка заканчивается `HOST_POLICY_ROUTER_V116_OK` на роутере Успешная установка заканчивается `HOST_POLICY_ROUTER_V117_OK` на роутере
и `HOST_POLICY_VPS_V116_Install_OK` на VPS. Старые резервные копии предыдущей и `HOST_POLICY_VPS_V117_Install_OK` на VPS. Старые резервные копии предыдущей
версии сохраняются; новые команды отката используют отдельные копии для v116. версии сохраняются; новые команды отката используют отдельные копии для v117.
### Переход с nftables-маркировки на адресную маршрутизацию ### Переход с nftables-маркировки на адресную маршрутизацию
@@ -148,11 +148,11 @@ CGI-панели, и в странице входа в неё через LuCI.
Обновление VPS для этого изменения не требуется. Обновление VPS для этого изменения не требуется.
```powershell ```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v116" -Action Install powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -PreserveConnectionSettings -BackupName "podkop-compat-v117" -Action Install
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v116" -Action Rollback powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-host-policy.ps1" -Target Router -Machine "root@192.168.2.1" -BackupName "podkop-compat-v117" -Action Rollback
``` ```
Копия: `/root/owrt-remote-backup-before-podkop-compat-v116`. Копия: `/root/owrt-remote-backup-before-podkop-compat-v117`.
После установки повторите диагностику Podkop. Предупреждение от прежних правил После установки повторите диагностику Podkop. Предупреждение от прежних правил
Remote должно исчезнуть. Настоящие дополнительные правила других приложений Remote должно исчезнуть. Настоящие дополнительные правила других приложений
по-прежнему обнаруживаются штатной диагностикой. по-прежнему обнаруживаются штатной диагностикой.
+155
View File
@@ -0,0 +1,155 @@
# Persistent client journal (v117 local fix)
The helper is `/usr/lib/owrt-remote/logs.sh`. It changes only Remote's private
journal, not logd, network configuration, policy routing or Xray configuration.
## Storage and limits
- Persistent: normally `/overlay/owrt-remote/logs`, only when `/overlay` is an
actual writable JFFS2/UBIFS/ext4/f2fs/ext2/ext3/btrfs mount. A directly mounted
persistent root uses `/etc/owrt-remote/logs`. Root OverlayFS and tmpfs are
rejected. Existing ancestors/symlinks are resolved against `/proc/mounts`;
the longest containing mount wins. An explicit `OWRT_REMOTE_LOG_DIR` is
validated the same way.
- Runtime: `/tmp/owrt-remote-log-runtime`, which must be on tmpfs. Contains a
buffer <=16 KiB, <=64 hashed dedup entries, a heartbeat fingerprint, PID locks,
one export <=16 KiB, one HTTP body about 22 KiB, one viewer snapshot <=16 KiB
plus a bounded filename list. No whole-journal shell variable or RAM archive.
HTTP response files are limited by a sender-only RLIMIT_FSIZE (32 KiB in
BusyBox ash); this does not alter heartbeat or Xray process limits.
- Flash data: <=2 MiB across <=256 immutable chunks, each <=16 KiB. This is an
upper bound, not guaranteed capacity: many small chunks reach the file-count
bound first. Metadata, filesystem allocation units and the one quarantined
damaged chunk add small overhead beyond the raw-data bound.
- Rotation runs each minute. Acknowledged chunks expire after 24 hours or
earlier to admit new chunks. Unacknowledged chunks do not expire. If the
bounded queue/RAM buffer fills, **new** events are skipped; existing
unacknowledged chunks remain intact. A warning and dropped-event count make
that loss visible. A bounded queue cannot preserve infinitely many events
during an infinitely long outage.
- Default batch flush: once per 60 seconds, at most 16 KiB of new data; explicit
`flush force`, normal service stop and shutdown can flush sooner. There is no
fsync per line. Idle polls do not rewrite persistent state. Metadata and
filesystem write amplification are additional to the data rate; no universal
Flash-lifetime guarantee is possible across devices/filesystems.
- Less than 2 MiB free space or write/mount failure stops Flash writes and
switches this boot to a warning-marked RAM fallback: <=64 KiB data in <=8
chunks +16 KiB buffer. The old Flash queue is left intact. Restore storage
and reboot to resume that queue. No automatic mid-boot merging of two journal
identities, which could acknowledge the wrong records.
## Crash and synchronization rules
Each chunk filename includes journal identity, start/end byte offsets and a validated SHA-256/MD5/POSIX cksum digest. Data is written to `chunk.new` on the persistent filesystem and renamed;
state is written to `state.new` on that filesystem and atomically replaced.
Batch sync (`sync -f`, or BusyBox `sync` fallback) completes durability. A small
RAM in-flight marker reconciles a killed flusher before another append. At
boot, checksums reject partial/corrupt chunks, incomplete temporary files are
removed, and a completed rename beyond the saved cursor is recovered. One
corrupt chunk can be retained as `damaged.log` for inspection. A reboot never
clears a valid journal.
Last-minute events still in RAM can be lost on sudden power removal. This is
the explicit buffering tradeoff, not a claim of per-line durable logging.
Only a successful authenticated Hub reply advances the ack. The recent ack is
checkpointed with the next periodic batch before pruning; if power is lost
first, records may replay with the same journal/offset. Existing Hub ingestion
suppresses these duplicates. Packets are <=16 KiB raw and transport them from
files without an entire packet in a shell variable. One `logs-sync` sender is
allowed; stale locks and stale payloads recover on the next attempt. Explicit
local/Hub clear intentionally discards both confirmed and unconfirmed data.
Log-sender curl temporaries use a private runtime subdirectory, reclaimed on
the next attempt after SIGKILL; heartbeat's temporary directory is untouched.
The local installer replaces the helper, log service, agent, CGI and LuCI view. It
does not restart the tunnel, heartbeat or network. With the previous collector
stopped, it streams the old bounded RAM journal into Flash chunks, preserving
its journal ID, total cursor, ack and clear token. Already delivered bytes are
not resent; undelivered records remain eligible for delivery. No FIFO, lock or
temporary request is migrated. A same-filesystem staging manifest allows a
power-interrupted migration to finish even after the old /tmp has disappeared.
Migration needs 6 MiB free Flash; if it cannot safely migrate legacy data, the
local installer fails and restores code instead of silently deleting history.
With no usable Flash, only a bounded legacy tail is retained in RAM with an
explicit loss warning, and the former large RAM files are removed. An ordinary
upgrade stops the previous collector before replacing files and invokes this
migration on log-service startup. Rollback restores code/service state and
leaves Flash logs intact for recovery; the old helper starts a fresh RAM log.
## Automated checks
```powershell
$env:OWRT_LOG_TEST_SHELL = "$PWD\.test-output\busybox\busybox64.exe"
python -m unittest discover -s tests -p test_persistent_logs.py -v
Remove-Item Env:OWRT_LOG_TEST_SHELL
python -m unittest discover -s tests -p test_global_logs.py -v
python -m unittest discover -s tests -p test_router_log_transport.py -v
```
Tests use real shell functions and disposable mount/filesystem fixtures. The
BusyBox Windows build exercises ash and its applets; it is **not** a claim of
testing OpenWrt 24.10.3's exact BusyBox/kernel/UBIFS or physical power loss.
## Device checks before release
Install locally, then inspect from Windows PowerShell:
```powershell
ssh root@192.168.2.1 'sh /usr/lib/owrt-remote/logs.sh status; df -h /overlay /tmp; free; ubus call service list; owrt-remote heartbeat-diagnostics'
```
Expect `persistent=1`, a persistent storage path and tmpfs runtime. Confirm the
mount explicitly using `mount` and verify `state` and `part.*` appear on Flash.
Append a marker and flush before a user-planned reboot:
```powershell
ssh root@192.168.2.1 'printf "persistent-check\n" | sh /usr/lib/owrt-remote/logs.sh append; sh /usr/lib/owrt-remote/logs.sh flush force; sh /usr/lib/owrt-remote/logs.sh status'
```
After reboot, `read` must still show the marker and the same journal ID. Test an
outage only on a controlled router with a separate management path; do not
change firewall/WAN settings on a remote production router merely for this
test. During outage, watch data/chunk/runtime bounds; after reconnect, watch
`ack_bytes` advance without duplicate Hub entries. Launch concurrent
`owrt-remote logs-sync` calls and inspect service/process counts. Confirm UI,
heartbeat, Xray and WAN direct remain functional. Actual sustained RAM,
Flash-write and reboot observations must be collected on the device.
## VPS archive and malformed-fragment recovery
The Hub retains its existing persistent SQLite archive, normally at
`/var/lib/owrt-remote/hub.db`, <=2 MiB of content per router. No new RAM archive
or whole-journal queue is introduced on the VPS. The live view still transfers
only the last 64 KiB. A storage diagnostic resolves DB symlinks and the longest
containing Linux mount, marking tmpfs/ramfs as nonpersistent. The dedicated
`deploy-persistent-logs-vps.ps1` verifies the running Hub's actual DB path and
writable disk before installing code. It preserves the database and all settings;
rollback replaces code only. It refuses installation over an unconfirmed/RAM DB
rather than automatically relocating a live database.
Some OpenWrt builds lack the optional `cksum` applet. The previous local helper
failed to check its exit status and could publish names ending in an empty digest.
Export skipped these files, but the empty status packet claimed the full cursor;
the previous Hub accepted it and falsely acknowledged missing data. Regression
checks now simulate missing cksum and failed checksum tools. New chunks use an
available SHA-256/MD5/POSIX checksum with validated output. An empty status packet
cannot advance the delivered cursor on either side.
On upgrade, a versioned recovery marker forces inspection even during the same
boot. Retained malformed fragments of the expected byte length are copied into a
same-filesystem recovery generation with real digests and a committed migration
manifest. Old falsely acknowledged offsets cannot suppress that generation.
Recovery requires enough free disk for the bounded copy and retains originals
until the new manifest is committed. This recovers retained bytes, not already
removed files, and cannot retroactively verify their former integrity. A warning
records that limitation. A new generation can repeat already delivered history
if mixed with affected fragments; normal retries remain deduplicated by offsets.
Run the real client spool-to-Hub ingestion regression as well as storage/deploy
checks:
```powershell
python -m unittest discover -s tests -p test_router_log_archive.py -v
python -m unittest discover -s tests -p test_persistent_logs_vps_deploy.py -v
```
+3 -3
View File
@@ -1,4 +1,4 @@
# reCAPTCHA v3 в OpenWrt Remote Hub v116 # reCAPTCHA v3 в OpenWrt Remote Hub v117
В меню владельца, в блоке «Капча на экране входа», доступны цифры, В меню владельца, в блоке «Капча на экране входа», доступны цифры,
Google reCAPTCHA v2 Checkbox и Google reCAPTCHA v3. Установка обновления Google reCAPTCHA v2 Checkbox и Google reCAPTCHA v3. Установка обновления
@@ -74,8 +74,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-recaptcha-v3
непосредственно перед попыткой. Резервные копии при повторном запуске непосредственно перед попыткой. Резервные копии при повторном запуске
не перезаписываются: не перезаписываются:
* `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v116` * `/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v117`
* `/opt/owrt-remote/captcha-before-recaptcha-v3-v116.json` (режим 0600). * `/opt/owrt-remote/captcha-before-recaptcha-v3-v117.json` (режим 0600).
Скрипт рассчитан на стандартные пути установленного Hub: Скрипт рассчитан на стандартные пути установленного Hub:
`/opt/owrt-remote/owrt-remote-hub.py`, `/var/lib/owrt-remote/hub-auth.json` `/opt/owrt-remote/owrt-remote-hub.py`, `/var/lib/owrt-remote/hub-auth.json`
+3 -3
View File
@@ -1,6 +1,6 @@
# Аудит ресурсов OpenWrt Remote Hub v116 # Аудит ресурсов OpenWrt Remote Hub v117
Исправления внесены поверх `7f809fd`. Версия панели остаётся v116. Исправления внесены поверх `7f809fd`. Версия панели остаётся v117.
## SQLite ## SQLite
@@ -115,7 +115,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix
Установщик загружает локальный файл без Git, запускает изолированные тесты, Установщик загружает локальный файл без Git, запускает изолированные тесты,
сохраняет исходный серверный файл: сохраняет исходный серверный файл:
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v116`, `/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v117`,
атомарно заменяет Python файл, перезапускает `owrt-remote` и проверяет health. атомарно заменяет Python файл, перезапускает `owrt-remote` и проверяет health.
Существующая резервная копия не перезаписывается. При ошибке установки Существующая резервная копия не перезаписывается. При ошибке установки
восстанавливается файл, работавший непосредственно перед текущей попыткой. восстанавливается файл, работавший непосредственно перед текущей попыткой.
+4 -4
View File
@@ -1,4 +1,4 @@
# Виджеты ресурсов VPS (v116) # Виджеты ресурсов VPS (v117)
Три карточки ЦП, ОЗУ и ДИСК находятся над кнопками «Рестарт Xray VPS», Три карточки ЦП, ОЗУ и ДИСК находятся над кнопками «Рестарт Xray VPS»,
профилем и выходом. На ПК ширина и высота совпадают с кнопками под ними. профилем и выходом. На ПК ширина и высота совпадают с кнопками под ними.
@@ -23,7 +23,7 @@ vCPU и объёмы занятой/общей памяти и диска.
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-vps-widgets.ps1" -Vps "root@193.233.82.38" -Action Install powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-vps-widgets.ps1" -Vps "root@193.233.82.38" -Action Install
``` ```
После `VPS_WIDGETS_V116_Install_OK` обновите страницу через Ctrl+F5. После `VPS_WIDGETS_V117_Install_OK` обновите страницу через Ctrl+F5.
Скрипт загружает локальный файл, проверяет его до замены, перезапускает Скрипт загружает локальный файл, проверяет его до замены, перезапускает
службу и проверяет health. При ошибке после замены восстанавливает файл службу и проверяет health. При ошибке после замены восстанавливает файл
перед текущей попыткой. База данных и настройки не заменяются. перед текущей попыткой. База данных и настройки не заменяются.
@@ -33,8 +33,8 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
``` ```
Откат использует неизменяемую копию первого состояния до установки: Откат использует неизменяемую копию первого состояния до установки:
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v116`. `/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v117`.
Успешное завершение: `VPS_WIDGETS_V116_Rollback_OK`. Успешное завершение: `VPS_WIDGETS_V117_Rollback_OK`.
## Локальные проверки ## Локальные проверки
+2 -2
View File
@@ -1,4 +1,4 @@
# WAN IPv4 DNS fallback (local v116 fix) # WAN IPv4 DNS fallback (local v117 fix)
`files/usr/lib/owrt-remote/wan.sh` now explicitly queries `files/usr/lib/owrt-remote/wan.sh` now explicitly queries
`nslookup -type=A -timeout=3 -retry=1 "$host" "$dns"`. `nslookup -type=A -timeout=3 -retry=1 "$host" "$dns"`.
@@ -58,7 +58,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
``` ```
The installer saves the first backup at The installer saves the first backup at
`/root/owrt-remote-backup-before-wan-dns-v116/wan.sh`, checks shell syntax and `/root/owrt-remote-backup-before-wan-dns-v117/wan.sh`, checks shell syntax and
reloads only the supervised heartbeat instance so it reads the new helper. reloads only the supervised heartbeat instance so it reads the new helper.
It does not restart Xray or regenerate configuration/routing. Installation It does not restart Xray or regenerate configuration/routing. Installation
verification/reload failures restore the file from that attempt. verification/reload failures restore the file from that attempt.
+2 -2
View File
@@ -1,4 +1,4 @@
# Несколько WAN интерфейсов и резервирование (v116) # Несколько WAN интерфейсов и резервирование (v117)
Кнопка **Выбрать интерфейсы** раскрывает список с галочками. Можно отметить Кнопка **Выбрать интерфейсы** раскрывает список с галочками. Можно отметить
один, два, три и больше интерфейсов. Список содержит имена один, два, три и больше интерфейсов. Список содержит имена
@@ -46,7 +46,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Скрипт устанавливает CGI, LuCI, агент, WAN helper и hotplug вместе. До замены Скрипт устанавливает CGI, LuCI, агент, WAN helper и hotplug вместе. До замены
сохраняются прежние файлы, UCI-настройки и Xray config в сохраняются прежние файлы, UCI-настройки и Xray config в
`/root/owrt-remote-backup-before-wan-multi-v116`. `/root/owrt-remote-backup-before-wan-multi-v117`.
Успех подтверждает `WAN_MULTI_ROUTER_Install_OK`. После установки обновить Успех подтверждает `WAN_MULTI_ROUTER_Install_OK`. После установки обновить
панель роутера через Ctrl+F5, отметить интерфейсы и сохранить выбор. панель роутера через Ctrl+F5, отметить интерфейсы и сохранить выбор.
VPS обновлять не требуется. Старый `deploy-wan-interface-picker.ps1` VPS обновлять не требуется. Старый `deploy-wan-interface-picker.ps1`
+96
View File
@@ -0,0 +1,96 @@
# Восстановление Xray после пропажи выбранного WAN (v117)
Сообщения `No selected WAN has a physical IPv4 default route`, ошибки
генерации конфигурации и `crash loop 6 crashes` описывают последовательность:
временно исчез маршрут на выбранном uplink, каждый запуск агента завершался,
а procd прекратил повторные попытки запуска Xray. Затем Heartbeat смог
восстановиться самостоятельно. Он делает HTTP-запрос, поэтому его успех
не подтверждает работу двух обратных соединений Xray для Админки и SSH.
В прежнем `wan_reconcile_xray` остановленный Xray игнорировался; при возврате
того же интерфейса также ничего не происходило. Это подтверждённый дефект
восстановления. Причину исчезновения исходного маршрута и фактическое
текущее состояние Xray на удалённом устройстве только эти логи не определяют.
## Исправление
- Перед генерацией JSON служебный процесс ожидает выбранный WAN с физическим
IPv4 default route. Проверка каждые 5 секунд, одна запись о начале ожидания
и одна о возвращении WAN. Конфигурация в ожидании не заменяется.
- Ошибки UUID, JSON или проверки Xray при доступном WAN не маскируются
ожиданием сети: генерация по-прежнему завершается с настоящей ошибкой.
- Heartbeat не перезапускает всю службу, пока все выбранные WAN недоступны.
- После подготовки WAN проверяются процесс Xray и состояние procd. Если
Xray отсутствует и его instance не работает, но служба включена и Heartbeat
работает, повторно регистрируется только instance `xray` через `service add`.
Работающий процесс запуска, который ожидает WAN или готовит Xray, не трогается.
- Живой Heartbeat и остальные instance не удаляются. При прежнем здоровом
туннеле на том же интерфейсе перезапуск не запрашивается.
- Блокировка в RAM исключает одновременные запросы восстановления; PID
завершившегося worker освобождается. Повторные запросы ограничены интервалом
300 секунд, в том числе при ошибке ubus. Успешный ответ ubus означает принятие
запроса запуска, а не доказанную доступность Админки.
- Явная остановка через панель (`enabled=0`) или остановка всей службы
не отменяется автоматикой. Невыбранные интерфейсы не используются.
- Источники установки, настройки LAN, DNS, Podkop и логика WAN policy routing
не меняются этим исправлением.
Формат `service add` и строковых параметров respawn сверялся с исходниками:
- https://github.com/openwrt/procd/blob/master/service/service.c
- https://github.com/openwrt/procd/blob/master/service/instance.c
- https://github.com/openwrt/openwrt/blob/openwrt-24.10/package/system/procd/files/procd.sh
## Установка и откат
Windows PowerShell 5.1, роутер пользователя:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Install
```
Скрипт обновляет агент и WAN helper парой, сохраняет первую копию в
`/root/owrt-remote-backup-before-wan-recovery-v117`, проверяет shell-синтаксис
и побайтовое совпадение. Затем перезагружает только работающий Heartbeat,
чтобы в RAM не остался прежний shell с уже загруженными функциями.
Туннель при исправном Xray не перезапускается. Остановленный Heartbeat
не запускается принудительно. Ошибка установки восстанавливает файлы
до текущей попытки. UCI и рабочий JSON скрипт установки не заменяет;
обычный запуск Xray выполняет штатную проверку и генерацию конфигурации.
Установка подтверждается `WAN_RECOVERY_ROUTER_Install_OK`, это подтверждение
установки кода. Проверка состояния:
```powershell
ssh root@192.168.2.1 'owrt-remote status; owrt-remote heartbeat-diagnostics; logread -e owrt-remote | tail -n 60'
```
Откат к коду до первого применения:
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\Documents\openwrt overlay\vps\deploy-wan-recovery-fix.ps1" -Router "root@192.168.2.1" -Action Rollback
```
Для роутера друга нужен его собственный доступный SSH-адрес, VPN или человек
в локальной сети. Через оборванные обратные SSH/HTTP-туннели установить код
невозможно. Если WAN уже вернулся, для разового восстановления на самом
роутере можно выполнить `/etc/init.d/owrt-remote restart`, затем проверить
`owrt-remote status`. Сам по себе этот перезапуск не устанавливает исправление.
## Локальная проверка
`test_wan_recovery.py` моделирует длительное отсутствие маршрута, возврат
того же интерфейса, остановленный Xray при работающем Heartbeat, начальную
подготовку, исправный туннель, ручную остановку, параллельные запросы,
оставшуюся блокировку, отказ ubus и настоящую ошибку генерации JSON.
Поддерживает `OWRT_RECOVERY_TEST_SHELL` для BusyBox ash.
`test_wan_recovery_deploy.py` проверяет установку, повторную установку,
сохранение первой резервной копии, откат, отсутствие копии, повреждённый
payload, отказ проверки и отказ перезапуска Heartbeat без реального SSH.
Дополнительно запускаются регрессии WAN failover, host policy, HTTP/HTTPS
авторизации и перенаправлений. Проверка BusyBox ash выполняется локальной
Windows-сборкой BusyBox; OpenWrt 24.10.3 и настоящий procd на устройстве
требуют проверки пользователем. Прямой WAN на удалённом устройстве не
отключать для искусственного теста без независимого канала доступа.
+3 -3
View File
@@ -1,4 +1,4 @@
# Исправление запуска прямого WAN в v116 # Исправление запуска прямого WAN в v117
На проблемном роутере обычный `wan` был поднят, имел IPv4-адрес, На проблемном роутере обычный `wan` был поднят, имел IPv4-адрес,
маршрут и рабочий DNS. Проверка `ip -4 route show table 210` возвращала маршрут и рабочий DNS. Проверка `ip -4 route show table 210` возвращала
@@ -16,7 +16,7 @@ instance после шести падений. Из-за остановленн
конфигурации останавливают агент до запуска Xray. конфигурации останавливают агент до запуска Xray.
Изменены только `files/usr/lib/owrt-remote/wan.sh` и Изменены только `files/usr/lib/owrt-remote/wan.sh` и
`files/usr/sbin/owrt-remote`. Версия остаётся v116. Настройки Podkop, `files/usr/sbin/owrt-remote`. Версия остаётся v117. Настройки Podkop,
LAN и источники установки GitHub/GitLab не меняются. Обновление VPS LAN и источники установки GitHub/GitLab не меняются. Обновление VPS
для этого исправления не требуется. для этого исправления не требуется.
@@ -28,7 +28,7 @@ powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\k.zolotarev95\
Скрипт перед заменой проверяет синтаксис и подготовку WAN новым helper, Скрипт перед заменой проверяет синтаксис и подготовку WAN новым helper,
сохраняет агент, helper, UCI и Xray-конфигурацию в сохраняет агент, helper, UCI и Xray-конфигурацию в
`/root/owrt-remote-backup-before-wan-stability-v116`. Сохранённая копия `/root/owrt-remote-backup-before-wan-stability-v117`. Сохранённая копия
не перезаписывается при повторной установке. После замены выполняются не перезаписывается при повторной установке. После замены выполняются
перезапуск, проверка процесса Xray и настоящий heartbeat до Hub. перезапуск, проверка процесса Xray и настоящий heartbeat до Hub.
При ошибке восстанавливается состояние до текущей попытки. При ошибке восстанавливается состояние до текущей попытки.
+6 -6
View File
@@ -1,4 +1,4 @@
# Применение OpenWrt config через панель роутера (v116) # Применение OpenWrt config через панель роутера (v117)
При отправке textarea браузер кодирует переносы строк как CRLF. CGI При отправке textarea браузер кодирует переносы строк как CRLF. CGI
декодировал их без нормализации и записывал CR в исполняемый shell-скрипт. декодировал их без нормализации и записывал CR в исполняемый shell-скрипт.
@@ -22,7 +22,7 @@ CRLF и LF, heredoc, пустые строки, кодированные сим
Скрипт обновляет только CGI на роутере. VPS обновлять для этого фикса Скрипт обновляет только CGI на роутере. VPS обновлять для этого фикса
не требуется. До замены сохраняется первая резервная копия CGI: не требуется. До замены сохраняется первая резервная копия CGI:
`/root/owrt-remote-backup-before-xray-apply-v116/owrt-remote`. `/root/owrt-remote-backup-before-xray-apply-v117/owrt-remote`.
Из PowerShell Windows 11; замените IP, если адрес роутера отличается: Из PowerShell Windows 11; замените IP, если адрес роутера отличается:
@@ -53,7 +53,7 @@ python tests/test_xray_apply.py -v
Переносы строк в формах описаны в [стандарте HTML](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html). Переносы строк в формах описаны в [стандарте HTML](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html).
## Мобильная вставка команд (локальный фикс v116) ## Мобильная вставка команд (локальный фикс v117)
Воспроизведён отдельный случай `rc=255`: в строке Воспроизведён отдельный случай `rc=255`: в строке
`uci set owrtremote.main.router_name=‘Дом у моря’` типографские кавычки `uci set owrtremote.main.router_name=‘Дом у моря’` типографские кавычки
@@ -77,7 +77,7 @@ python tests/test_xray_apply.py -v
Обновляется только `/www/cgi-bin/owrt-remote` на роутере, без перезапуска Обновляется только `/www/cgi-bin/owrt-remote` на роутере, без перезапуска
служб и изменений настроек при установке. Резервная копия отдельная: служб и изменений настроек при установке. Резервная копия отдельная:
`/root/owrt-remote-backup-before-xray-mobile-v116/owrt-remote`. `/root/owrt-remote-backup-before-xray-mobile-v117/owrt-remote`.
Установка из PowerShell Windows 11: Установка из PowerShell Windows 11:
@@ -101,7 +101,7 @@ python tests/test_xray_mobile_deploy.py -v
``` ```
## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v116) ## Проверка Xray и ошибка Heartbeat после применения (локальный фикс v117)
В предоставленном журнале Xray 26.3.27 запускается успешно, а завершающая В предоставленном журнале Xray 26.3.27 запускается успешно, а завершающая
команда `owrt-remote heartbeat` возвращает `rc=1` с `Failed to allocate команда `owrt-remote heartbeat` возвращает `rc=1` с `Failed to allocate
@@ -153,7 +153,7 @@ CGI получает этап выполнения от агента. Ошибк
Новый скрипт `vps/deploy-xray-validation-fix.ps1` обновляет CGI и агент, Новый скрипт `vps/deploy-xray-validation-fix.ps1` обновляет CGI и агент,
не пересобирает настройки и не перезапускает службы при установке/откате. не пересобирает настройки и не перезапускает службы при установке/откате.
Первая резервная копия обоих файлов: Первая резервная копия обоих файлов:
`/root/owrt-remote-backup-before-xray-validation-v116`. Если установка `/root/owrt-remote-backup-before-xray-validation-v117`. Если установка
оборвалась или проверка файлов не прошла, возвращаются файлы до этой попытки. оборвалась или проверка файлов не прошла, возвращаются файлы до этой попытки.
На роутере друга нужно указать его доступный SSH-адрес вместо локального IP. На роутере друга нужно указать его доступный SSH-адрес вместо локального IP.
+12 -90
View File
@@ -1,4 +1,4 @@
"""Exercise the real RAM journal and authorized router CGI with temporary files.""" """Exercise the bounded persistent journal and authorized router CGI with temporary files."""
import concurrent.futures import concurrent.futures
import base64 import base64
import json import json
@@ -22,12 +22,14 @@ class GlobalLogTests(unittest.TestCase):
self.addCleanup(self.temp.cleanup) self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name) self.root = Path(self.temp.name)
self.log_dir = self.root / 'journal' self.log_dir = self.root / 'journal'
self.run_dir = self.root / 'runtime'
self.helper = self.root / 'logs.sh' self.helper = self.root / 'logs.sh'
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8') source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }', source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
'log_clock() { echo "${TEST_UPTIME:-1000}"; }') 'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
self.helper_source = source self.helper_source = source
source = source.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1) source = source.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
source = source.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n') self.helper.write_text(source, encoding='utf-8', newline='\n')
self.cgi = self.root / 'cgi' self.cgi = self.root / 'cgi'
self.cgi.write_text((ROOT / 'files/www/cgi-bin/owrt-remote').read_text(encoding='utf-8'), self.cgi.write_text((ROOT / 'files/www/cgi-bin/owrt-remote').read_text(encoding='utf-8'),
@@ -35,12 +37,15 @@ class GlobalLogTests(unittest.TestCase):
self.key = self.root / 'key' self.key = self.root / 'key'
self.key.write_text('test-key', encoding='utf-8') self.key.write_text('test-key', encoding='utf-8')
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': self.log_dir.as_posix(), self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': self.log_dir.as_posix(),
'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
'OWRT_REMOTE_LOG_HELPER': self.helper.as_posix(), 'TMPDIR': self.root.as_posix(), 'OWRT_REMOTE_LOG_HELPER': self.helper.as_posix(), 'TMPDIR': self.root.as_posix(),
'KEY_FILE': self.key.as_posix(), 'TEST_UPTIME': '1000'} 'KEY_FILE': self.key.as_posix(), 'TEST_UPTIME': '1000'}
if os.name == 'nt': if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH'] self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def helper_call(self, command, text='', *args, env=None): def helper_call(self, command, text='', *args, env=None):
if command == 'export':
self.helper_call('flush', '', 'force', env=env)
result = subprocess.run([SH, self.helper.as_posix(), command, *args], input=text, result = subprocess.run([SH, self.helper.as_posix(), command, *args], input=text,
encoding='utf-8', capture_output=True, env=env or self.env, timeout=20) encoding='utf-8', capture_output=True, env=env or self.env, timeout=20)
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
@@ -61,8 +66,9 @@ class GlobalLogTests(unittest.TestCase):
self.assertIn('Xray connected', downloaded) self.assertIn('Xray connected', downloaded)
self.assertIn('Cache-Control: no-store', downloaded) self.assertIn('Cache-Control: no-store', downloaded)
# The bounded read is independent of append filtering; supply an existing large journal. # The bounded read is independent of append filtering; supply an existing large journal.
with (self.log_dir / 'current.log').open('a', encoding='utf-8') as journal: state = (self.log_dir / 'state').read_text().split()
journal.write(''.join(f'{i}: ' + 'y' * 1024 + '\n' for i in range(70))) for i in range(5):
(self.log_dir / f'part.{state[0]}.{i:018d}.{i+1:018d}.1').write_text('y' * 15000 + '\n')
live = self.cgi_call('logs_live').split('\n\n', 1)[1] live = self.cgi_call('logs_live').split('\n\n', 1)[1]
# Git for Windows can translate CRLF inside native pipes. # Git for Windows can translate CRLF inside native pipes.
self.assertLessEqual(len(live.encode()), 65536) self.assertLessEqual(len(live.encode()), 65536)
@@ -141,32 +147,6 @@ class GlobalLogTests(unittest.TestCase):
lines = self.helper_call('read').splitlines() lines = self.helper_call('read').splitlines()
self.assertEqual([line.split(']', 1)[0] for line in lines], ['[ERROR', '[ERROR', '[ERROR', '[WARN']) self.assertEqual([line.split(']', 1)[0] for line in lines], ['[ERROR', '[ERROR', '[ERROR', '[WARN'])
def test_persistent_storage_is_rejected_before_any_file_is_written(self):
mounts = self.root / 'mounts'
mounts.write_text('/dev/root / overlay rw 0 0\n', encoding='utf-8')
source = self.helper_source.replace('/proc/mounts', mounts.as_posix())
self.helper.write_text(source, encoding='utf-8', newline='\n')
env = self.env
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not be written\n',
encoding='utf-8', capture_output=True, env=env, timeout=20)
self.assertNotEqual(result.returncode, 0)
self.assertIn('refusing persistent storage', result.stderr)
self.assertFalse(self.log_dir.exists())
self.env = env
self.assertIn('503 Service Unavailable', self.cgi_call('logs_live'))
self.assertFalse(self.log_dir.exists())
# The same location is permitted only when its containing mount is tmpfs.
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix() + ' tmpfs rw 0 0\n', encoding='utf-8')
self.helper_call('append', 'RAM fact\n')
self.assertEqual(self.helper_call('read'), '[OK] RAM fact\n')
mounts.write_text('/dev/root / overlay rw 0 0\ntmpfs ' + self.root.as_posix()
+ ' tmpfs rw 0 0\n/dev/flash ' + self.log_dir.as_posix()
+ ' overlay rw 0 0\n', encoding='utf-8')
result = subprocess.run([SH, self.helper.as_posix(), 'append'], input='must not enter nested Flash mount\n',
encoding='utf-8', capture_output=True, env=env, timeout=20)
self.assertNotEqual(result.returncode, 0)
self.assertEqual((self.log_dir / 'current.log').read_text(), '[OK] RAM fact\n')
def test_clear_requires_post_and_correct_key(self): def test_clear_requires_post_and_correct_key(self):
self.helper_call('append', 'retain this event\n') self.helper_call('append', 'retain this event\n')
self.assertIn('Доступ запрещен', self.cgi_call('logs_download', key='wrong')) self.assertIn('Доступ запрещен', self.cgi_call('logs_download', key='wrong'))
@@ -178,13 +158,6 @@ class GlobalLogTests(unittest.TestCase):
self.helper_call('append', 'new event\n') self.helper_call('append', 'new event\n')
self.assertEqual(self.helper_call('read'), '[OK] new event\n') self.assertEqual(self.helper_call('read'), '[OK] new event\n')
def test_retention_uses_uptime_and_clears_on_reboot(self):
self.helper_call('append', 'old event\n')
self.assertIn('old event', self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87399'}))
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '87400'}), '')
self.helper_call('append', 'after daily clear\n', env={**self.env, 'TEST_UPTIME': '87401'})
self.assertEqual(self.helper_call('read', env={**self.env, 'TEST_UPTIME': '20'}), '')
def test_export_acknowledges_only_received_bytes_and_remote_clear_is_idempotent(self): def test_export_acknowledges_only_received_bytes_and_remote_clear_is_idempotent(self):
self.helper_call('append', 'First event\n') self.helper_call('append', 'First event\n')
first = json.loads(self.helper_call('export')) first = json.loads(self.helper_call('export'))
@@ -207,39 +180,6 @@ class GlobalLogTests(unittest.TestCase):
self.helper_call('ack', '', second['journal_id'], str(second['to']), 'clear-123') self.helper_call('ack', '', second['journal_id'], str(second['to']), 'clear-123')
self.assertIn('After clear', self.helper_call('read'), 'retrying clear must not clear new events') self.assertIn('After clear', self.helper_call('read'), 'retrying clear must not clear new events')
def test_export_catches_up_in_bounded_batches_after_rotation(self):
self.helper_call('clear')
raw = b'[OK] ' + b'x' * 100000 + b'\n'
(self.log_dir / 'current.log').write_bytes(raw)
first = json.loads(self.helper_call('export'))
first_raw = base64.b64decode(first['data'])
self.assertLessEqual(len(first_raw), 65536)
self.helper_call('ack', '', first['journal_id'], str(first['to']))
second = json.loads(self.helper_call('export'))
self.assertEqual(first_raw + base64.b64decode(second['data']), raw)
# Bytes dropped by local rotation are detected by the server from the gap.
(self.log_dir / 'current.log').write_bytes(b'[OK] Latest\n')
(self.log_dir / 'total').write_text('200000')
late = json.loads(self.helper_call('export'))
self.assertEqual(late['from'], 200000 - len(b'[OK] Latest\n'))
self.assertEqual(base64.b64decode(late['data']), b'[OK] Latest\n')
def test_rotation_retains_recent_log_and_clear_removes_both(self):
self.helper_call('clear')
current = self.log_dir / 'current.log'
current.write_text('A' * 1048576, encoding='utf-8')
self.helper_call('append', 'second file\n')
self.assertTrue((self.log_dir / 'previous.log').exists())
self.assertEqual(current.read_text(), '[OK] second file\n')
current.write_text('B' * 1048576, encoding='utf-8')
self.helper_call('append', 'third file\n')
output = self.helper_call('read')
self.assertNotIn('A', output)
self.assertTrue(output.endswith('third file\n'))
self.helper_call('clear')
self.assertFalse((self.log_dir / 'previous.log').exists())
self.assertEqual(current.read_text(), '')
def test_secrets_are_redacted_before_storage_and_html_is_literal(self): def test_secrets_are_redacted_before_storage_and_html_is_literal(self):
text = ('Heartbeat OK\nuci set owrtremote.main.hub_token=SECRET_TOKEN\n' text = ('Heartbeat OK\nuci set owrtremote.main.hub_token=SECRET_TOKEN\n'
'Authorization: Bearer SECRET_AUTH\nhttps://hub.test/?key=SECRET_KEY\n' 'Authorization: Bearer SECRET_AUTH\nhttps://hub.test/?key=SECRET_KEY\n'
@@ -256,11 +196,11 @@ class GlobalLogTests(unittest.TestCase):
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool: with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
list(pool.map(lambda i: self.helper_call('append', f'event-{i}\n'), range(12))) list(pool.map(lambda i: self.helper_call('append', f'event-{i}\n'), range(12)))
self.assertEqual(set(self.helper_call('read').splitlines()), {f'[OK] event-{i}' for i in range(12)}) self.assertEqual(set(self.helper_call('read').splitlines()), {f'[OK] event-{i}' for i in range(12)})
self.assertFalse((self.log_dir / 'lock').exists()) self.assertFalse((self.run_dir / 'lock').exists())
def test_stale_lock_recovers(self): def test_stale_lock_recovers(self):
self.helper_call('clear') self.helper_call('clear')
lock = self.log_dir / 'lock' lock = self.run_dir / 'lock'
lock.mkdir() lock.mkdir()
(lock / 'pid').write_text('99999999') (lock / 'pid').write_text('99999999')
self.helper_call('append', 'recovered\n') self.helper_call('append', 'recovered\n')
@@ -272,28 +212,10 @@ class GlobalLogTests(unittest.TestCase):
def test_empty_lock_left_by_killed_writer_recovers(self): def test_empty_lock_left_by_killed_writer_recovers(self):
self.helper_call('clear') self.helper_call('clear')
(self.log_dir / 'lock').mkdir() (self.run_dir / 'lock').mkdir()
self.helper_call('append', 'after empty lock\n') self.helper_call('append', 'after empty lock\n')
self.assertEqual(self.helper_call('read'), '[OK] after empty lock\n') self.assertEqual(self.helper_call('read'), '[OK] after empty lock\n')
def test_collector_failure_cleans_both_children_and_fifo(self):
source = self.helper.read_text(encoding='utf-8')
mocks = '''
log_init() { return 0; }
mkfifo() { return 0; }
logread() { return 0; }
log_lock() { return 1; }
kill() { printf 'KILL=%s\\n' "$*"; }
wait() { return 0; }
rm() { printf 'REMOVE=%s\\n' "$*"; }
'''
self.helper.write_text(source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1),
encoding='utf-8', newline='\n')
result = subprocess.run([SH, self.helper.as_posix(), 'collect'], capture_output=True,
encoding='utf-8', env=self.env, timeout=10)
self.assertNotEqual(result.returncode, 0)
self.assertEqual(result.stdout.count('KILL='), 2)
self.assertIn('REMOVE=-f ' + self.log_dir.as_posix() + '/stream.', result.stdout)
if __name__ == '__main__': if __name__ == '__main__':
+5 -4
View File
@@ -43,7 +43,7 @@ class GlobalLogsDeploymentTests(unittest.TestCase):
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix() mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix() mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix() mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
mappings['/root/owrt-remote-backup-before-global-logs-stop-v116'] = self.backup.as_posix() mappings['/root/owrt-remote-backup-before-global-logs-stop-v117'] = self.backup.as_posix()
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True)) pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script) self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
# Existing sandbox service scripts only print markers. Mock readiness. # Existing sandbox service scripts only print markers. Mock readiness.
@@ -55,7 +55,7 @@ jsonfilter() { return 0; }
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; } logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
''' '''
self.script = self.script.replace('for item in cgi agent init;', mocks + '\nfor item in cgi agent init;', 1) self.script = self.script.replace('for item in cgi agent init;', mocks + '\nfor item in cgi agent init;', 1)
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix()} self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix() self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
if os.name == 'nt': if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH'] self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
@@ -75,6 +75,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
if item == 'logs.sh': if item == 'logs.sh':
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }') text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1) text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
if item == 'logs-init': if item == 'logs-init':
text = '#!/bin/sh\necho logs-service-$1\n' text = '#!/bin/sh\necho logs-service-$1\n'
if bad and item == 'cgi': if bad and item == 'cgi':
@@ -103,7 +104,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
self.assertIn('GLOBAL_LOGS_ROUTER_Install_OK', result.stdout) self.assertIn('GLOBAL_LOGS_ROUTER_Install_OK', result.stdout)
for item in ('cgi', 'luci'): for item in ('cgi', 'luci'):
installed = self.root / self.targets[item].lstrip('/') installed = self.root / self.targets[item].lstrip('/')
self.assertIn('v116', installed.read_text(encoding='utf-8')) self.assertIn('v117', installed.read_text(encoding='utf-8'))
for item, data in self.initial.items(): for item, data in self.initial.items():
self.assertEqual(self.backup.joinpath(item).read_bytes(), data) self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
result = self.deploy('Rollback') result = self.deploy('Rollback')
@@ -130,7 +131,7 @@ logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
self.assert_initial() self.assert_initial()
def test_rollback_restores_existing_log_service_and_helper(self): def test_rollback_restores_existing_log_service_and_helper(self):
# This fix is normally applied on top of v116's already installed journal. # This fix is normally applied on top of v117's already installed journal.
for item in ('logs.sh', 'logs-init'): for item in ('logs.sh', 'logs-init'):
target = self.root / self.targets[item].lstrip('/') target = self.root / self.targets[item].lstrip('/')
target.parent.mkdir(parents=True, exist_ok=True) target.parent.mkdir(parents=True, exist_ok=True)
+2 -2
View File
@@ -20,7 +20,7 @@ class VpsLogDeployTests(unittest.TestCase):
self.target = self.root / 'hub.py' self.target = self.root / 'hub.py'
self.previous = b'print("previous Hub")\n' self.previous = b'print("previous Hub")\n'
self.target.write_bytes(self.previous) self.target.write_bytes(self.previous)
self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v116' self.backup = self.root / 'hub.py.bak-before-global-logs-ui-v117'
source = (ROOT / 'vps/deploy-global-logs-vps.ps1').read_text(encoding='utf-8') source = (ROOT / 'vps/deploy-global-logs-vps.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix()) self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
@@ -55,7 +55,7 @@ sleep() { return 0; }
for _ in range(2): for _ in range(2):
result = self.deploy('Install') result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('GLOBAL_LOGS_VPS_V116_Install_OK', result.stdout) self.assertIn('GLOBAL_LOGS_VPS_V117_Install_OK', result.stdout)
self.assertIn(b'def receive_router_logs(', self.target.read_bytes()) self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
self.assertEqual(self.backup.read_bytes(), self.previous) self.assertEqual(self.backup.read_bytes(), self.previous)
result = self.deploy('Rollback') result = self.deploy('Rollback')
+1 -1
View File
@@ -25,7 +25,7 @@ class HeartbeatTransportDeploymentTests(unittest.TestCase):
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"] self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-heartbeat-transport-fix.ps1").read_text(encoding="utf-8") source = (ROOT / "vps/deploy-heartbeat-transport-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v116", self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix())) "backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix())) self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix())) self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
+423
View File
@@ -0,0 +1,423 @@
"""Persistent spool recovery/resource limits using the real shell helper.
Mount fixtures replace /proc/mounts only in disposable copies. Set
OWRT_LOG_TEST_SHELL to a BusyBox executable to exercise its actual ash/applets.
"""
import base64
import concurrent.futures
import json
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import time
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
BUSYBOX = os.environ.get('OWRT_LOG_TEST_SHELL', '')
class PersistentLogsTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='persistent-logs-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name).resolve()
self.run_dir = self.root / 'runtime'
self.flash = self.root / 'flash'
self.flash.mkdir()
self.helper = self.root / 'logs.sh'
self.mounts = self.root / 'mounts'
self.root_mount = self.root.as_posix()
self.flash_mount = self.flash.as_posix()
if os.name == 'nt' and not BUSYBOX:
# Git's bundled readlink may emit native C:/ or MSYS /c paths.
# The fixture must use what the actual applet returns, not assume it.
env = {**os.environ, 'PATH': str(GIT_BIN) + os.pathsep + os.environ['PATH']}
def resolved_mount(path):
return subprocess.check_output([SH, '-c', 'readlink -f "$1"', '--', path.as_posix()],
env=env, text=True).strip()
self.root_mount = resolved_mount(self.root)
self.flash_mount = resolved_mount(self.flash)
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
f'/dev/ubi0_1 {self.flash_mount} ubifs rw 0 0\n')
source = (ROOT / 'files/usr/lib/owrt-remote/logs.sh').read_text(encoding='utf-8')
source = source.replace('/proc/mounts', self.mounts.as_posix())
if os.name == 'nt':
source = source.replace('case "$path" in /*)', 'case "$path" in [a-zA-Z]:/*|/*)')
source = source.replace('log_clock() { cut -d. -f1 /proc/uptime; }',
'log_clock() { echo "${TEST_UPTIME:-1000}"; }')
source = source.replace('log_epoch() { date +%s; }',
'log_epoch() { echo "${TEST_EPOCH:-' + str(int(time.time())) + '}"; }')
# Real df remains covered in integration tests. Fixtures emulate free space.
source = source.replace('free="$(df -Pk "$LOG_DIR" | awk \'NR==2 {print $4}\')"',
'free="${TEST_FREE_KB:-8192}"')
self.source = source
self.source = self.source.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + (self.root / 'legacy').as_posix())
self.helper.write_text(self.source, encoding='utf-8', newline='\n')
self.env = {**os.environ, 'OWRT_REMOTE_LOG_RUNTIME': self.run_dir.as_posix(),
'OWRT_REMOTE_LOG_DIR': (self.flash / 'journal').as_posix()}
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
self.shell = [BUSYBOX, 'ash'] if BUSYBOX else [SH]
if BUSYBOX:
self.env['BB_OVERRIDE_APPLETS'] = '1'
def call(self, command, text='', *args, check=True, **flags):
result = subprocess.run([*self.shell, self.helper.as_posix(), command, *args],
input=text, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25)
if check:
self.assertEqual(result.returncode, 0, result.stderr)
return result
def status(self):
return dict(line.split('=', 1) for line in self.call('status').stdout.splitlines())
def flush(self, **flags):
return self.call('flush', '', 'force', **flags)
def export(self):
return json.loads(self.call('export').stdout)
def append(self, text, **flags):
return self.call('append', text, **flags)
def acknowledge(self, batch):
self.call('ack', '', batch['journal_id'], str(batch['to']))
@property
def journal(self):
return self.flash / 'journal'
def reboot(self):
shutil.rmtree(self.run_dir)
def test_persistent_storage_selected_and_survives_runtime_loss(self):
self.append('before reboot\n')
self.flush()
first = self.export()
self.acknowledge(first)
self.flush()
self.reboot()
self.assertEqual(self.status()['persistent'], '1')
self.assertIn('before reboot', self.call('read').stdout)
after = self.export()
self.assertEqual(after['journal_id'], first['journal_id'])
self.assertEqual(after['from'], first['to'])
self.assertEqual(after['data'], '')
self.assertFalse(any(self.journal.glob('stream.*')))
self.assertFalse((self.journal / 'lock').exists())
def test_missing_optional_cksum_does_not_create_empty_digest_or_drop_facts(self):
source = self.source.replace('case "${1:-}" in', 'cksum() { return 127; }\ncase "${1:-}" in', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.append('first event\nsecond event\n')
self.flush()
part = next(self.journal.glob('part.*'))
self.assertIn('.s256-', part.name)
self.assertIn(b'second event', base64.b64decode(self.export()['data']))
self.reboot()
self.assertIn(b'first event', base64.b64decode(self.export()['data']))
def test_digest_fingerprint_cache_stays_small_at_64_entries(self):
self.call('check')
# Seed 63 existing fingerprints; exercise eviction with two real events
# instead of starting thousands of Windows utility processes for a fill.
(self.run_dir / 'facts').write_text(''.join('1000\ts256-%035d\n' % i for i in range(63)))
self.append('distinct event 63\ndistinct event 64\n')
self.assertEqual(len((self.run_dir / 'facts').read_text().splitlines()), 64)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
def test_empty_digest_files_falsely_acked_by_old_hub_are_recovered(self):
self.append('retained restart event\n'); self.flush()
before = self.export()
self.acknowledge(before); self.flush()
part = next(self.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
# Upgrade during the same boot must override the former recovery marker.
(self.run_dir / 'recovered-v2').unlink()
(self.run_dir / 'recovered').touch()
repaired = self.export()
self.assertNotEqual(repaired['journal_id'], before['journal_id'])
self.assertTrue(repaired['journal_id'].startswith('recovered-'))
self.assertEqual(repaired['from'], 0)
self.assertIn(b'retained restart event', base64.b64decode(repaired['data']))
self.assertFalse(any(p.name.endswith('.') for p in self.journal.glob('part.*')))
self.acknowledge(repaired)
self.assertEqual(self.export()['data'], '')
self.flush()
# Recovery's warning was buffered after the replay chunk; deliver it too.
self.acknowledge(self.export())
self.flush()
self.reboot()
self.assertEqual(self.export()['data'], '')
def test_checksum_failure_keeps_pending_and_does_not_advance_total(self):
self.append('keep until checksum works\n')
before = self.status()['committed_bytes']
source = self.source.replace('log_checksum() {', 'log_checksum() { return 1;', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.call('flush', '', 'force', check=False)
self.assertFalse(list(self.journal.glob('part.*')))
self.assertEqual(self.status()['committed_bytes'], before)
self.assertIn('keep until checksum works', (self.run_dir / 'pending').read_text())
def test_md5_and_legacy_numeric_checksum_formats_remain_readable(self):
for digest in ('md5', 'crc'):
with self.subTest(digest=digest):
self.call('clear')
source = self.source.replace('if command -v sha256sum >/dev/null 2>&1; then method=s256',
'if false; then method=s256')
if digest == 'crc':
source = source.replace('elif command -v md5sum >/dev/null 2>&1; then method=md5',
'elif false; then method=md5')
self.helper.write_text(source, encoding='utf-8', newline='\n')
self.append('digest compatibility ' + digest + '\n'); self.flush()
value = self.export()
self.reboot()
self.assertEqual(self.export(), value)
self.assertIn(digest.encode(), base64.b64decode(value['data']))
name = next(self.journal.glob('part.*')).name.rsplit('.', 1)[1]
self.assertEqual(name.startswith('md5-'), digest == 'md5')
def test_empty_export_never_claims_unreadable_chunks_as_delivered(self):
self.append('not yet delivered\n'); self.flush()
part = next(self.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.invalid'))
empty = self.export()
self.assertEqual((empty['from'], empty['to'], empty['data']), (0, 0, ''))
def test_failed_or_unmounted_overlay_does_not_receive_journal(self):
for kind, flags in [('overlay', 'rw'), ('tmpfs', 'rw'), ('ubifs', 'ro')]:
with self.subTest(kind=kind, flags=flags):
if self.run_dir.exists(): self.reboot()
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n'
f'fake {self.flash_mount} {kind} {flags} 0 0\n')
self.append('RAM fallback\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertIn('Flash недоступна', self.call('read').stdout)
self.assertFalse(self.journal.exists())
def test_nested_tmpfs_and_symlink_do_not_pass_flash_check(self):
nested = self.flash / 'ram'
nested.mkdir()
self.mounts.write_text(self.mounts.read_text() + f'tmpfs {self.flash_mount}/ram tmpfs rw 0 0\n')
self.env['OWRT_REMOTE_LOG_DIR'] = (nested / 'logs').as_posix()
self.append('nested tmpfs\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertFalse((nested / 'logs').exists())
def test_periodic_flush_and_idle_polling_do_not_rewrite_flash(self):
self.call('check')
state = self.journal / 'state'
before = state.stat().st_mtime_ns
self.append('buffered record\n')
self.assertEqual(list(self.journal.glob('part.*')), [])
self.assertEqual(self.export()['data'], '')
self.call('flush', TEST_UPTIME='1059')
self.assertEqual(list(self.journal.glob('part.*')), [])
self.call('flush', TEST_UPTIME='1060')
self.assertEqual(len(list(self.journal.glob('part.*'))), 1)
self.assertNotEqual(state.stat().st_mtime_ns, before)
stable = {p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}
for _ in range(3):
self.call('flush', TEST_UPTIME='1120')
self.export()
self.assertEqual({p.name: p.stat().st_mtime_ns for p in self.journal.iterdir()}, stable)
def test_buffer_and_number_of_archives_are_bounded_without_dropping_unacked(self):
self.helper.write_text(self.source.replace('LOG_LIMIT=2097152', 'LOG_LIMIT=32768')
.replace('LOG_PART_LIMIT=256', 'LOG_PART_LIMIT=2'),
encoding='utf-8', newline='\n')
for i in range(4):
self.append(f'event {i} ' + 'x' * 4000 + '\n')
self.flush()
parts = sorted(self.journal.glob('part.*'))
self.assertEqual(len(parts), 2)
original = {p.name: p.read_bytes() for p in parts}
for i in range(5):
self.append(f'overflow {i} ' + 'y' * 4000 + '\n')
self.flush()
self.assertEqual({p.name: p.read_bytes() for p in parts}, original)
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
self.assertLessEqual((self.run_dir / 'facts').stat().st_size, 4096)
first = self.export()
self.acknowledge(first)
self.flush()
self.assertFalse(parts[0].exists(), 'only acknowledged chunks make room')
self.assertIn('пропущено новых событий', self.call('read').stdout)
def test_ack_is_monotonic_and_does_not_delete_unconfirmed_after_one_day(self):
self.append('not delivered\n')
self.flush()
batch = self.export()
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
self.assertEqual(self.export(), batch)
invalid = self.call('ack', '', batch['journal_id'], str(batch['to'] + 1), check=False)
self.assertNotEqual(invalid.returncode, 0)
self.acknowledge(batch)
self.call('flush', '', 'force', TEST_EPOCH='2000000000')
self.assertEqual(list(self.journal.glob('part.*')), [])
invalid = self.call('ack', '', batch['journal_id'], '0', check=False)
self.assertNotEqual(invalid.returncode, 0)
def test_crash_after_chunk_rename_recovers_original_id_and_offsets(self):
self.append('committed first\n')
self.flush()
first = self.export()
previous_state = (self.journal / 'state').read_bytes()
self.append('second chunk before state checkpoint\n')
self.flush()
(self.journal / 'state').write_bytes(previous_state)
(self.journal / 'chunk.new').write_text('interrupted write')
(self.journal / 'state.new').write_text('partial state')
self.reboot()
self.assertEqual(self.export()['journal_id'], first['journal_id'])
self.acknowledge(first)
second = self.export()
self.assertEqual(second['from'], first['to'])
self.assertIn(b'second chunk', base64.b64decode(second['data']))
self.assertFalse((self.journal / 'chunk.new').exists())
self.assertFalse((self.journal / 'state.new').exists())
def test_corrupt_chunk_is_quarantined_and_later_chunks_still_deliver(self):
self.append('first\n'); self.flush()
self.append('second\n'); self.flush()
first = sorted(self.journal.glob('part.*'))[0]
first.write_bytes(b'partial')
self.reboot()
batch = self.export()
self.assertIn(b'second', base64.b64decode(batch['data']))
self.assertTrue((self.journal / 'damaged.log').exists())
self.assertIn('Повреждённый фрагмент', self.call('read').stdout)
def test_disk_full_falls_back_without_touching_router_configuration(self):
self.append('retained on flash\n'); self.flush()
self.append('buffered before ENOSPC\n')
self.call('flush', '', 'force', TEST_FREE_KB='100', check=False)
self.append('after disk full\n')
self.assertEqual(self.status()['persistent'], '0')
self.assertIn('after disk full', self.call('read').stdout)
self.assertTrue(list(self.journal.glob('part.*')))
def test_confirmed_ack_can_replay_after_crash_without_changing_cursor_identity(self):
self.append('duplicate safe\n'); self.flush()
batch = self.export()
self.acknowledge(batch)
self.assertEqual(self.export()['data'], '')
self.reboot() # ack not yet checkpointed: safe replay, Hub suppresses it.
self.assertEqual(self.export(), batch)
def test_killed_flusher_reconciles_inflight_without_duplicate_data(self):
self.append('already renamed\n')
self.flush()
batch = self.export()
part = next(self.journal.glob('part.*'))
fields = part.name.rsplit('.', 3)
(self.run_dir / 'pending').write_bytes(base64.b64decode(batch['data']))
(self.run_dir / 'inflight').write_text(f'{batch["journal_id"]} 0 {batch["to"]} {fields[-1]}\n')
self.append('after killed writer\n')
text = self.call('read').stdout
self.assertEqual(text.count('already renamed'), 1)
self.assertIn('after killed writer', text)
self.assertFalse((self.run_dir / 'inflight').exists())
def test_corrupt_state_recovers_committed_generation_and_stale_temporaries(self):
self.append('state recovery\n'); self.flush()
batch = self.export()
(self.journal / 'state').write_text('partial garbage')
self.reboot()
self.assertEqual(self.export(), batch)
(self.run_dir / 'export.12345').write_text('orphan')
self.export()
self.assertFalse(list(self.run_dir.glob('export.*')))
def test_parallel_writers_and_stale_lock_recovery(self):
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
list(pool.map(lambda i: self.append(f'concurrent {i}\n'), range(6)))
self.assertEqual(len(self.call('read').stdout.splitlines()), 6)
(self.run_dir / 'lock').mkdir()
(self.run_dir / 'lock/pid').write_text('99999999')
self.append('recovered lock\n')
self.assertIn('recovered lock', self.call('read').stdout)
self.assertFalse((self.run_dir / 'lock').exists())
def test_stale_lock_reaper_itself_recovers_after_process_crash(self):
self.call('check')
lock = self.run_dir / 'lock'; lock.mkdir()
(lock / 'pid').write_text('99999999')
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
self.append('after killed reaper\n')
self.assertIn('after killed reaper', self.call('read').stdout)
self.assertFalse(lock.exists())
def test_collector_failure_reaps_children_and_removes_fifo(self):
# Actual FIFO integration is tested on POSIX; Windows BusyBox has no
# FIFO support. Exercise its traps with mocked process/FIFO operations.
mocks = r'''
mkfifo() { : >"$1"; }
logread() { return 0; }
sleep() { return 0; }
kill() {
printf '%s\n' "$*" >>"$RUN_DIR/kills"
[ "$1" != -0 ]
}
wait() { return 0; }
'''
source = self.source.replace('case "${1:-}" in', mocks + '\ncase "${1:-}" in', 1)
self.helper.write_text(source, encoding='utf-8', newline='\n')
result = self.call('collect', check=False)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(list(self.run_dir.glob('stream.*')))
self.assertFalse((self.run_dir / 'collector-lock').exists())
lines = (self.run_dir / 'kills').read_text().splitlines()
self.assertGreaterEqual(len([line for line in lines if not line.startswith('-0')]), 2)
def test_legacy_migration_preserves_identity_ack_and_only_undelivered_bytes(self):
legacy = self.root / 'legacy'
legacy.mkdir()
(legacy / 'current.log').write_bytes(b'legacy current\n')
(legacy / 'previous.log').write_bytes(b'legacy previous\n')
raw = b'legacy previous\nlegacy current\n'
(legacy / 'total').write_text(str(len(raw)))
(legacy / 'journal-id').write_text('legacy-journal')
(legacy / 'sync-ack').write_text('legacy-journal 16')
(legacy / 'stream.dead').write_text('must stay transient')
self.call('migrate')
self.assertEqual(self.call('read').stdout, raw.decode())
self.assertFalse((self.journal / 'stream.dead').exists())
self.assertFalse((legacy / 'current.log').exists())
self.assertFalse((legacy / 'total').exists())
self.assertTrue((legacy / 'stream.dead').exists())
batch = self.export()
self.assertEqual(batch['journal_id'], 'legacy-journal')
self.assertEqual(batch['from'], 16)
self.assertEqual(base64.b64decode(batch['data']), b'legacy current\n')
def test_unavailable_flash_legacy_migration_does_not_keep_large_ram_archive(self):
self.mounts.write_text(f'tmpfs {self.root_mount} tmpfs rw 0 0\n')
legacy = self.root / 'legacy'; legacy.mkdir()
(legacy / 'current.log').write_text('old data\n' * 10000)
self.call('migrate')
self.assertFalse((legacy / 'current.log').exists())
self.assertEqual(self.status()['persistent'], '0')
self.assertLessEqual((self.run_dir / 'pending').stat().st_size, 16384)
self.assertIn('ограниченный хвост', self.call('read').stdout)
def test_interrupted_legacy_migration_finishes_after_old_ram_is_gone(self):
self.append('staged persistent data\n'); self.flush()
batch = self.export()
staging = self.journal / 'migrating'; staging.mkdir()
for part in self.journal.glob('part.*'):
part.rename(staging / part.name)
shutil.copyfile(self.journal / 'state', staging / 'state')
(self.journal / 'state').write_text('interrupted state replacement')
self.reboot()
self.assertEqual(self.export(), batch)
self.assertFalse(staging.exists())
if __name__ == '__main__':
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
"""Install/rollback journal files against a sandbox, never a real router."""
import os
import re
from pathlib import Path
import shlex
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
@unittest.skipUnless(SH, 'POSIX shell required')
class PersistentLogsDeploymentTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='persistent-log-deploy-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.targets = {
'cgi': '/www/cgi-bin/owrt-remote', 'agent': '/usr/sbin/owrt-remote',
'init': '/etc/init.d/owrt-remote', 'logs.sh': '/usr/lib/owrt-remote/logs.sh',
'logs-init': '/etc/init.d/owrt-remote-logs',
'luci': '/www/luci-static/resources/view/owrt_remote.js',
}
self.initial = {}
for item in ('cgi', 'agent', 'logs.sh', 'logs-init', 'luci'):
path = self.root / self.targets[item].lstrip('/')
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text('#!/bin/sh\necho previous-' + item + '\n', encoding='utf-8', newline='\n')
self.initial[item] = path.read_bytes()
self.backup = self.root / 'backup'
source = (ROOT / 'vps/deploy-persistent-logs.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
# Absolute paths are replaced only in this disposable test copy.
mappings = {target: (self.root / target.lstrip('/')).as_posix() for target in self.targets.values()}
mappings['/usr/lib/owrt-remote'] = (self.root / 'usr/lib/owrt-remote').as_posix()
mappings['/www/luci-static/resources/view'] = (self.root / 'www/luci-static/resources/view').as_posix()
mappings['/tmp/luci-indexcache'] = (self.root / 'luci-indexcache').as_posix()
mappings['/tmp/luci-modulecache'] = (self.root / 'luci-modulecache').as_posix()
mappings['/etc/init.d/rpcd'] = (self.root / 'etc/init.d/rpcd').as_posix()
mappings['/etc/init.d/uhttpd'] = (self.root / 'etc/init.d/uhttpd').as_posix()
mappings['/tmp/owrt-remote-logs'] = (self.root / 'journal').as_posix()
mappings['/root/owrt-remote-backup-before-persistent-logs-v117'] = self.backup.as_posix()
pattern = '|'.join(re.escape(target) for target in sorted(mappings, key=len, reverse=True))
self.script = re.sub(pattern, lambda match: mappings[match.group()], self.script)
# Existing sandbox service scripts only print markers. Mock readiness.
mocks = '''
logread() { return 0; }
ubus() { printf '{"running": true}\\n'; }
mkfifo() { return 0; }
jsonfilter() { return 0; }
logger() { printf '%s\\n' "$3" | sh "$TEST_LOG_HELPER" append; }
'''
self.script = self.script.replace('for item in cgi agent logs.sh logs-init;', mocks + '\nfor item in cgi agent logs.sh logs-init;', 1)
self.env = {**os.environ, 'OWRT_REMOTE_LOG_DIR': (self.root / 'journal').as_posix(), 'OWRT_REMOTE_LOG_RUNTIME': (self.root / 'runtime').as_posix()}
self.env['TEST_LOG_HELPER'] = (self.root / 'usr/lib/owrt-remote/logs.sh').as_posix()
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def deploy(self, action, bad=False, fail_verify=False):
stage = self.root / 'stage'
stage.mkdir()
candidates = {
'cgi': 'files/www/cgi-bin/owrt-remote', 'agent': 'files/usr/sbin/owrt-remote',
'init': 'files/etc/init.d/owrt-remote', 'logs.sh': 'files/usr/lib/owrt-remote/logs.sh',
'logs-init': 'files/etc/init.d/owrt-remote-logs',
'luci': 'files/www/luci-static/resources/view/owrt_remote.js',
}
if action == 'Install':
for item, source in candidates.items():
text = (ROOT / source).read_text(encoding='utf-8')
if item == 'logs.sh':
text = text.replace('legacy=/tmp/owrt-remote-logs', 'legacy=' + shlex.quote((self.root / 'legacy').as_posix()))
text = text.replace('log_clock() { cut -d. -f1 /proc/uptime; }', 'log_clock() { echo 1000; }')
text = text.replace('log_ram_storage() {', 'log_ram_storage() { return 0;', 1)
text = text.replace('log_flash_storage() {', 'log_flash_storage() { return 0;', 1)
if item == 'logs-init':
text = '#!/bin/sh\necho logs-service-$1\n'
if bad and item == 'cgi':
text = '#!/bin/sh\nif true; then\n'
stage.joinpath(item).write_text(text, encoding='utf-8', newline='\n')
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(stage.as_posix()))
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
if fail_verify:
script = script.replace('cmp "$stage/$item" "$path"', 'false')
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env=self.env, timeout=30)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.rglob('*.new-sandbox')))
return result
def assert_initial(self):
for item, data in self.initial.items():
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
def test_install_then_rollback_retains_first_backup(self):
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_ROUTER_Install_OK', result.stdout)
for item in ('cgi', 'luci'):
installed = self.root / self.targets[item].lstrip('/')
self.assertIn('v117', installed.read_text(encoding='utf-8'))
for item, data in self.initial.items():
self.assertEqual(self.backup.joinpath(item).read_bytes(), data)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_ROUTER_Rollback_OK', result.stdout)
self.assert_initial()
def test_missing_backup_changes_nothing(self):
result = self.deploy('Rollback')
self.assertNotEqual(result.returncode, 0)
self.assertIn('No complete backup:', result.stderr)
self.assert_initial()
def test_invalid_candidate_rejected_before_backup(self):
result = self.deploy('Install', bad=True)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(self.backup.exists())
self.assert_initial()
def test_failed_verification_restores_all_previous_files(self):
result = self.deploy('Install', fail_verify=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn('Restoring files from this attempt.', result.stderr)
self.assert_initial()
def test_rollback_restores_existing_log_service_and_helper(self):
# This fix is normally applied on top of v117's already installed journal.
for item in ('logs.sh', 'logs-init'):
target = self.root / self.targets[item].lstrip('/')
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text('#!/bin/sh\necho previous-' + item + '-$1\n', encoding='utf-8', newline='\n')
self.initial[item] = target.read_bytes()
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
for item, data in self.initial.items():
self.assertEqual((self.root / self.targets[item].lstrip('/')).read_bytes(), data)
self.assertIn('previous-logs-init-start', result.stdout)
if __name__ == '__main__':
unittest.main()
+106
View File
@@ -0,0 +1,106 @@
"""Validate VPS install, rollback and automatic recovery in a disposable tree."""
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
GIT_BIN = Path(os.environ.get('ProgramFiles', 'C:/Program Files')) / 'Git/usr/bin'
SH = str(GIT_BIN / 'sh.exe') if os.name == 'nt' else shutil.which('sh')
@unittest.skipUnless(SH, 'POSIX shell required')
class VpsLogDeployTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='vps-log-deploy-')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.target = self.root / 'hub.py'
self.previous = b'print("previous Hub")\n'
self.target.write_bytes(self.previous)
self.backup = self.root / 'hub.py.bak-before-persistent-logs-sync-v117'
source = (ROOT / 'vps/deploy-persistent-logs-vps.ps1').read_text(encoding='utf-8')
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace('/opt/owrt-remote/owrt-remote-hub.py', self.target.as_posix())
self.script = self.script.replace('python_bin=/opt/owrt-remote/venv/bin/python', 'python_bin=python')
self.script = self.script.replace('[ -x "$python_bin" ] || python_bin=python3', '')
mocks = '''
systemctl() { [ "${TEST_RESTART_FAIL:-0}" != 1 ] || [ "$1" != restart ]; }
journalctl() { echo sandbox-journal; }
curl() { [ "${TEST_HEALTH_FAIL:-0}" != 1 ] && printf '{"ok":true}'; }
sleep() { return 0; }
'''
self.script = self.script.replace('test -s "$target"', mocks + '\ntest -s "$target"', 1)
# Exercise the actual mount/path validation with an isolated process proc
# fixture. Storage classification is covered independently on real paths.
original = self.root / 'running.py'
original.write_text('print("running")')
proc = self.root / 'proc' / '123'
proc.mkdir(parents=True)
(proc / 'environ').write_bytes(('OWRT_REMOTE_STATE_DIR=' + self.root.as_posix()).encode() + b'\0')
(proc / 'cmdline').write_bytes(b'python\0' + original.as_posix().encode() + b'\0')
database = self.root / 'hub.db'
database.write_bytes(b'existing archive must not change')
self.database = database
self.script = self.script.replace("Path('/proc/' + pid", "Path('" + (self.root / 'proc').as_posix() + "/' + pid")
self.script = self.script.replace("storage = hub.router_log_storage(db_path)",
"storage = hub.router_log_storage(db_path)\n" +
"storage.update(persistent=os.environ.get('TEST_RAM_DISK') != '1', writable=True)")
self.script = self.script.replace('systemctl() { ',
'systemctl() { if [ "$1" = show ]; then echo 123; return 0; fi; ')
self.env = dict(os.environ)
if os.name == 'nt':
self.env['PATH'] = str(GIT_BIN) + os.pathsep + self.env['PATH']
def deploy(self, action, invalid=False, **flags):
stage = self.root / 'stage'
stage.mkdir()
if action == 'Install':
data = 'if invalid:\n' if invalid else (ROOT / 'vps/owrt-remote-hub.py').read_text(encoding='utf-8')
stage.joinpath('hub.py').write_text(data, encoding='utf-8', newline='\n')
script = self.script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage="' + stage.as_posix() + '"')
script = script.replace('__UPLOAD_NAME__', 'sandbox').replace('__ACTION__', action)
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.glob('*.new-sandbox')))
self.assertEqual(self.database.read_bytes(), b'existing archive must not change')
return result
def test_install_and_rollback_preserve_original_backup(self):
for _ in range(2):
result = self.deploy('Install')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('PERSISTENT_LOGS_VPS_V117_Install_OK', result.stdout)
self.assertIn(b'def receive_router_logs(', self.target.read_bytes())
self.assertEqual(self.backup.read_bytes(), self.previous)
result = self.deploy('Rollback')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(self.target.read_bytes(), self.previous)
def test_failed_health_restores_file_from_this_attempt(self):
self.assertEqual(self.deploy('Install').returncode, 0)
installed = self.target.read_bytes()
result = self.deploy('Rollback', TEST_HEALTH_FAIL='1')
self.assertNotEqual(result.returncode, 0)
self.assertIn('Restoring the previous Hub file', result.stderr)
self.assertEqual(self.target.read_bytes(), installed)
def test_ram_database_preflight_does_not_install_or_modify_archive(self):
result = self.deploy('Install', TEST_RAM_DISK='1')
self.assertNotEqual(result.returncode, 0)
self.assertIn('not confirmed on writable persistent disk', result.stderr)
self.assertEqual(self.target.read_bytes(), self.previous)
self.assertFalse(self.backup.exists())
def test_invalid_candidate_and_missing_backup_change_nothing(self):
self.assertNotEqual(self.deploy('Install', invalid=True).returncode, 0)
self.assertFalse(self.backup.exists())
self.assertNotEqual(self.deploy('Rollback').returncode, 0)
self.assertEqual(self.target.read_bytes(), self.previous)
if __name__ == '__main__':
unittest.main()
+1 -1
View File
@@ -131,7 +131,7 @@ class HubFixture:
assert "groups" in json.loads(self.request("/api/router-groups")) assert "groups" in json.loads(self.request("/api/router-groups"))
assert "notifications" in json.loads(self.request("/api/notifications")) assert "notifications" in json.loads(self.request("/api/notifications"))
assert self.request("/access/test-router/") == b"router proxy fixture" assert self.request("/access/test-router/") == b"router proxy fixture"
assert b"v116" in self.request("/") assert b"v117" in self.request("/")
self.app.snapshot_router_states() self.app.snapshot_router_states()
def __exit__(self, *args): def __exit__(self, *args):
+75
View File
@@ -113,6 +113,81 @@ class ArchiveTests(unittest.TestCase):
self.assertIn('новый журнал', self.snapshot()['text']) self.assertIn('новый журнал', self.snapshot()['text'])
self.assertIn('first', self.snapshot()['text']) self.assertIn('first', self.snapshot()['text'])
def test_empty_status_cannot_confirm_missing_records_or_invent_rotation_gap(self):
reply = self.ingest(batch(b'', start=298))
self.assertEqual(reply['offset'], 0)
self.assertEqual(self.snapshot()['text'], '')
raw = b'[OK] retained records arrive after status\n'
self.ingest(batch(raw))
reply = self.ingest(batch(b'', start=1000))
self.assertEqual(reply['offset'], len(raw))
self.assertEqual(self.snapshot()['text'].encode(), raw)
def test_actual_gap_with_data_is_reported_once_but_retries_keep_data(self):
value = batch(b'[OK] surviving records\n', start=50)
self.ingest(value)
self.ingest(value)
text = self.snapshot()['text']
self.assertEqual(text.count('не доставлена'), 1)
self.assertEqual(text.count('surviving records'), 1)
def test_empty_new_generation_cannot_erase_current_journal_cursor(self):
raw = b'[OK] previous generation\n'
self.ingest(batch(raw))
self.ingest(batch(b'', start=2000, journal='new-journal'))
self.ingest(batch(raw))
self.assertEqual(self.snapshot()['text'].encode(), raw)
def test_recovery_generation_replays_previously_falsely_confirmed_chunks(self):
# Reproduce the cursor left by the former empty-batch ingestion bug.
with self.app.conn() as conn:
conn.execute("insert into router_logs(router_id,journal_id,cursor) values ('main','broken',298)")
self.ingest(batch(b'[OK] retained restart event\n', journal='recovered-old'))
self.assertIn('retained restart event', self.snapshot()['text'])
def test_real_router_spool_recovers_bad_digest_and_delivers_after_restart(self):
from test_persistent_logs import PersistentLogsTests
client = PersistentLogsTests()
client.setUp()
self.addCleanup(client.doCleanups)
client.append('Панель: действие=stop; начало\nСервис: остановка туннеля\n')
client.flush()
original = client.export()
client.acknowledge(original); client.flush()
with self.app.conn() as conn:
conn.execute('insert into router_logs(router_id,journal_id,cursor) values (?,?,?)',
('main', original['journal_id'], original['to']))
part = next(client.journal.glob('part.*'))
part.rename(part.with_name(part.name.rsplit('.', 1)[0] + '.'))
(client.run_dir / 'recovered-v2').unlink()
repaired = client.export()
reply = self.ingest(repaired, {'enabled': False, 'xray': 'stopped'})
client.call('ack', '', reply['journal_id'], str(reply['offset']))
# A lost acknowledgment may replay, but the archive must stay unchanged.
self.ingest(repaired)
self.assertEqual(self.snapshot()['text'].count('Сервис: остановка туннеля'), 1)
client.append('Сервис: запуск\n'); client.flush()
second = client.export()
reply = self.ingest(second, {'enabled': True, 'xray': 'running'})
client.call('ack', '', reply['journal_id'], str(reply['offset']))
text = self.snapshot()['text']
self.assertIn('Сервис: запуск', text)
self.assertNotIn('не доставлена до ротации', text)
self.assertEqual(client.export()['data'], '')
def test_disk_mount_diagnostic_resolves_deepest_mount_and_escaped_paths(self):
path = self.app.db_path.resolve()
mount = str(path.parent).replace(' ', r'\040')
root = str(Path(path.anchor)).replace(' ', r'\040')
mountinfo = f'1 0 8:1 / {root} rw - ext4 /dev/vda1 rw\n'
mountinfo += f'2 1 0:1 / {mount} rw - tmpfs tmpfs rw\n'
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo)):
result = hub.router_log_storage(path)
self.assertIs(result['persistent'], False)
self.assertEqual(result['filesystem'], 'tmpfs')
with mock.patch.object(hub.sys, 'platform', 'linux'), mock.patch('builtins.open', mock.mock_open(read_data=mountinfo.splitlines()[0])):
self.assertIs(hub.router_log_storage(path)['persistent'], True)
def test_offline_clear_does_not_restore_old_logs_on_reconnection(self): def test_offline_clear_does_not_restore_old_logs_on_reconnection(self):
old = b'[ERROR] old\n' old = b'[ERROR] old\n'
self.ingest(batch(old)) self.ingest(batch(old))
+3
View File
@@ -35,6 +35,7 @@ const server = http.createServer((req, res) => {
reads++; reads++;
res.setHeader('Content-Type', 'application/json'); res.setHeader('Content-Type', 'application/json');
return res.end(JSON.stringify({ok:true, online:false, text, bytes:Buffer.byteLength(text), last_sync:1791954000, return res.end(JSON.stringify({ok:true, online:false, text, bytes:Buffer.byteLength(text), last_sync:1791954000,
storage:{path:'/var/lib/owrt-remote/hub.db',filesystem:'ext4',persistent:true},
diagnosis:{level:'warn',text:'Remote выключен на роутере. Туннель и Heartbeat остановлены.',at:1791954000}})); diagnosis:{level:'warn',text:'Remote выключен на роутере. Туннель и Heartbeat остановлены.',at:1791954000}}));
} }
if (url.pathname === '/api/router/main/logs/clear') { if (url.pathname === '/api/router/main/logs/clear') {
@@ -95,6 +96,8 @@ const server = http.createServer((req, res) => {
text=initialText();const before=reads; text=initialText();const before=reads;
await page.goto(`http://127.0.0.1:${server.address().port}/router/main/logs`); await page.goto(`http://127.0.0.1:${server.address().port}/router/main/logs`);
await page.waitForFunction(()=>document.querySelector('#logsText').textContent.includes('Event 119')); await page.waitForFunction(()=>document.querySelector('#logsText').textContent.includes('Event 119'));
assert((await page.locator('#logsStorage').textContent()).includes('Архив VPS: постоянный диск.'));
assert((await page.locator('#logsStorage').textContent()).includes('/var/lib/owrt-remote/hub.db'));
assert.deepEqual(await page.locator('body').evaluate(el=>{ assert.deepEqual(await page.locator('body').evaluate(el=>{
const s=getComputedStyle(el);return {image:s.backgroundImage,color:s.backgroundColor,attachment:s.backgroundAttachment}; const s=getComputedStyle(el);return {image:s.backgroundImage,color:s.backgroundColor,attachment:s.backgroundAttachment};
}),terminalBackground,'log page uses the VPS terminal background'); }),terminalBackground,'log page uses the VPS terminal background');
+41 -8
View File
@@ -23,6 +23,8 @@ class RouterLogTransportTests(unittest.TestCase):
self.env = {**self.journal.env, 'OWRT_REMOTE_WAN_HELPER': (ROOT / 'files/usr/lib/owrt-remote/wan.sh').as_posix(), self.env = {**self.journal.env, 'OWRT_REMOTE_WAN_HELPER': (ROOT / 'files/usr/lib/owrt-remote/wan.sh').as_posix(),
'TEST_ROOT': self.journal.root.as_posix(), 'TEST_PYTHON': Path(sys.executable).as_posix()} 'TEST_ROOT': self.journal.root.as_posix(), 'TEST_PYTHON': Path(sys.executable).as_posix()}
self.mocks = r''' self.mocks = r'''
# Windows has no RLIMIT_FSIZE; production ash enforces the actual file limit.
ulimit() { return 0; }
uci_get() { uci_get() {
case "$1" in case "$1" in
enabled) printf '%s' "${TEST_ENABLED:-1}" ;; hub_token) printf test-token ;; router_id) printf main ;; enabled) printf '%s' "${TEST_ENABLED:-1}" ;; hub_token) printf test-token ;; router_id) printf main ;;
@@ -60,6 +62,7 @@ jsonfilter() {
''' '''
def sync(self, **flags): def sync(self, **flags):
self.journal.helper_call('flush', '', 'force')
script = self.source + '\n' + self.mocks + '\nsync_global_logs\n' script = self.source + '\n' + self.mocks + '\nsync_global_logs\n'
return subprocess.run([SH], input=script, encoding='utf-8', capture_output=True, return subprocess.run([SH], input=script, encoding='utf-8', capture_output=True,
env={**self.env, **flags}, timeout=25) env={**self.env, **flags}, timeout=25)
@@ -80,14 +83,34 @@ jsonfilter() {
self.assertEqual('direct' in requests, direct == '1') self.assertEqual('direct' in requests, direct == '1')
self.assertEqual('prepare' in requests, direct == '1') self.assertEqual('prepare' in requests, direct == '1')
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '') self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
self.assertFalse(list(self.journal.log_dir.glob('sync.*'))) self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
def test_failed_delivery_preserves_unsent_records_and_removes_payload_files(self): def test_failed_delivery_preserves_unsent_records_and_removes_payload_files(self):
before = json.loads(self.journal.helper_call('export')) before = json.loads(self.journal.helper_call('export'))
result = self.sync(TEST_HTTP_FAIL='1') result = self.sync(TEST_HTTP_FAIL='1')
self.assertEqual(result.returncode, 1, result.stderr) self.assertEqual(result.returncode, 1, result.stderr)
self.assertEqual(json.loads(self.journal.helper_call('export')), before) self.assertEqual(json.loads(self.journal.helper_call('export')), before)
self.assertFalse(list(self.journal.log_dir.glob('sync.*'))) self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
def test_agent_update_delivers_with_older_helper_without_its_own_runtime_directory(self):
# The pre-persistent helper creates only /tmp/owrt-remote-logs. A WAN
# agent update must not depend on that helper creating the new runtime.
helper = self.journal.root / 'old-helper'
record = base64.b64encode(b'[OK] older helper record\n').decode()
batch = json.dumps({'journal_id': 'older', 'from': 0, 'to': 25, 'data': record})
helper.write_text('#!/bin/sh\ncase "$1" in\ncheck) exit 0 ;;\nexport) printf \'%s\\n\' \'' + batch +
'\' ;;\nack) printf \'%s\\n\' "$2 $3" >"$TEST_ROOT/older-ack" ;;\nesac\n',
encoding='utf-8', newline='\n')
runtime = self.journal.root / 'new-runtime'
result = subprocess.run([SH], input=self.source + '\n' + self.mocks + '\nsync_global_logs\n',
encoding='utf-8', capture_output=True,
env={**self.env, 'OWRT_REMOTE_LOG_HELPER': helper.as_posix(),
'OWRT_REMOTE_LOG_RUNTIME': runtime.as_posix()}, timeout=25)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertTrue(runtime.is_dir())
self.assertTrue((self.journal.root / 'older-ack').is_file())
self.assertFalse((runtime / 'sync-lock').exists())
self.assertFalse(list(runtime.glob('sync.*')))
def test_disabled_remote_still_delivers_stop_reason_and_state_without_restart(self): def test_disabled_remote_still_delivers_stop_reason_and_state_without_restart(self):
self.journal.helper_call('append', 'Сервис: остановка туннеля\n') self.journal.helper_call('append', 'Сервис: остановка туннеля\n')
@@ -97,14 +120,14 @@ jsonfilter() {
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
payload = json.loads((self.journal.root / 'delivered.json').read_text()) payload = json.loads((self.journal.root / 'delivered.json').read_text())
self.assertEqual(payload['state'], {'enabled': False, 'xray': 'stopped'}) self.assertEqual(payload['state'], {'enabled': False, 'xray': 'stopped'})
self.assertFalse(list(self.journal.log_dir.glob('sync.*'))) self.assertFalse(list(self.journal.run_dir.glob('sync.*')))
self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '') self.assertEqual(json.loads(self.journal.helper_call('export'))['data'], '')
def test_overlapping_delivery_does_not_start_second_sender(self): def test_overlapping_delivery_does_not_start_second_sender(self):
lock = self.journal.log_dir / 'sync-lock' lock = self.journal.run_dir / 'sync-lock'
lock.mkdir() lock.mkdir()
# The shell's own PID is alive for the duration of sync_global_logs. # The shell's own PID is alive for the duration of sync_global_logs.
script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_DIR/sync-lock/pid"\nsync_global_logs\n' script = self.source + '\n' + self.mocks + '\nprintf "%s" "$$" >"$OWRT_REMOTE_LOG_RUNTIME/sync-lock/pid"\nsync_global_logs\n'
result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True, env=self.env, timeout=20) result = subprocess.run([SH], input=script, encoding='utf-8', capture_output=True, env=self.env, timeout=20)
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((self.journal.root / 'requests').exists()) self.assertFalse((self.journal.root / 'requests').exists())
@@ -113,15 +136,25 @@ jsonfilter() {
def test_wan_failure_releases_delivery_lock_for_next_attempt(self): def test_wan_failure_releases_delivery_lock_for_next_attempt(self):
result = self.sync(TEST_DIRECT='1', TEST_WAN_FAIL='1', TEST_ENABLED='0') result = self.sync(TEST_DIRECT='1', TEST_WAN_FAIL='1', TEST_ENABLED='0')
self.assertEqual(result.returncode, 1, result.stderr) self.assertEqual(result.returncode, 1, result.stderr)
self.assertFalse((self.journal.log_dir / 'sync-lock').exists()) self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
self.assertNotEqual(json.loads(self.journal.helper_call('export'))['data'], '') self.assertNotEqual(json.loads(self.journal.helper_call('export'))['data'], '')
self.assertEqual(self.sync(TEST_DIRECT='1', TEST_ENABLED='0').returncode, 0) self.assertEqual(self.sync(TEST_DIRECT='1', TEST_ENABLED='0').returncode, 0)
def test_abandoned_empty_delivery_lock_is_recovered(self): def test_abandoned_empty_delivery_lock_is_recovered(self):
(self.journal.log_dir / 'sync-lock').mkdir() (self.journal.run_dir / 'sync-lock').mkdir()
result = self.sync(TEST_ENABLED='0') result = self.sync(TEST_ENABLED='0')
self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((self.journal.log_dir / 'sync-lock').exists()) self.assertFalse((self.journal.run_dir / 'sync-lock').exists())
self.assertTrue((self.journal.root / 'delivered.json').exists())
self.assertFalse(list(self.journal.run_dir.glob('http/owrt-remote-http.*')))
def test_abandoned_delivery_lock_reaper_is_cleaned_for_next_attempt(self):
lock = self.journal.run_dir / 'sync-lock'
lock.mkdir(); (lock / 'pid').write_text('99999999')
(lock / 'reap').mkdir(); (lock / 'reap/pid').write_text('99999998')
self.assertEqual(self.sync().returncode, 0)
self.assertEqual(self.sync().returncode, 0)
self.assertFalse(lock.exists())
self.assertTrue((self.journal.root / 'delivered.json').exists()) self.assertTrue((self.journal.root / 'delivered.json').exists())
def test_panel_stop_sends_completed_result_without_waiting_for_restart(self): def test_panel_stop_sends_completed_result_without_waiting_for_restart(self):
+1 -1
View File
@@ -35,7 +35,7 @@ class WanDnsDeploymentTests(unittest.TestCase):
source = (ROOT / "vps/deploy-wan-dns-fix.ps1").read_text(encoding="utf-8") source = (ROOT / "vps/deploy-wan-dns-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("target=/usr/lib/owrt-remote/wan.sh", "target=" + shlex.quote(self.target.as_posix())) self.script = self.script.replace("target=/usr/lib/owrt-remote/wan.sh", "target=" + shlex.quote(self.target.as_posix()))
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v116", self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-dns-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix())) "backup_dir=" + shlex.quote(self.backup.as_posix()))
def deploy(self, action="Install", bad=False, fail_verify=False, fail_signal=False): def deploy(self, action="Install", bad=False, fail_verify=False, fail_signal=False):
+1 -1
View File
@@ -129,7 +129,7 @@ class WanFailoverTests(unittest.TestCase):
self.prepare() self.prepare()
config = self.directory / "xray.json" config = self.directory / "xray.json"
config.write_text(json.dumps({"outbounds": [{"tag": "vps-interconn", "streamSettings": {"sockopt": {"interface": "eth0"}}}]})) config.write_text(json.dumps({"outbounds": [{"tag": "vps-interconn", "streamSettings": {"sockopt": {"interface": "eth0"}}}]}))
self.config(wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix()) self.config(enabled="1", wan_interface="wan wwan", wan_direct="1", vps_host="192.0.2.10", xray_config=config.as_posix())
result = self.shell(''' result = self.shell('''
xray_running() { return 0; } xray_running() { return 0; }
wan_reconnect_xray() { printf '%s -> %s\n' "$1" "$2" >>"$TEST_ROOT/reconnect"; } wan_reconnect_xray() { printf '%s -> %s\n' "$1" "$2" >>"$TEST_ROOT/reconnect"; }
+208
View File
@@ -0,0 +1,208 @@
"""WAN outage/recovery using the real agent shell and a simulated procd."""
import json
import os
import subprocess
import unittest
import test_host_policy as base
from test_wan_failover import UPLINK_MOCKS, JSONFILTER
@unittest.skipUnless(base.SH, "POSIX shell required")
class WanRecoveryTests(unittest.TestCase):
setUp = base.RouterShellTests.setUp
config = base.RouterShellTests.config
def shell(self, commands):
shell = os.environ.get("OWRT_RECOVERY_TEST_SHELL", base.SH)
args = [shell, "ash"] if "busybox" in shell.lower() else [shell]
return subprocess.run(args, input=self.agent + "\n" + base.MOCKS + "\n" + UPLINK_MOCKS + "\n" + commands,
encoding="utf-8", capture_output=True, cwd=base.ROOT, env=self.env, timeout=40)
def prepare(self, enabled="1"):
self.config(enabled=enabled, wan_direct="1", wan_interface="wan", vps_host="192.0.2.10",
hub_url="https://hub.example", hub_token="secret-test-token")
self.directory.joinpath("jsonfilter.py").write_text(JSONFILTER, encoding="utf-8")
self.directory.joinpath("service.json").write_text(json.dumps({"owrt-remote": {"instances": {
"heartbeat": {"running": True, "pid": 123}, "xray": {"running": False}
}}}), encoding="utf-8")
PROCD = r'''
xray_running() { return 1; }
ubus() {
case "$1 $2 $3" in
'call service list') cat "$TEST_ROOT/service.json" ;;
'call service add')
printf '%s\n' "$4" >>"$TEST_ROOT/add-calls"
[ ! -e "$TEST_ROOT/add-fail" ] ;;
*) return 1 ;;
esac
}
jsonfilter() {
"$TEST_PYTHON" -c 'import json,sys
state=json.load(sys.stdin)
instance="heartbeat" if ".heartbeat." in sys.argv[1] else "xray"
value=state.get("owrt-remote",{}).get("instances",{}).get(instance,{}).get("running","")
print(str(value).lower())' "$2"
}
wan_select_interface() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf wan; }
wan_device() { [ ! -e "$TEST_ROOT/wan-down" ] || return 1; printf eth0; }
'''
def test_online_heartbeat_after_outage_recovers_dead_tunnel_on_same_uplink_once(self):
self.prepare()
result = self.shell(self.PROCD + r'''
touch "$TEST_ROOT/wan-down"
wan_reconcile_xray || true
test ! -e "$TEST_ROOT/add-calls" || exit 1
rm "$TEST_ROOT/wan-down"
wan_reconcile_xray || exit
wan_reconcile_xray
''')
self.assertEqual(result.returncode, 0, result.stderr)
requests = self.directory.joinpath("add-calls").read_text().splitlines()
self.assertEqual(len(requests), 1)
request = json.loads(requests[0])
self.assertEqual(set(request["instances"]), {"xray"})
self.assertEqual(request["instances"]["xray"]["command"], ["/usr/sbin/owrt-remote", "run"])
self.assertEqual(request["instances"]["xray"]["respawn"], ["3600", "5", "5"])
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
self.assertNotIn("secret-test-token", result.stderr)
def test_initial_start_waiting_for_wan_is_left_to_procd(self):
self.prepare()
state = self.directory / "service.json"
data = json.loads(state.read_text())
data["owrt-remote"]["instances"]["xray"]["running"] = True
state.write_text(json.dumps(data))
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_successful_heartbeat_recovers_dead_xray_and_still_returns_hub_response(self):
self.prepare()
result = self.shell(self.PROCD + r'''
wan_prepare() { return 0; }
heartbeat_json() { printf '{}'; }
heartbeat_candidate_endpoints() { printf 'https://hub.example/api/heartbeat'; }
post_json() { printf '{"ok":true}'; }
heartbeat_apply_hub_update() { return 0; }
send_heartbeat
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(json.loads(result.stdout), {"ok": True})
request = json.loads(self.directory.joinpath("add-calls").read_text())
self.assertEqual(set(request["instances"]), {"xray"})
self.assertIn("Heartbeat: OK", result.stderr)
def test_healthy_xray_is_not_restarted_when_wan_returns(self):
self.prepare()
result = self.shell(self.PROCD + "xray_running() { return 0; }; wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_panel_stop_and_external_service_stop_are_respected(self):
self.prepare(enabled="0")
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
self.prepare()
self.directory.joinpath("service.json").write_text('{}')
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("add-calls").exists())
def test_parallel_requests_and_stale_lock_produce_one_recovery(self):
self.prepare()
lock = self.directory / "owrt-remote-xray-recovery"
lock.mkdir()
lock.joinpath("pid").write_text("2147483000")
result = self.shell(self.PROCD + "wan_reconcile_xray & wan_reconcile_xray & wait")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(lock.exists())
def test_ubus_failure_reports_error_cleans_lock_and_honors_backoff(self):
self.prepare()
self.directory.joinpath("add-fail").touch()
result = self.shell(self.PROCD + "wan_reconcile_xray; rc=$?; wan_reconcile_xray; exit $rc")
self.assertEqual(result.returncode, 1, result.stderr)
self.assertIn("procd", result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(self.directory.joinpath("owrt-remote-xray-recovery").exists())
def test_empty_lock_and_dead_reaper_do_not_block_later_recovery(self):
self.prepare()
lock = self.directory / "owrt-remote-xray-recovery"
lock.mkdir()
(lock / "reap").mkdir()
(lock / "reap/pid").write_text("2147483000")
result = self.shell(self.PROCD + "wan_reconcile_xray")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(len(self.directory.joinpath("add-calls").read_text().splitlines()), 1)
self.assertFalse(lock.exists())
def test_missing_wan_does_not_cause_repeated_service_restarts(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
result = self.shell(self.PROCD + r'''
active_ssh_session() { return 1; }
heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal; heartbeat_self_heal
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse(self.directory.joinpath("owrt-remote-heartbeat.restart").exists())
self.assertIn("IPv4 default route", result.stderr)
def test_supervised_start_waits_beyond_six_retries_then_launches_with_valid_config(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
executable = self.directory / "fake-xray"
executable.write_text('#!/bin/sh\n[ "$1" != help ] || { echo run; exit; }; printf "%s\\n" "$*" >"$TEST_ROOT/xray-called"\n', newline="\n")
executable.chmod(0o755)
config = self.directory / "valid-config.json"
config.write_text('{"outbounds":[]}')
result = self.shell(r'''
sleep() {
local n
n=$(cat "$TEST_ROOT/waits" 2>/dev/null || echo 0)
n=$((n+1)); printf '%s' "$n" >"$TEST_ROOT/waits"
[ "$n" -lt 8 ] || rm -f "$TEST_ROOT/wan-down"
}
ensure_xray_available_with_retry() { return 0; }
xray_bin_path() { printf '%s' "$TEST_ROOT/fake-xray"; }
render_client_config() { printf '%s' "$TEST_ROOT/valid-config.json"; }
run_xray
''')
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(self.directory.joinpath("waits").read_text(), "8")
self.assertEqual(self.directory.joinpath("xray-called").read_text().strip(), 'run -config ' + config.as_posix())
self.assertEqual(result.stderr.count("конфигурация сохранена"), 1)
self.assertIn("продолжается запуск", result.stderr)
def test_disabling_service_cancels_wait_without_rewriting_configuration(self):
self.prepare()
self.directory.joinpath("wan-down").touch()
result = self.shell(r'''
sleep() { uci set owrtremote.main.enabled=0; }
render_client_config() { touch "$TEST_ROOT/rendered"; }
run_xray
''')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertFalse(self.directory.joinpath("rendered").exists())
def test_configuration_error_with_available_wan_is_not_retried_as_wan_outage(self):
self.prepare()
result = self.shell(r'''
sleep() { touch "$TEST_ROOT/slept"; }
ensure_xray_available_with_retry() { return 0; }
xray_bin_path() { printf /usr/bin/xray; }
render_client_config() { printf 'invalid configuration\n' >&2; return 7; }
run_xray
''')
self.assertEqual(result.returncode, 1, result.stderr)
self.assertIn("invalid configuration", result.stderr)
self.assertFalse(self.directory.joinpath("slept").exists())
if __name__ == "__main__":
unittest.main()
+122
View File
@@ -0,0 +1,122 @@
"""Run installation and rollback in a sandbox, without SSH or services."""
import os
from pathlib import Path
import shlex
import subprocess
import tempfile
import unittest
from test_xray_mobile_deploy import SH, GIT_BIN, ROOT
@unittest.skipUnless(SH, "POSIX shell required")
class WanRecoveryDeploymentTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix="wan-recovery-deploy-")
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.targets = {"wan.sh": self.root / "wan.sh", "agent": self.root / "agent"}
for name, path in self.targets.items():
path.write_text("#!/bin/sh\necho previous-" + name + "\n", encoding="utf-8", newline="\n")
self.original = {name: path.read_bytes() for name, path in self.targets.items()}
self.backup = self.root / "backup"
self.env = dict(os.environ)
if os.name == "nt":
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-wan-recovery-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-wan-recovery-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /usr/lib/owrt-remote/wan.sh", "printf %s " + shlex.quote(self.targets["wan.sh"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
def deploy(self, action, bad_item=None, fail_verify=False, fail_signal=False, stopped=False):
stage = self.root / "stage"
stage.mkdir()
for item, source in {"wan.sh": "files/usr/lib/owrt-remote/wan.sh", "agent": "files/usr/sbin/owrt-remote"}.items():
text = (ROOT / source).read_text(encoding="utf-8")
if bad_item == item:
text = "#!/bin/sh\nif true; then\n"
stage.joinpath(item).write_text(text, encoding="utf-8", newline="\n")
script = self.script.replace("stage=/tmp/__UPLOAD_NAME__", "stage=" + shlex.quote(stage.as_posix()))
script = script.replace("__ACTION__", action).replace("__UPLOAD_NAME__", "sandbox")
if fail_verify:
script = script.replace('cmp "$source_dir/$item" "$path"', "false")
mocks = r'''
ubus() {
printf '%s\n' "$*" >>"$TEST_ROOT/ubus-calls"
if [ "$3" = signal ]; then
[ "${TEST_SIGNAL_FAIL:-0}" = 0 ] || return 1
printf 202 >"$TEST_ROOT/pid"
fi
printf '{}'
}
jsonfilter() { cat >/dev/null; cat "$TEST_ROOT/pid"; }
sleep() { :; }
'''
self.root.joinpath("pid").write_text("" if stopped else "101")
result = subprocess.run([SH], input=mocks + script, encoding="utf-8", capture_output=True,
env={**self.env, "TEST_ROOT": self.root.as_posix(), "TEST_SIGNAL_FAIL": "1" if fail_signal else "0"}, timeout=30)
self.assertFalse(stage.exists())
self.assertFalse(list(self.root.glob("*.new-sandbox")))
return result
def test_install_twice_then_rollback_preserves_first_backup(self):
for attempt in range(2):
result = self.deploy("Install")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("WAN_RECOVERY_ROUTER_Install_OK", result.stdout)
for item, path in self.targets.items():
self.assertNotEqual(path.read_bytes(), self.original[item])
self.assertEqual(self.backup.joinpath(item).read_bytes(), self.original[item])
result = self.deploy("Rollback")
self.assertEqual(result.returncode, 0, result.stderr)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
def test_bad_candidate_and_missing_backup_leave_paired_files_unchanged(self):
for item in ("agent", "wan.sh"):
result = self.deploy("Install", bad_item=item)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(self.backup.exists())
for name, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[name])
result = self.deploy("Rollback")
self.assertNotEqual(result.returncode, 0)
self.assertIn("No complete backup", result.stderr)
for name, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[name])
def test_reloads_only_heartbeat_without_touching_xray(self):
result = self.deploy("Install")
self.assertEqual(result.returncode, 0, result.stderr)
calls = self.root.joinpath("ubus-calls").read_text()
self.assertIn('service signal {"name":"owrt-remote","instance":"heartbeat","signal":9}', calls)
self.assertNotIn('"instance":"xray"', calls)
self.assertIn('Heartbeat loop reloaded', result.stdout)
def test_stopped_loop_stays_stopped(self):
result = self.deploy("Install", stopped=True)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertNotIn('service signal', self.root.joinpath("ubus-calls").read_text())
self.assertIn('Heartbeat loop is stopped', result.stdout)
def test_failed_loop_reload_restores_paired_files_and_reports_failure(self):
result = self.deploy("Install", fail_signal=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn('Failed during heartbeat-reload', result.stderr)
self.assertNotIn('WAN_RECOVERY_ROUTER_Install_OK', result.stdout)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
def test_verify_failure_restores_paired_files(self):
result = self.deploy("Install", fail_verify=True)
self.assertNotEqual(result.returncode, 0)
self.assertIn("Restoring the agent and WAN helper", result.stderr)
for item, path in self.targets.items():
self.assertEqual(path.read_bytes(), self.original[item])
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -31,7 +31,7 @@ class WanStabilityRollbackTests(unittest.TestCase):
source = (ROOT / "vps/deploy-wan-stability-fix.ps1").read_text(encoding="utf-8") source = (ROOT / "vps/deploy-wan-stability-fix.ps1").read_text(encoding="utf-8")
script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
script = script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(self.stage.as_posix())) script = script.replace('stage=/tmp/__UPLOAD_NAME__', 'stage=' + shlex.quote(self.stage.as_posix()))
script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v116', 'backup=' + shlex.quote(self.backup.as_posix())) script = script.replace('backup=/root/owrt-remote-backup-before-wan-stability-v117', 'backup=' + shlex.quote(self.backup.as_posix()))
script = script.replace('config=/etc/config/owrtremote', 'config=' + shlex.quote(self.files["config"].as_posix())) script = script.replace('config=/etc/config/owrtremote', 'config=' + shlex.quote(self.files["config"].as_posix()))
for name, original in (("agent", "/usr/sbin/owrt-remote"), ("wan.sh", "/usr/lib/owrt-remote/wan.sh")): for name, original in (("agent", "/usr/sbin/owrt-remote"), ("wan.sh", "/usr/lib/owrt-remote/wan.sh")):
script = script.replace('printf ' + original + ' ;;', "printf '%s' " + shlex.quote(self.files[name].as_posix()) + ' ;;') script = script.replace('printf ' + original + ' ;;', "printf '%s' " + shlex.quote(self.files[name].as_posix()) + ' ;;')
+1 -1
View File
@@ -29,7 +29,7 @@ class XrayMobileDeploymentTests(unittest.TestCase):
source = (ROOT / "vps/deploy-xray-mobile-fix.ps1").read_text(encoding="utf-8") source = (ROOT / "vps/deploy-xray-mobile-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("target=/www/cgi-bin/owrt-remote", "target=" + shlex.quote(self.target.as_posix())) self.script = self.script.replace("target=/www/cgi-bin/owrt-remote", "target=" + shlex.quote(self.target.as_posix()))
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v116", self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-mobile-v117",
"backup_dir=" + shlex.quote(self.backup_dir.as_posix())) "backup_dir=" + shlex.quote(self.backup_dir.as_posix()))
def deploy(self, action, content=None, fail_verify=False): def deploy(self, action, content=None, fail_verify=False):
+1 -1
View File
@@ -25,7 +25,7 @@ class XrayValidationDeploymentTests(unittest.TestCase):
self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"] self.env["PATH"] = str(GIT_BIN) + os.pathsep + self.env["PATH"]
source = (ROOT / "vps/deploy-xray-validation-fix.ps1").read_text(encoding="utf-8") source = (ROOT / "vps/deploy-xray-validation-fix.ps1").read_text(encoding="utf-8")
self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0] self.script = source.split("$RemoteScript = @'\n", 1)[1].split("\n'@", 1)[0]
self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v116", self.script = self.script.replace("backup_dir=/root/owrt-remote-backup-before-xray-validation-v117",
"backup_dir=" + shlex.quote(self.backup.as_posix())) "backup_dir=" + shlex.quote(self.backup.as_posix()))
self.script = self.script.replace("printf /www/cgi-bin/owrt-remote", "printf %s " + shlex.quote(self.targets["cgi"].as_posix())) self.script = self.script.replace("printf /www/cgi-bin/owrt-remote", "printf %s " + shlex.quote(self.targets["cgi"].as_posix()))
self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix())) self.script = self.script.replace("printf /usr/sbin/owrt-remote", "printf %s " + shlex.quote(self.targets["agent"].as_posix()))
+2 -2
View File
@@ -15,7 +15,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-global-logs-ui-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-global-logs-ui-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
@@ -107,7 +107,7 @@ if [ "$healthy" -ne 1 ]; then
exit 1 exit 1
fi fi
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nGLOBAL_LOGS_VPS_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nGLOBAL_LOGS_VPS_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
+4 -3
View File
@@ -13,7 +13,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-global-logs-stop-v116 backup=/root/owrt-remote-backup-before-global-logs-stop-v117
items='cgi agent init logs.sh logs-init luci' items='cgi agent init logs.sh logs-init luci'
changed=0 changed=0
phase=preflight phase=preflight
@@ -104,7 +104,7 @@ if [ '__ACTION__' = Install ]; then
command -v base64 >/dev/null command -v base64 >/dev/null
command -v jsonfilter >/dev/null command -v jsonfilter >/dev/null
grep -F -q 'globalLogsToggle' "$stage/cgi" grep -F -q 'globalLogsToggle' "$stage/cgi"
for item in cgi luci; do grep -F -q 'v116' "$stage/$item"; done for item in cgi luci; do grep -F -q 'v117' "$stage/$item"; done
grep -F -q 'LOG_KEEP_SECONDS=86400' "$stage/logs.sh" grep -F -q 'LOG_KEEP_SECONDS=86400' "$stage/logs.sh"
grep -F -q 'log_sync_loop' "$stage/logs.sh" grep -F -q 'log_sync_loop' "$stage/logs.sh"
grep -F -q 'sync_global_logs' "$stage/agent" grep -F -q 'sync_global_logs' "$stage/agent"
@@ -131,7 +131,8 @@ if [ '__ACTION__' = Install ]; then
sleep 1 sleep 1
done done
if [ "$ready" != 1 ]; then echo 'Log collector did not start.' >&2; logread -e owrt-remote | tail -n 30 >&2; exit 1; fi if [ "$ready" != 1 ]; then echo 'Log collector did not start.' >&2; logread -e owrt-remote | tail -n 30 >&2; exit 1; fi
marker='Глобал логи: collector-__UPLOAD_NAME__' # Generate UTF-8 on the router; Windows PowerShell 5.1 reads scripts as ANSI.
marker="$(printf '\320\223\320\273\320\276\320\261\320\260\320\273\040\320\273\320\276\320\263\320\270\072') collector-__UPLOAD_NAME__"
ready=0 ready=0
for attempt in 1 2 3 4 5; do for attempt in 1 2 3 4 5; do
logger -t owrt-remote "$marker" logger -t owrt-remote "$marker"
+149
View File
@@ -0,0 +1,149 @@
param(
[Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()][string]$Vps,
[ValidateSet('Install', 'Rollback')][string]$Action = 'Install',
[switch]$PrepareOnly
)
$ErrorActionPreference = 'Stop'
$RepoRoot = Split-Path -Parent $PSScriptRoot
$UploadName = 'owrt-group-colors-v117-' + [guid]::NewGuid().ToString('N')
$StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName
$Utf8 = [System.Text.UTF8Encoding]::new($false)
$RemoteScript = @'
#!/bin/sh
set -eu
stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-group-colors-v117
candidate=$target.new-__UPLOAD_NAME__
changed=0
show_failure() {
echo '--- owrt-remote status ---' >&2
systemctl --no-pager -l status owrt-remote >&2 || true
echo '--- owrt-remote journal ---' >&2
journalctl -u owrt-remote -n 80 --no-pager -l >&2 || true
}
cleanup() {
code=$?
trap - EXIT
if [ "$code" -ne 0 ]; then
echo "Remote step failed with rc=$code" >&2
show_failure
fi
if [ "$code" -ne 0 ] && [ "$changed" = 1 ] && [ -s "$stage/previous.py" ]; then
echo 'Restoring the previous Hub file...' >&2
cp -p "$stage/previous.py" "$candidate"
mv -f "$candidate" "$target"
systemctl restart owrt-remote || true
fi
rm -f "$candidate"
rm -rf "$stage"
exit "$code"
}
trap cleanup EXIT
test -s "$target"
python_bin=/opt/owrt-remote/venv/bin/python
[ -x "$python_bin" ] || python_bin=python3
if [ '__ACTION__' = Install ]; then
test -s "$stage/hub.py"
echo 'Validating candidate...'
"$python_bin" -m py_compile "$stage/hub.py"
"$python_bin" - "$stage/hub.py" <<'PY'
from pathlib import Path
import sys
source = Path(sys.argv[1]).read_text(encoding='utf-8')
required = ('v117', '#routerForm select option,#routerGroupFilter option', 'background-color:#221a34', 'id="backupDownload"', 'id="backupRestore"')
if 'class="backupCmds"' in source:
print('Old backup command instructions remain in candidate', file=sys.stderr)
raise SystemExit(1)
if 'v' + '116' in source:
print('Old UI version remains in candidate', file=sys.stderr)
raise SystemExit(1)
missing = [marker for marker in required if marker not in source]
if missing:
for marker in missing:
print('Missing marker:', repr(marker), file=sys.stderr)
raise SystemExit(1)
print('Candidate validation OK')
PY
if [ ! -e "$backup" ]; then
cp -p "$target" "$backup"
fi
source_file="$stage/hub.py"
else
test -s "$backup"
"$python_bin" -m py_compile "$backup"
source_file="$backup"
fi
cp -p "$target" "$stage/previous.py"
cp "$source_file" "$candidate"
chmod 0755 "$candidate"
changed=1
mv -f "$candidate" "$target"
echo 'Restarting owrt-remote...'
if ! systemctl restart owrt-remote; then
show_failure
exit 1
fi
healthy=0
for attempt in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
if systemctl is-active --quiet owrt-remote && curl -fsS --max-time 5 http://127.0.0.1:8088/health; then
healthy=1
break
fi
sleep 2
done
if [ "$healthy" -ne 1 ]; then
echo 'Health check did not become ready.' >&2
show_failure
exit 1
fi
cmp "$source_file" "$target"
printf '\nBackup: %s\nGROUP_COLORS_V117_%s_OK\n' "$backup" '__ACTION__'
'@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
try {
[System.IO.Directory]::CreateDirectory($StageRoot) | Out-Null
[System.IO.File]::WriteAllText(
(Join-Path $StageRoot 'run.sh'),
$RemoteScript.Replace("`r`n", "`n") + "`n",
$Utf8
)
if ($Action -eq 'Install') {
$Content = [System.IO.File]::ReadAllText((Join-Path $RepoRoot 'vps\owrt-remote-hub.py')).Replace("`r`n", "`n")
[System.IO.File]::WriteAllText((Join-Path $StageRoot 'hub.py'), $Content, $Utf8)
}
if ($PrepareOnly) {
Write-Host "Prepared only: $StageRoot"
return
}
& scp -O -r -- $StageRoot "${Vps}:/tmp/"
if ($LASTEXITCODE -ne 0) { throw 'Upload failed. VPS was not changed.' }
& ssh -T -o ConnectTimeout=10 $Vps "sh /tmp/$UploadName/run.sh"
if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." }
Write-Host "Group colors fix $Action finished successfully. Refresh the Hub panel with Ctrl+F5."
}
finally {
if (-not $PrepareOnly) {
$Resolved = [System.IO.Path]::GetFullPath($StageRoot)
$Expected = [System.IO.Path]::GetFullPath((Join-Path ([System.IO.Path]::GetTempPath()) $UploadName))
if ($Resolved -ne $Expected -or (Split-Path -Leaf $Resolved) -ne $UploadName) {
throw 'Unsafe staging cleanup path.'
}
if (Test-Path -LiteralPath $Resolved) {
Remove-Item -LiteralPath $Resolved -Recurse -Force
}
}
}
+1 -1
View File
@@ -13,7 +13,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-heartbeat-redirect-v116 backup=/root/owrt-remote-backup-before-heartbeat-redirect-v117
agent=/usr/sbin/owrt-remote agent=/usr/sbin/owrt-remote
wan=/usr/lib/owrt-remote/wan.sh wan=/usr/lib/owrt-remote/wan.sh
changed=0 changed=0
+1 -1
View File
@@ -17,7 +17,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v116 backup_dir=/root/owrt-remote-backup-before-heartbeat-transport-v117
changed=0 changed=0
phase=preflight phase=preflight
item_path() { item_path() {
+9 -9
View File
@@ -5,7 +5,7 @@ param(
[string]$PinnedVpsHost = '193.233.82.38', [string]$PinnedVpsHost = '193.233.82.38',
[ValidatePattern('^[a-zA-Z0-9_-]+$')][string]$WanInterface = 'wan', [ValidatePattern('^[a-zA-Z0-9_-]+$')][string]$WanInterface = 'wan',
[switch]$PreserveConnectionSettings, [switch]$PreserveConnectionSettings,
[ValidatePattern('^[a-zA-Z0-9_-]+$')][string]$BackupName = 'host-policy-v116', [ValidatePattern('^[a-zA-Z0-9_-]+$')][string]$BackupName = 'host-policy-v117',
[switch]$PrepareOnly [switch]$PrepareOnly
) )
@@ -90,7 +90,7 @@ test -s "$config"
if [ '__ACTION__' = Install ]; then if [ '__ACTION__' = Install ]; then
for item in agent wan.sh cgi init; do test -s "$stage/$item"; sh -n "$stage/$item"; done for item in agent wan.sh cgi init; do test -s "$stage/$item"; sh -n "$stage/$item"; done
test -s "$stage/luci" test -s "$stage/luci"
for item in cgi luci; do grep -F -q 'v116' "$stage/$item"; done for item in cgi luci; do grep -F -q 'v117' "$stage/$item"; done
# curl is needed to preserve HTTPS SNI and bypass the local DNS/proxy. # curl is needed to preserve HTTPS SNI and bypass the local DNS/proxy.
if ! command -v curl >/dev/null 2>&1; then if ! command -v curl >/dev/null 2>&1; then
if command -v opkg >/dev/null 2>&1; then opkg update && opkg install curl ca-bundle if command -v opkg >/dev/null 2>&1; then opkg update && opkg install curl ca-bundle
@@ -156,16 +156,16 @@ if [ '__ACTION__' = Install ]; then
exit 1 exit 1
fi fi
/usr/sbin/owrt-remote status /usr/sbin/owrt-remote status
grep -F -q 'v116' /www/cgi-bin/owrt-remote grep -F -q 'v117' /www/cgi-bin/owrt-remote
grep -F -q 'v116' /www/luci-static/resources/view/owrt_remote.js grep -F -q 'v117' /www/luci-static/resources/view/owrt_remote.js
echo HOST_POLICY_ROUTER_V116_OK echo HOST_POLICY_ROUTER_V117_OK
else else
test -f "$backup/complete" test -f "$backup/complete"
save_files "$stage/previous" save_files "$stage/previous"
changed=1 changed=1
restore_files "$backup" restore_files "$backup"
/usr/sbin/owrt-remote status /usr/sbin/owrt-remote status
echo HOST_POLICY_ROUTER_V116_ROLLBACK_OK echo HOST_POLICY_ROUTER_V117_ROLLBACK_OK
fi fi
'@ '@
} else { } else {
@@ -180,7 +180,7 @@ fi
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-host-policy-v117
changed=0 changed=0
cleanup() { cleanup() {
code=$? code=$?
@@ -200,7 +200,7 @@ test -s "$target"
python_bin=/opt/owrt-remote/venv/bin/python python_bin=/opt/owrt-remote/venv/bin/python
[ -x "$python_bin" ] || python_bin=python3 [ -x "$python_bin" ] || python_bin=python3
if [ '__ACTION__' = Install ]; then if [ '__ACTION__' = Install ]; then
grep -F -q 'v116' "$stage/hub.py" grep -F -q 'v117' "$stage/hub.py"
"$python_bin" -m py_compile "$stage/hub.py" "$python_bin" -m py_compile "$stage/hub.py"
OWRT_HOST_POLICY_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/test_host_policy.py" HostPolicyTests -v OWRT_HOST_POLICY_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/test_host_policy.py" HostPolicyTests -v
OWRT_RESOURCE_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/resources.py" -v OWRT_RESOURCE_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/resources.py" -v
@@ -226,7 +226,7 @@ for attempt in 1 2 3 4 5 6 7 8 9 10; do
done done
test "$healthy" = 1 test "$healthy" = 1
cmp "$candidate" "$target" cmp "$candidate" "$target"
printf '\nHOST_POLICY_VPS_V116_%s_OK\n' '__ACTION__' printf '\nHOST_POLICY_VPS_V117_%s_OK\n' '__ACTION__'
'@ '@
} }
+6 -6
View File
@@ -34,7 +34,7 @@ if ($Target -eq 'Router') {
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-hub-url-v116 backup=/root/owrt-remote-backup-before-hub-url-v117
chmod 700 "$stage" chmod 700 "$stage"
config=/etc/config/owrtremote config=/etc/config/owrtremote
changed=0 changed=0
@@ -99,7 +99,7 @@ if [ '__ACTION__' = Install ]; then
grep -F -q 'heartbeat_set_url_if_changed()' "$stage/agent" grep -F -q 'heartbeat_set_url_if_changed()' "$stage/agent"
grep -F -q 'reason=https-downgrade' "$stage/agent" grep -F -q 'reason=https-downgrade' "$stage/agent"
for item in agent wan.sh; do grep -F -q -- '--post301' "$stage/$item"; done for item in agent wan.sh; do grep -F -q -- '--post301' "$stage/$item"; done
for item in cgi luci; do grep -F -q 'v116' "$stage/$item"; done for item in cgi luci; do grep -F -q 'v117' "$stage/$item"; done
if [ ! -e "$backup" ]; then if [ ! -e "$backup" ]; then
save_files "$stage/retained-backup" save_files "$stage/retained-backup"
mv "$stage/retained-backup" "$backup" mv "$stage/retained-backup" "$backup"
@@ -138,7 +138,7 @@ else
changed=1 changed=1
restore_files "$backup" restore_files "$backup"
fi fi
printf '\nBackup: %s\nHUB_URL_ROUTER_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nHUB_URL_ROUTER_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
} else { } else {
$Payloads = @{ $Payloads = @{
@@ -152,7 +152,7 @@ set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
chmod 700 "$stage" chmod 700 "$stage"
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-hub-url-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
@@ -174,7 +174,7 @@ python_bin=/opt/owrt-remote/venv/bin/python
[ -x "$python_bin" ] || python_bin=python3 [ -x "$python_bin" ] || python_bin=python3
if [ '__ACTION__' = Install ]; then if [ '__ACTION__' = Install ]; then
grep -F -q 'def router_hub_origins(' "$stage/hub.py" grep -F -q 'def router_hub_origins(' "$stage/hub.py"
grep -F -q 'v116' "$stage/hub.py" grep -F -q 'v117' "$stage/hub.py"
"$python_bin" -m py_compile "$stage/hub.py" "$python_bin" -m py_compile "$stage/hub.py"
OWRT_HOST_POLICY_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/host_policy.py" HostPolicyTests -v OWRT_HOST_POLICY_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/host_policy.py" HostPolicyTests -v
OWRT_RESOURCE_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/resources.py" IntegrationTests -v OWRT_RESOURCE_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/resources.py" IntegrationTests -v
@@ -198,7 +198,7 @@ for attempt in 1 2 3 4 5 6 7 8 9 10; do
done done
test "$healthy" = 1 test "$healthy" = 1
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nHUB_URL_VPS_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nHUB_URL_VPS_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
} }
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action).Replace('__HUB_URL__', $HubUrl) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action).Replace('__HUB_URL__', $HubUrl)
+2 -2
View File
@@ -14,7 +14,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-online-status-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-online-status-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
@@ -59,7 +59,7 @@ for attempt in 1 2 3 4 5 6 7 8 9 10; do
done done
test "$healthy" = 1 test "$healthy" = 1
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nONLINE_STATUS_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nONLINE_STATUS_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
try { try {
+2 -2
View File
@@ -15,7 +15,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-password-logout-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-password-logout-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
@@ -105,7 +105,7 @@ if [ "$healthy" -ne 1 ]; then
exit 1 exit 1
fi fi
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nPASSWORD_LOGOUT_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nPASSWORD_LOGOUT_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
+187
View File
@@ -0,0 +1,187 @@
param(
[ValidateNotNullOrEmpty()][string]$Vps = 'root@193.233.82.38',
[ValidateSet('Install', 'Rollback')][string]$Action = 'Install',
[switch]$PrepareOnly
)
$ErrorActionPreference = 'Stop'
$RepoRoot = Split-Path -Parent $PSScriptRoot
$UploadName = 'owrt-persistent-logs-vps-' + [guid]::NewGuid().ToString('N')
$StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName
$Utf8 = [System.Text.UTF8Encoding]::new($false)
$RemoteScript = @'
#!/bin/sh
set -eu
stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-persistent-logs-sync-v117
candidate=$target.new-__UPLOAD_NAME__
changed=0
show_failure() {
echo '--- owrt-remote status ---' >&2
systemctl --no-pager -l status owrt-remote >&2 || true
echo '--- owrt-remote journal ---' >&2
journalctl -u owrt-remote -n 80 --no-pager -l >&2 || true
}
cleanup() {
code=$?
trap - EXIT
if [ "$code" -ne 0 ]; then
echo "Remote step failed with rc=$code" >&2
show_failure
fi
if [ "$code" -ne 0 ] && [ "$changed" = 1 ] && [ -s "$stage/previous.py" ]; then
echo 'Restoring the previous Hub file...' >&2
cp -p "$stage/previous.py" "$candidate"
mv -f "$candidate" "$target"
systemctl restart owrt-remote || true
fi
rm -f "$candidate"
rm -rf "$stage"
exit "$code"
}
trap cleanup EXIT
test -s "$target"
python_bin=/opt/owrt-remote/venv/bin/python
[ -x "$python_bin" ] || python_bin=python3
if [ '__ACTION__' = Install ]; then
test -s "$stage/hub.py"
echo 'Validating candidate...'
"$python_bin" -m py_compile "$stage/hub.py"
"$python_bin" - "$stage/hub.py" <<'PY'
from pathlib import Path
import sys
source = Path(sys.argv[1]).read_text(encoding='utf-8')
required = (
'def receive_router_logs(',
'def router_log_storage(',
'def router_log_diagnosis(',
'def router_logs_html(',
'def router_logs_api(',
'data-global-logs-link=',
'ROUTER_LOG_MAX_BYTES = 2 * 1024 * 1024',
)
missing = [marker for marker in required if marker not in source]
if missing:
for marker in missing:
print('Missing marker:', repr(marker), file=sys.stderr)
raise SystemExit(1)
print('Candidate validation OK')
PY
# Read only the running Hub's relevant environment/argv; never print tokens.
hub_pid=$(systemctl show -p MainPID --value owrt-remote)
case "$hub_pid" in ''|*[!0-9]*|0) echo 'Cannot identify the running Hub storage.' >&2; exit 1 ;; esac
"$python_bin" - "$stage/hub.py" "$hub_pid" <<'PY'
import importlib.util
import json
import os
from pathlib import Path
import sys
sys.dont_write_bytecode = True
pid = sys.argv[2]
environment = Path('/proc/' + pid + '/environ').read_bytes().split(b'\0')
for item in environment:
key, _, value = item.partition(b'=')
if key in (b'OWRT_REMOTE_STATE_DIR', b'OWRT_REMOTE_DB'):
os.environ[key.decode()] = os.fsdecode(value)
spec = importlib.util.spec_from_file_location('storage_preflight', sys.argv[1])
hub = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = hub
spec.loader.exec_module(hub)
db_path = hub.DB_PATH
args = Path('/proc/' + pid + '/cmdline').read_bytes().split(b'\0')
for index, arg in enumerate(args):
if arg == b'--db' and index + 1 < len(args):
db_path = Path(os.fsdecode(args[index + 1]))
elif arg.startswith(b'--db='):
db_path = Path(os.fsdecode(arg[5:]))
if not db_path.is_absolute():
db_path = Path('/proc/' + pid + '/cwd').resolve() / db_path
storage = hub.router_log_storage(db_path)
print('VPS log archive storage:', json.dumps(storage, ensure_ascii=False))
if storage['persistent'] is not True or not storage.get('writable'):
raise SystemExit('Archive is not confirmed on writable persistent disk. Nothing installed; check the Hub database mount.')
if not db_path.is_file() or not os.access(db_path, os.W_OK):
raise SystemExit('Hub database is missing or not writable. Nothing installed.')
PY
if [ ! -e "$backup" ]; then
cp -p "$target" "$backup"
fi
source_file="$stage/hub.py"
else
test -s "$backup"
"$python_bin" -m py_compile "$backup"
source_file="$backup"
fi
cp -p "$target" "$stage/previous.py"
cp "$source_file" "$candidate"
chmod 0755 "$candidate"
changed=1
mv -f "$candidate" "$target"
echo 'Restarting owrt-remote...'
if ! systemctl restart owrt-remote; then
show_failure
exit 1
fi
healthy=0
for attempt in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
if systemctl is-active --quiet owrt-remote && curl -fsS --max-time 5 http://127.0.0.1:8088/health; then
healthy=1
break
fi
sleep 2
done
if [ "$healthy" -ne 1 ]; then
echo 'Health check did not become ready.' >&2
show_failure
exit 1
fi
cmp "$source_file" "$target"
printf '\nBackup: %s\nPERSISTENT_LOGS_VPS_V117_%s_OK\n' "$backup" '__ACTION__'
'@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
try {
[System.IO.Directory]::CreateDirectory($StageRoot) | Out-Null
[System.IO.File]::WriteAllText(
(Join-Path $StageRoot 'run.sh'),
$RemoteScript.Replace("`r`n", "`n") + "`n",
$Utf8
)
if ($Action -eq 'Install') {
$Content = [System.IO.File]::ReadAllText((Join-Path $RepoRoot 'vps\owrt-remote-hub.py')).Replace("`r`n", "`n")
[System.IO.File]::WriteAllText((Join-Path $StageRoot 'hub.py'), $Content, $Utf8)
}
if ($PrepareOnly) {
Write-Host "Prepared only: $StageRoot"
return
}
& scp -O -r -- $StageRoot "${Vps}:/tmp/"
if ($LASTEXITCODE -ne 0) { throw 'Upload failed. VPS was not changed.' }
& ssh -T -o ConnectTimeout=10 $Vps "sh /tmp/$UploadName/run.sh"
if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." }
Write-Host "Persistent logs VPS $Action finished successfully. Refresh Hub with Ctrl+F5."
}
finally {
if (-not $PrepareOnly) {
$Resolved = [System.IO.Path]::GetFullPath($StageRoot)
$Expected = [System.IO.Path]::GetFullPath((Join-Path ([System.IO.Path]::GetTempPath()) $UploadName))
if ($Resolved -ne $Expected -or (Split-Path -Leaf $Resolved) -ne $UploadName) {
throw 'Unsafe staging cleanup path.'
}
if (Test-Path -LiteralPath $Resolved) {
Remove-Item -LiteralPath $Resolved -Recurse -Force
}
}
}
+195
View File
@@ -0,0 +1,195 @@
param(
[ValidateNotNullOrEmpty()][string]$Router = 'root@192.168.2.1',
[ValidateSet('Install', 'Rollback')][string]$Action = 'Install',
[switch]$PrepareOnly
)
$ErrorActionPreference = 'Stop'
$RepoRoot = Split-Path -Parent $PSScriptRoot
$UploadName = 'owrt-persistent-logs-' + [guid]::NewGuid().ToString('N')
$StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName
$Utf8 = [System.Text.UTF8Encoding]::new($false)
$RemoteScript = @'
#!/bin/sh
set -eu
stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-persistent-logs-v117
items='cgi agent logs.sh logs-init luci'
changed=0
phase=preflight
item_path() {
case "$1" in
cgi) echo /www/cgi-bin/owrt-remote ;;
agent) echo /usr/sbin/owrt-remote ;;
init) echo /etc/init.d/owrt-remote ;;
logs.sh) echo /usr/lib/owrt-remote/logs.sh ;;
logs-init) echo /etc/init.d/owrt-remote-logs ;;
luci) echo /www/luci-static/resources/view/owrt_remote.js ;;
esac
}
refresh_router_ui() {
# Updated CGI and LuCI assets need no rpcd restart; keep management sessions alive.
if [ -x /etc/init.d/uhttpd ]; then /etc/init.d/uhttpd reload || true; fi
}
save_files() {
local dir item path
dir=$1
mkdir -p "$dir"
chmod 700 "$dir"
for item in $items; do
path=$(item_path "$item")
if [ -e "$path" ]; then cp -p "$path" "$dir/$item"; else : >"$dir/$item.absent"; fi
done
if [ -x /etc/init.d/owrt-remote-logs ] && /etc/init.d/owrt-remote-logs enabled; then : >"$dir/logs-enabled"; fi
if ubus call service list '{"name":"owrt-remote-logs"}' 2>/dev/null | grep -q '"running": true'; then : >"$dir/logs-running"; fi
: >"$dir/complete"
}
validate_backup() {
local dir item
dir=$1
[ -f "$dir/complete" ] || return 1
for item in $items; do
[ -s "$dir/$item" ] || [ -f "$dir/$item.absent" ] || return 1
done
}
restore_files() {
local dir item path
dir=$1
validate_backup "$dir" || { echo "Incomplete backup: $dir" >&2; return 1; }
if [ -x /etc/init.d/owrt-remote-logs ]; then
/etc/init.d/owrt-remote-logs stop || true
/etc/init.d/owrt-remote-logs disable || true
fi
for item in $items; do
path=$(item_path "$item")
if [ -f "$dir/$item.absent" ]; then
rm -f "$path" || return 1
else
cp -p "$dir/$item" "$path.new-__UPLOAD_NAME__" && mv -f "$path.new-__UPLOAD_NAME__" "$path" || return 1
fi
done
if [ -x /etc/init.d/owrt-remote-logs ]; then
[ ! -f "$dir/logs-enabled" ] || /etc/init.d/owrt-remote-logs enable
[ ! -f "$dir/logs-running" ] || /etc/init.d/owrt-remote-logs start
else
: # Preserve both journals for manual recovery.
fi
refresh_router_ui
}
cleanup() {
code=$?
trap - EXIT
if [ "$code" -ne 0 ]; then
printf 'Failed during %s (rc=%s).\n' "$phase" "$code" >&2
if [ "$changed" = 1 ]; then
echo 'Restoring files from this attempt.' >&2
restore_files "$stage/previous" || echo 'Restore failed; retained backup is available.' >&2
fi
fi
for item in $items; do path=$(item_path "$item"); rm -f "$path.new-__UPLOAD_NAME__"; done
rm -rf "$stage"
exit "$code"
}
trap cleanup EXIT
for item in cgi agent logs.sh logs-init; do path=$(item_path "$item"); test -s "$path"; done
if [ '__ACTION__' = Install ]; then
for item in $items; do test -s "$stage/$item"; done
for item in cgi agent logs.sh logs-init; do sh -n "$stage/$item"; done
command -v logread >/dev/null
command -v ubus >/dev/null
command -v mkfifo >/dev/null
command -v base64 >/dev/null
command -v jsonfilter >/dev/null
if ! command -v sha256sum >/dev/null 2>&1 && ! command -v md5sum >/dev/null 2>&1 && ! command -v cksum >/dev/null 2>&1; then
echo 'No supported checksum utility (sha256sum, md5sum or cksum). Nothing installed.' >&2
exit 1
fi
grep -F -q 'globalLogsToggle' "$stage/cgi"
for item in cgi luci; do grep -F -q 'v117' "$stage/$item"; done
grep -F -q 'LOG_BUFFER_LIMIT=16384' "$stage/logs.sh"
grep -F -q 'LOG_KEEP_SECONDS=86400' "$stage/logs.sh"
grep -F -q 'log_sync_loop' "$stage/logs.sh"
grep -F -q 'log_repair_legacy_locked' "$stage/logs.sh"
grep -F -q 'sync_global_logs' "$stage/agent"
phase=backup
if [ ! -e "$backup" ]; then save_files "$stage/retained"; mv "$stage/retained" "$backup"; fi
validate_backup "$backup" || { echo "Incomplete backup: $backup. Nothing changed." >&2; exit 1; }
save_files "$stage/previous"
phase=install
changed=1
if [ -x /etc/init.d/owrt-remote-logs ]; then /etc/init.d/owrt-remote-logs stop; fi
mkdir -p /usr/lib/owrt-remote /www/luci-static/resources/view
for item in $items; do
path=$(item_path "$item")
cp "$stage/$item" "$path.new-__UPLOAD_NAME__"
case "$item" in logs.sh|luci) chmod 644 "$path.new-__UPLOAD_NAME__" ;; *) chmod 755 "$path.new-__UPLOAD_NAME__" ;; esac
mv -f "$path.new-__UPLOAD_NAME__" "$path"
done
phase=legacy-migration
sh /usr/lib/owrt-remote/logs.sh migrate
/etc/init.d/owrt-remote-logs enable
/etc/init.d/owrt-remote-logs start
phase=verify
ready=0
for attempt in 1 2 3 4 5; do
if ubus call service list '{"name":"owrt-remote-logs"}' | grep -q '"running": true'; then ready=1; break; fi
sleep 1
done
if [ "$ready" != 1 ]; then echo 'Log collector did not start.' >&2; logread -e owrt-remote | tail -n 30 >&2; exit 1; fi
# Generate UTF-8 on the router; Windows PowerShell 5.1 reads scripts as ANSI.
marker="$(printf '\320\223\320\273\320\276\320\261\320\260\320\273\040\320\273\320\276\320\263\320\270\072') collector-__UPLOAD_NAME__"
ready=0
for attempt in 1 2 3 4 5; do
logger -t owrt-remote "$marker"
sleep 1
if sh /usr/lib/owrt-remote/logs.sh read | grep -F -q "$marker"; then ready=1; break; fi
done
if [ "$ready" != 1 ]; then echo 'Log collector is running but did not capture the test event.' >&2; exit 1; fi
for item in $items; do path=$(item_path "$item"); cmp "$stage/$item" "$path"; done
phase=storage-check
sh /usr/lib/owrt-remote/logs.sh flush force
sh /usr/lib/owrt-remote/logs.sh status
refresh_router_ui
echo 'Log collector: running. Router settings preserved.'
else
validate_backup "$backup" || { echo "No complete backup: $backup. Nothing changed." >&2; exit 1; }
save_files "$stage/previous"
changed=1
phase=rollback
restore_files "$backup"
fi
printf 'Backup: %s\n' "$backup"
printf '\nPERSISTENT_LOGS_ROUTER_%s_OK\n' '__ACTION__'
'@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
$Files = @{
'cgi' = 'files\www\cgi-bin\owrt-remote'
'agent' = 'files\usr\sbin\owrt-remote'
'logs.sh' = 'files\usr\lib\owrt-remote\logs.sh'
'logs-init' = 'files\etc\init.d\owrt-remote-logs'
'luci' = 'files\www\luci-static\resources\view\owrt_remote.js'
}
try {
[System.IO.Directory]::CreateDirectory($StageRoot) | Out-Null
[System.IO.File]::WriteAllText((Join-Path $StageRoot 'run.sh'), $RemoteScript.Replace("`r`n", "`n") + "`n", $Utf8)
if ($Action -eq 'Install') {
foreach ($Item in $Files.GetEnumerator()) {
$Content = [System.IO.File]::ReadAllText((Join-Path $RepoRoot $Item.Value)).Replace("`r`n", "`n")
[System.IO.File]::WriteAllText((Join-Path $StageRoot $Item.Key), $Content, $Utf8)
}
}
if ($PrepareOnly) { Write-Host "Prepared only: $StageRoot"; return }
& scp -O -r -- $StageRoot "${Router}:/tmp/"
if ($LASTEXITCODE -ne 0) { throw 'Upload failed. The installation has not started.' }
& ssh -T -o ConnectTimeout=10 $Router "sh /tmp/$UploadName/run.sh"
if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." }
Write-Host "Persistent logs $Action finished successfully. Refresh the router panel with Ctrl+F5."
} finally {
if (-not $PrepareOnly) {
$Resolved = [System.IO.Path]::GetFullPath($StageRoot)
$Expected = [System.IO.Path]::GetFullPath((Join-Path ([System.IO.Path]::GetTempPath()) $UploadName))
if ($Resolved -ne $Expected -or (Split-Path -Leaf $Resolved) -ne $UploadName) { throw 'Unsafe staging cleanup path.' }
if (Test-Path -LiteralPath $Resolved) { Remove-Item -LiteralPath $Resolved -Recurse -Force }
}
}
+6 -6
View File
@@ -17,8 +17,8 @@ if ($Action -eq 'Install') {
set -Eeuo pipefail set -Eeuo pipefail
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v117
captcha_backup=/opt/owrt-remote/captcha-before-recaptcha-v3-v116.json captcha_backup=/opt/owrt-remote/captcha-before-recaptcha-v3-v117.json
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
@@ -45,7 +45,7 @@ if [ ! -x "$python_bin" ]; then python_bin=python3; fi
import ast, pathlib, sys import ast, pathlib, sys
source = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8') source = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8')
tree = ast.parse(source) tree = ast.parse(source)
assert 'v116' in source assert 'v117' in source
assert 'v108' not in source assert 'v108' not in source
assert 'CAPTCHA_MODE_RECAPTCHA_V3' in source assert 'CAPTCHA_MODE_RECAPTCHA_V3' in source
assert 'prepareLoginRecaptcha' in source assert 'prepareLoginRecaptcha' in source
@@ -88,7 +88,7 @@ pid=$(systemctl show owrt-remote -p MainPID --value)
test "$pid" -gt 0 test "$pid" -gt 0
printf 'PID=%s FD=%s\n' "$pid" "$(find /proc/$pid/fd -maxdepth 1 -type l | wc -l)" printf 'PID=%s FD=%s\n' "$pid" "$(find /proc/$pid/fd -maxdepth 1 -type l | wc -l)"
systemctl --no-pager --full status owrt-remote | sed -n '1,16p' systemctl --no-pager --full status owrt-remote | sed -n '1,16p'
echo RECAPTCHA_V3_V116_OK echo RECAPTCHA_V3_V117_OK
'@ '@
} elseif ($Action -eq 'Rollback') { } elseif ($Action -eq 'Rollback') {
$RemoteScript = @' $RemoteScript = @'
@@ -96,8 +96,8 @@ echo RECAPTCHA_V3_V116_OK
set -Eeuo pipefail set -Eeuo pipefail
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-recaptcha-v3-v117
captcha_backup=/opt/owrt-remote/captcha-before-recaptcha-v3-v116.json captcha_backup=/opt/owrt-remote/captcha-before-recaptcha-v3-v117.json
candidate=$target.rollback-__UPLOAD_NAME__ candidate=$target.rollback-__UPLOAD_NAME__
changed=0 changed=0
restore_captcha() { restore_captcha() {
+4 -4
View File
@@ -18,7 +18,7 @@ if ($Action -eq 'Install') {
set -Eeuo pipefail set -Eeuo pipefail
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
@@ -45,7 +45,7 @@ if [ ! -x "$python_bin" ]; then python_bin=python3; fi
import ast, pathlib, sys import ast, pathlib, sys
source = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8') source = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8')
tree = ast.parse(source) tree = ast.parse(source)
assert 'v116' in source assert 'v117' in source
assert 'v108' not in source assert 'v108' not in source
factory = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == 'connect') factory = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == 'connect')
assert any(isinstance(item, ast.Name) and item.id == 'contextmanager' for item in factory.decorator_list) assert any(isinstance(item, ast.Name) and item.id == 'contextmanager' for item in factory.decorator_list)
@@ -78,7 +78,7 @@ pid=$(systemctl show owrt-remote -p MainPID --value)
test "$pid" -gt 0 test "$pid" -gt 0
printf 'PID=%s FD=%s\n' "$pid" "$(find /proc/$pid/fd -maxdepth 1 -type l | wc -l)" printf 'PID=%s FD=%s\n' "$pid" "$(find /proc/$pid/fd -maxdepth 1 -type l | wc -l)"
systemctl --no-pager --full status owrt-remote | sed -n '1,16p' systemctl --no-pager --full status owrt-remote | sed -n '1,16p'
echo RESOURCE_FIX_V116_OK echo RESOURCE_FIX_V117_OK
'@ '@
} elseif ($Action -eq 'Rollback') { } elseif ($Action -eq 'Rollback') {
$RemoteScript = @' $RemoteScript = @'
@@ -86,7 +86,7 @@ echo RESOURCE_FIX_V116_OK
set -Eeuo pipefail set -Eeuo pipefail
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v117
candidate=$target.rollback-__UPLOAD_NAME__ candidate=$target.rollback-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
+2 -2
View File
@@ -15,7 +15,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-router-card-ui-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-router-card-ui-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
@@ -106,7 +106,7 @@ if [ "$healthy" -ne 1 ]; then
exit 1 exit 1
fi fi
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nROUTER_CARD_UI_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nROUTER_CARD_UI_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
@@ -6,7 +6,7 @@ param(
$ErrorActionPreference = 'Stop' $ErrorActionPreference = 'Stop'
$RepoRoot = Split-Path -Parent $PSScriptRoot $RepoRoot = Split-Path -Parent $PSScriptRoot
$UploadName = 'owrt-version-v116-' + [guid]::NewGuid().ToString('N') $UploadName = 'owrt-version-v117-' + [guid]::NewGuid().ToString('N')
$StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName $StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName
$Utf8 = [System.Text.UTF8Encoding]::new($false) $Utf8 = [System.Text.UTF8Encoding]::new($false)
@@ -15,7 +15,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
@@ -57,8 +57,8 @@ if [ '__ACTION__' = Install ]; then
from pathlib import Path from pathlib import Path
import sys import sys
source = Path(sys.argv[1]).read_text(encoding='utf-8') source = Path(sys.argv[1]).read_text(encoding='utf-8')
required = ('v116',) required = ('v117',)
if 'v' + '115' in source: if 'v' + '116' in source:
print('Old UI version remains in candidate', file=sys.stderr) print('Old UI version remains in candidate', file=sys.stderr)
raise SystemExit(1) raise SystemExit(1)
missing = [marker for marker in required if marker not in source] missing = [marker for marker in required if marker not in source]
@@ -103,7 +103,7 @@ if [ "$healthy" -ne 1 ]; then
exit 1 exit 1
fi fi
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nHUB_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nHUB_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
@@ -130,7 +130,7 @@ try {
if ($LASTEXITCODE -ne 0) { throw 'Upload failed. VPS was not changed.' } if ($LASTEXITCODE -ne 0) { throw 'Upload failed. VPS was not changed.' }
& ssh -T -o ConnectTimeout=10 $Vps "sh /tmp/$UploadName/run.sh" & ssh -T -o ConnectTimeout=10 $Vps "sh /tmp/$UploadName/run.sh"
if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." } if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." }
Write-Host "Hub v116 $Action finished successfully. Refresh the Hub panel." Write-Host "Hub v117 $Action finished successfully. Refresh the Hub panel."
} }
finally { finally {
if (-not $PrepareOnly) { if (-not $PrepareOnly) {
+3 -3
View File
@@ -14,7 +14,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-vps-widgets-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
cleanup() { cleanup() {
@@ -37,7 +37,7 @@ python_bin=/opt/owrt-remote/venv/bin/python
[ -x "$python_bin" ] || python_bin=python3 [ -x "$python_bin" ] || python_bin=python3
if [ '__ACTION__' = Install ]; then if [ '__ACTION__' = Install ]; then
test -s "$stage/hub.py" test -s "$stage/hub.py"
grep -F -q 'v116' "$stage/hub.py" grep -F -q 'v117' "$stage/hub.py"
"$python_bin" -m py_compile "$stage/hub.py" "$python_bin" -m py_compile "$stage/hub.py"
OWRT_VPS_WIDGETS_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/tests.py" -v OWRT_VPS_WIDGETS_HUB_SOURCE="$stage/hub.py" "$python_bin" "$stage/tests.py" -v
if [ ! -e "$backup" ]; then cp -p "$target" "$backup"; fi if [ ! -e "$backup" ]; then cp -p "$target" "$backup"; fi
@@ -67,7 +67,7 @@ if [ '__ACTION__' = Install ]; then
code=$(curl -sS --max-time 5 -o "$stage/response.json" -w '%{http_code}' http://127.0.0.1:8088/api/vps/resources) code=$(curl -sS --max-time 5 -o "$stage/response.json" -w '%{http_code}' http://127.0.0.1:8088/api/vps/resources)
test "$code" = 401 test "$code" = 401
fi fi
printf '\nVPS_WIDGETS_V116_%s_OK\n' '__ACTION__' printf '\nVPS_WIDGETS_V117_%s_OK\n' '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
try { try {
+1 -1
View File
@@ -14,7 +14,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/usr/lib/owrt-remote/wan.sh target=/usr/lib/owrt-remote/wan.sh
backup_dir=/root/owrt-remote-backup-before-wan-dns-v116 backup_dir=/root/owrt-remote-backup-before-wan-dns-v117
changed=0 changed=0
phase=preflight phase=preflight
heartbeat_pid() { heartbeat_pid() {
+2 -2
View File
@@ -20,7 +20,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-wan-multi-v116 backup=/root/owrt-remote-backup-before-wan-multi-v117
config=/etc/config/owrtremote config=/etc/config/owrtremote
changed=0 changed=0
item_path() { item_path() {
@@ -81,7 +81,7 @@ test -s "$config"
if [ '__ACTION__' = Install ]; then if [ '__ACTION__' = Install ]; then
for item in agent wan.sh cgi hotplug; do test -s "$stage/$item"; sh -n "$stage/$item"; done for item in agent wan.sh cgi hotplug; do test -s "$stage/$item"; sh -n "$stage/$item"; done
test -s "$stage/luci" test -s "$stage/luci"
for item in cgi luci; do grep -F -q 'v116' "$stage/$item"; done for item in cgi luci; do grep -F -q 'v117' "$stage/$item"; done
grep -F -q 'wan_interface_picker' "$stage/cgi" grep -F -q 'wan_interface_picker' "$stage/cgi"
grep -F -q 'wan_select_interface()' "$stage/wan.sh" grep -F -q 'wan_select_interface()' "$stage/wan.sh"
grep -F -q 'wan_reconcile_xray()' "$stage/agent" grep -F -q 'wan_reconcile_xray()' "$stage/agent"
+154
View File
@@ -0,0 +1,154 @@
param(
[ValidateNotNullOrEmpty()][string]$Router = 'root@192.168.2.1',
[ValidateSet('Install', 'Rollback')][string]$Action = 'Install',
[switch]$PrepareOnly
)
$ErrorActionPreference = 'Stop'
$RepoRoot = Split-Path -Parent $PSScriptRoot
$UploadName = 'owrt-wan-recovery-' + [guid]::NewGuid().ToString('N')
$StageRoot = Join-Path ([System.IO.Path]::GetTempPath()) $UploadName
$Utf8 = [System.Text.UTF8Encoding]::new($false)
$Payloads = @{
'wan.sh' = 'files\usr\lib\owrt-remote\wan.sh'
'agent' = 'files\usr\sbin\owrt-remote'
}
$RemoteScript = @'
#!/bin/sh
set -eu
stage=/tmp/__UPLOAD_NAME__
backup_dir=/root/owrt-remote-backup-before-wan-recovery-v117
changed=0
phase=preflight
item_path() {
case "$1" in
wan.sh) printf /usr/lib/owrt-remote/wan.sh ;;
agent) printf /usr/sbin/owrt-remote ;;
esac
}
complete_backup() {
[ -f "$1/complete" ] && [ -s "$1/wan.sh" ] && [ -s "$1/agent" ]
}
restore_files() {
local dir item path
dir=$1
complete_backup "$dir" || return 1
for item in agent wan.sh; do
path=$(item_path "$item")
cp -p "$dir/$item" "$path.new-__UPLOAD_NAME__" || return 1
mv -f "$path.new-__UPLOAD_NAME__" "$path" || return 1
done
}
heartbeat_pid() {
local state
state=$(ubus call service list '{"name":"owrt-remote"}') || return 1
printf '%s' "$state" | jsonfilter -e '@["owrt-remote"].instances.heartbeat.pid' 2>/dev/null || true
}
refresh_heartbeat() {
local old_pid new_pid attempt
old_pid=$(heartbeat_pid) || return 1
case "$old_pid" in ''|0) echo 'Heartbeat loop is stopped; files will be used at its next start.'; return 0 ;; *[!0-9]*) return 1 ;; esac
# Signal only the supervised heartbeat shell; procd respawns it from the
# new agent. A healthy Xray instance stays running; a crashed one can recover.
ubus call service signal '{"name":"owrt-remote","instance":"heartbeat","signal":9}' >/dev/null || return 1
attempt=0
while [ "$attempt" -lt 20 ]; do
sleep 1
new_pid=$(heartbeat_pid) || return 1
case "$new_pid" in ''|0|*[!0-9]*) ;; *)
[ "$new_pid" = "$old_pid" ] || { echo 'Heartbeat loop reloaded; WAN recovery is active for enabled tunnels.'; return 0; } ;;
esac
attempt=$((attempt+1))
done
echo 'Heartbeat loop did not respawn. Code will be restored.' >&2
return 1
}
cleanup() {
code=$?
trap - EXIT
if [ "$code" -ne 0 ]; then
printf 'Failed during %s (rc=%s).\n' "$phase" "$code" >&2
if [ "$changed" = 1 ]; then
echo 'Restoring the agent and WAN helper from this attempt.' >&2
restore_files "$stage/previous" && refresh_heartbeat || echo 'Restore failed; retained backup is available.' >&2
fi
fi
for item in agent wan.sh; do path=$(item_path "$item"); rm -f "$path.new-__UPLOAD_NAME__"; done
rm -rf "$stage"
exit "$code"
}
trap cleanup EXIT
for item in agent wan.sh; do
path=$(item_path "$item")
test -s "$path"
done
heartbeat_pid >/dev/null
if [ '__ACTION__' = Install ]; then
for item in agent wan.sh; do test -s "$stage/$item"; sh -n "$stage/$item"; done
grep -F -q 'wan_recover_xray()' "$stage/agent"
grep -F -q 'wait_for_selected_wan()' "$stage/agent"
grep -F -q 'reason=https_required' "$stage/agent"
grep -F -q 'remote_curl_post' "$stage/wan.sh"
if [ ! -e "$backup_dir" ]; then
mkdir -p "$stage/retained-backup"
chmod 700 "$stage/retained-backup"
for item in agent wan.sh; do path=$(item_path "$item"); cp -p "$path" "$stage/retained-backup/$item"; done
touch "$stage/retained-backup/complete"
mv "$stage/retained-backup" "$backup_dir"
fi
complete_backup "$backup_dir" || { echo "Incomplete backup: $backup_dir. Nothing changed." >&2; exit 1; }
source_dir=$stage
else
complete_backup "$backup_dir" || { echo "No complete backup: $backup_dir. Nothing changed." >&2; exit 1; }
for item in agent wan.sh; do sh -n "$backup_dir/$item"; done
source_dir=$backup_dir
fi
printf 'Backup: %s\n' "$backup_dir"
mkdir -p "$stage/previous"
for item in agent wan.sh; do path=$(item_path "$item"); cp -p "$path" "$stage/previous/$item"; done
touch "$stage/previous/complete"
phase=install
changed=1
for item in agent wan.sh; do
path=$(item_path "$item")
cp "$source_dir/$item" "$path.new-__UPLOAD_NAME__"
chmod 755 "$path.new-__UPLOAD_NAME__"
mv -f "$path.new-__UPLOAD_NAME__" "$path"
done
phase=verify
for item in agent wan.sh; do
path=$(item_path "$item")
sh -n "$path"
cmp "$source_dir/$item" "$path"
done
phase=heartbeat-reload
refresh_heartbeat
# The reloaded heartbeat recovers an enabled, crashed Xray instance when WAN
# returns. A healthy tunnel is not restarted by this deployment.
printf '\nWAN_RECOVERY_ROUTER_%s_OK\n' '__ACTION__'
'@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
try {
[System.IO.Directory]::CreateDirectory($StageRoot) | Out-Null
[System.IO.File]::WriteAllText((Join-Path $StageRoot 'run.sh'), $RemoteScript.Replace("`r`n", "`n") + "`n", $Utf8)
if ($Action -eq 'Install') {
foreach ($Item in $Payloads.GetEnumerator()) {
$Content = [System.IO.File]::ReadAllText((Join-Path $RepoRoot $Item.Value)).Replace("`r`n", "`n")
[System.IO.File]::WriteAllText((Join-Path $StageRoot $Item.Key), $Content, $Utf8)
}
}
if ($PrepareOnly) { Write-Host "Prepared only: $StageRoot"; return }
& scp -O -r -- $StageRoot "${Router}:/tmp/"
if ($LASTEXITCODE -ne 0) { throw 'Upload failed. The installation has not started.' }
& ssh -T -o ConnectTimeout=10 $Router "sh /tmp/$UploadName/run.sh"
if ($LASTEXITCODE -ne 0) { throw "$Action failed. See the output above; success was not confirmed." }
Write-Host "WAN recovery fix $Action finished successfully."
} finally {
if (-not $PrepareOnly) {
$Resolved = [System.IO.Path]::GetFullPath($StageRoot)
$Expected = [System.IO.Path]::GetFullPath((Join-Path ([System.IO.Path]::GetTempPath()) $UploadName))
if ($Resolved -ne $Expected -or (Split-Path -Leaf $Resolved) -ne $UploadName) { throw 'Unsafe staging cleanup path.' }
if (Test-Path -LiteralPath $Resolved) { Remove-Item -LiteralPath $Resolved -Recurse -Force }
}
}
+1 -1
View File
@@ -17,7 +17,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup=/root/owrt-remote-backup-before-wan-stability-v116 backup=/root/owrt-remote-backup-before-wan-stability-v117
config=/etc/config/owrtremote config=/etc/config/owrtremote
changed=0 changed=0
fail() { fail() {
+2 -2
View File
@@ -15,7 +15,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/opt/owrt-remote/owrt-remote-hub.py target=/opt/owrt-remote/owrt-remote-hub.py
backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-web-push-v116 backup=/opt/owrt-remote/owrt-remote-hub.py.bak-before-web-push-v117
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
@@ -72,7 +72,7 @@ for attempt in 1 2 3 4 5 6 7 8 9 10; do
done done
test "$healthy" = 1 test "$healthy" = 1
cmp "$source_file" "$target" cmp "$source_file" "$target"
printf '\nBackup: %s\nWEB_PUSH_V116_%s_OK\n' "$backup" '__ACTION__' printf '\nBackup: %s\nWEB_PUSH_V117_%s_OK\n' "$backup" '__ACTION__'
'@ '@
$RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action) $RemoteScript = $RemoteScript.Replace('__UPLOAD_NAME__', $UploadName).Replace('__ACTION__', $Action)
+1 -1
View File
@@ -14,7 +14,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/www/cgi-bin/owrt-remote target=/www/cgi-bin/owrt-remote
backup_dir=/root/owrt-remote-backup-before-xray-apply-v116 backup_dir=/root/owrt-remote-backup-before-xray-apply-v117
backup=$backup_dir/owrt-remote backup=$backup_dir/owrt-remote
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
+1 -1
View File
@@ -14,7 +14,7 @@ $RemoteScript = @'
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
target=/www/cgi-bin/owrt-remote target=/www/cgi-bin/owrt-remote
backup_dir=/root/owrt-remote-backup-before-xray-mobile-v116 backup_dir=/root/owrt-remote-backup-before-xray-mobile-v117
backup=$backup_dir/owrt-remote backup=$backup_dir/owrt-remote
candidate=$target.new-__UPLOAD_NAME__ candidate=$target.new-__UPLOAD_NAME__
changed=0 changed=0
+1 -1
View File
@@ -17,7 +17,7 @@ $RemoteScript = @'
#!/bin/sh #!/bin/sh
set -eu set -eu
stage=/tmp/__UPLOAD_NAME__ stage=/tmp/__UPLOAD_NAME__
backup_dir=/root/owrt-remote-backup-before-xray-validation-v116 backup_dir=/root/owrt-remote-backup-before-xray-validation-v117
changed=0 changed=0
phase=preflight phase=preflight
item_path() { item_path() {
+55 -32
View File
@@ -3087,6 +3087,38 @@ ROUTER_LOG_MAX_BYTES = 2 * 1024 * 1024
ROUTER_LOG_BATCH_MAX_BYTES = 65536 ROUTER_LOG_BATCH_MAX_BYTES = 65536
def router_log_storage(db_path):
"""Describe the archive's real backing mount, including symlinked DB paths."""
path = Path(db_path).resolve()
result = {'path': str(path), 'filesystem': 'unknown', 'persistent': None,
'limit_bytes': ROUTER_LOG_MAX_BYTES}
if sys.platform != 'linux':
return result
def unescape(value):
return re.sub(r'\\([0-7]{3})', lambda m: chr(int(m[1], 8)), value)
try:
mounts = []
with open('/proc/self/mountinfo', encoding='utf-8') as source:
for line in source:
fields, separator, tail = line.partition(' - ')
left, right = fields.split(), tail.split()
if not separator or len(left) < 6 or len(right) < 3:
continue
mount = Path(unescape(left[4]))
if path.is_relative_to(mount):
mounts.append((len(mount.parts), right[0], left[5]))
if mounts:
_, filesystem, flags = max(mounts, key=lambda item: item[0])
persistent = (False if filesystem in {'tmpfs', 'ramfs', 'rootfs'} else
True if filesystem in {'ext2', 'ext3', 'ext4', 'xfs', 'btrfs', 'f2fs',
'jfs', 'zfs', 'ubifs', 'jffs2', 'nfs', 'nfs4', 'ceph'} else None)
result.update(filesystem=filesystem, persistent=persistent,
writable='rw' in flags.split(','))
except OSError:
pass
return result
def redact_router_log_bytes(raw): def redact_router_log_bytes(raw):
"""Defense in depth: do not archive credentials supplied by older agents.""" """Defense in depth: do not archive credentials supplied by older agents."""
secret = re.compile(rb'authorization|hub_token|vless_uuid|ssh_vless_uuid|password|passwd|web[._ -]?key|[?&](?:key|token)=|"(?:uuid|privatekey)"|[0-9a-f]{8}-[0-9a-f-]{27,}', re.I) secret = re.compile(rb'authorization|hub_token|vless_uuid|ssh_vless_uuid|password|passwd|web[._ -]?key|[?&](?:key|token)=|"(?:uuid|privatekey)"|[0-9a-f]{8}-[0-9a-f-]{27,}', re.I)
@@ -3137,6 +3169,12 @@ def receive_router_logs(conn, router_id, batch, state=None):
conn.execute('update router_logs set last_sync = ? where router_id = ?', (ts, router_id)) conn.execute('update router_logs set last_sync = ? where router_id = ?', (ts, router_id))
return {'journal_id': journal_id, 'offset': end, 'clear_request': pending} return {'journal_id': journal_id, 'offset': end, 'clear_request': pending}
cursor = int(row['cursor']) if row['journal_id'] == journal_id else 0 cursor = int(row['cursor']) if row['journal_id'] == journal_id else 0
# A status-only packet cannot prove delivery of any journal bytes. Older
# helpers sent from=to=total even when every chunk was skipped. Acknowledging
# that number permanently hid the retained records and invented rotation gaps.
if not raw and not pending:
conn.execute('update router_logs set last_sync = ? where router_id = ?', (ts, router_id))
return {'journal_id': journal_id, 'offset': cursor, 'clear_request': ''}
if not pending and row['journal_id'] == journal_id and end <= cursor: if not pending and row['journal_id'] == journal_id and end <= cursor:
conn.execute('update router_logs set last_sync = ? where router_id = ?', (ts, router_id)) conn.execute('update router_logs set last_sync = ? where router_id = ?', (ts, router_id))
return {'journal_id': journal_id, 'offset': cursor, 'clear_request': ''} return {'journal_id': journal_id, 'offset': cursor, 'clear_request': ''}
@@ -3145,7 +3183,7 @@ def receive_router_logs(conn, router_id, batch, state=None):
cursor = start cursor = start
stored = b'' stored = b''
elif row['journal_id'] and row['journal_id'] != journal_id: elif row['journal_id'] and row['journal_id'] != journal_id:
stored += ('[OK] ' + iso_time(ts) + ' Роутер начал новый журнал: перезагрузка или очистка в ОЗУ.\n').encode('utf-8') stored += ('[OK] ' + iso_time(ts) + ' Роутер начал новый журнал: очистка или восстановление сохранённой истории.\n').encode('utf-8')
if start > cursor: if start > cursor:
stored += ('[WARN] ' + iso_time(ts) + ' Часть записей не доставлена до ротации журнала на роутере.\n').encode('utf-8') stored += ('[WARN] ' + iso_time(ts) + ' Часть записей не доставлена до ротации журнала на роутере.\n').encode('utf-8')
if end > cursor: if end > cursor:
@@ -4968,11 +5006,16 @@ body::before{{content:"";position:fixed;inset:-25%;z-index:0;pointer-events:none
.brand{{display:flex;align-items:center;gap:14px;width:100%;min-width:0}} .brand{{display:flex;align-items:center;gap:14px;width:100%;min-width:0}}
h1{{margin:0;font-size:29px;line-height:1.2;letter-spacing:0}}.appBanner{{position:relative;display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;padding:8px 14px;border:1px solid rgba(34,211,238,.38);border-radius:999px;background:linear-gradient(110deg,rgba(34,211,238,.14),rgba(124,58,237,.24),rgba(236,72,153,.14));color:#f3e8ff;text-decoration:none;font-weight:800;font-size:13px;line-height:1;white-space:nowrap;box-shadow:0 10px 24px rgba(124,58,237,.16),inset 0 1px 0 rgba(255,255,255,.10);overflow:hidden}}.appBanner::before{{content:"";position:absolute;inset:0;background:linear-gradient(90deg,transparent,rgba(255,255,255,.20),transparent);transform:translateX(-120%);animation:bannerShine 6.2s ease-in-out infinite}}.appBanner span{{position:relative}}.appBannerVersion{{color:#fb7185;text-shadow:0 0 12px rgba(251,113,133,.35)}}.muted{{color:var(--muted)}}.top p{{margin:4px 0 0}}.links,.headerActions{{display:flex;align-items:center;gap:8px}}.links{{margin-top:0;flex-wrap:nowrap}}.links a,.badge{{position:relative;display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;padding:8px 14px;border:1px solid var(--line);border-radius:999px;background:rgba(255,255,255,.08);color:#f3e8ff;text-decoration:none;font-weight:800;font-size:13px;line-height:1;white-space:nowrap;overflow:hidden}}.headerActions{{position:relative;display:grid;grid-template-columns:repeat(5,minmax(0,1fr));align-self:flex-start;justify-content:flex-start;align-content:flex-start;flex:1 1 auto;min-width:0;gap:8px;padding-top:0;max-width:none}}.headerActions .badge,.headerActions .btn{{width:100%;min-height:36px;min-width:0;padding:8px 10px;border-radius:999px;font-weight:800;font-size:12px;line-height:1;white-space:nowrap}}.headerActions .btn[href="/logout"]{{margin-left:0}}.badge{{background:rgba(255,255,255,.08);color:#f3e8ff;box-shadow:inset 0 1px 0 rgba(255,255,255,.06)}}.nethavenTop{{border-color:rgba(34,211,238,.46);background:linear-gradient(110deg,rgba(14,165,233,.20),rgba(168,85,247,.22),rgba(34,197,94,.14));color:#ecfeff;box-shadow:0 10px 24px rgba(14,165,233,.14),inset 0 1px 0 rgba(255,255,255,.10)}}.nethavenTop::before{{content:"";position:absolute;inset:0;background:linear-gradient(90deg,transparent,rgba(255,255,255,.24),transparent);transform:translateX(-120%);animation:bannerShine 6.2s ease-in-out infinite;pointer-events:none}}.authToggle{{cursor:pointer}}.dot{{width:9px;height:9px;border-radius:999px;background:var(--red);box-shadow:0 0 13px rgba(251,113,133,.72)}}.dot.on{{background:var(--green);box-shadow:0 0 13px rgba(34,197,94,.75)}}.dot.warn{{background:var(--amber);box-shadow:0 0 13px rgba(245,158,11,.75)}} h1{{margin:0;font-size:29px;line-height:1.2;letter-spacing:0}}.appBanner{{position:relative;display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;padding:8px 14px;border:1px solid rgba(34,211,238,.38);border-radius:999px;background:linear-gradient(110deg,rgba(34,211,238,.14),rgba(124,58,237,.24),rgba(236,72,153,.14));color:#f3e8ff;text-decoration:none;font-weight:800;font-size:13px;line-height:1;white-space:nowrap;box-shadow:0 10px 24px rgba(124,58,237,.16),inset 0 1px 0 rgba(255,255,255,.10);overflow:hidden}}.appBanner::before{{content:"";position:absolute;inset:0;background:linear-gradient(90deg,transparent,rgba(255,255,255,.20),transparent);transform:translateX(-120%);animation:bannerShine 6.2s ease-in-out infinite}}.appBanner span{{position:relative}}.appBannerVersion{{color:#fb7185;text-shadow:0 0 12px rgba(251,113,133,.35)}}.muted{{color:var(--muted)}}.top p{{margin:4px 0 0}}.links,.headerActions{{display:flex;align-items:center;gap:8px}}.links{{margin-top:0;flex-wrap:nowrap}}.links a,.badge{{position:relative;display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;padding:8px 14px;border:1px solid var(--line);border-radius:999px;background:rgba(255,255,255,.08);color:#f3e8ff;text-decoration:none;font-weight:800;font-size:13px;line-height:1;white-space:nowrap;overflow:hidden}}.headerActions{{position:relative;display:grid;grid-template-columns:repeat(5,minmax(0,1fr));align-self:flex-start;justify-content:flex-start;align-content:flex-start;flex:1 1 auto;min-width:0;gap:8px;padding-top:0;max-width:none}}.headerActions .badge,.headerActions .btn{{width:100%;min-height:36px;min-width:0;padding:8px 10px;border-radius:999px;font-weight:800;font-size:12px;line-height:1;white-space:nowrap}}.headerActions .btn[href="/logout"]{{margin-left:0}}.badge{{background:rgba(255,255,255,.08);color:#f3e8ff;box-shadow:inset 0 1px 0 rgba(255,255,255,.06)}}.nethavenTop{{border-color:rgba(34,211,238,.46);background:linear-gradient(110deg,rgba(14,165,233,.20),rgba(168,85,247,.22),rgba(34,197,94,.14));color:#ecfeff;box-shadow:0 10px 24px rgba(14,165,233,.14),inset 0 1px 0 rgba(255,255,255,.10)}}.nethavenTop::before{{content:"";position:absolute;inset:0;background:linear-gradient(90deg,transparent,rgba(255,255,255,.24),transparent);transform:translateX(-120%);animation:bannerShine 6.2s ease-in-out infinite;pointer-events:none}}.authToggle{{cursor:pointer}}.dot{{width:9px;height:9px;border-radius:999px;background:var(--red);box-shadow:0 0 13px rgba(251,113,133,.72)}}.dot.on{{background:var(--green);box-shadow:0 0 13px rgba(34,197,94,.75)}}.dot.warn{{background:var(--amber);box-shadow:0 0 13px rgba(245,158,11,.75)}}
.toolbar{{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1.2fr) minmax(84px,.52fr) minmax(100px,.65fr) minmax(0,1fr) minmax(0,1fr) minmax(116px,.68fr);gap:8px;margin:18px 0;padding:14px 16px;background:linear-gradient(180deg,rgba(255,255,255,.08),rgba(255,255,255,.045)),var(--panel);border:1px solid var(--line);border-radius:8px;box-shadow:0 18px 46px rgba(0,0,0,.20);backdrop-filter:blur(10px);width:100%;max-width:100%;box-sizing:border-box}}.toolbar>*{{width:100%;min-width:0}}.toolbar input,.toolbar select{{background:rgba(255,255,255,.08);border-color:var(--line);color:#f3e8ff;box-shadow:inset 0 1px 0 rgba(255,255,255,.05)}}.toolbar input::placeholder{{color:#b9adc9}} .toolbar{{display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1.2fr) minmax(84px,.52fr) minmax(100px,.65fr) minmax(0,1fr) minmax(0,1fr) minmax(116px,.68fr);gap:8px;margin:18px 0;padding:14px 16px;background:linear-gradient(180deg,rgba(255,255,255,.08),rgba(255,255,255,.045)),var(--panel);border:1px solid var(--line);border-radius:8px;box-shadow:0 18px 46px rgba(0,0,0,.20);backdrop-filter:blur(10px);width:100%;max-width:100%;box-sizing:border-box}}.toolbar>*{{width:100%;min-width:0}}.toolbar input,.toolbar select{{background:rgba(255,255,255,.08);border-color:var(--line);color:#f3e8ff;box-shadow:inset 0 1px 0 rgba(255,255,255,.05)}}.toolbar input::placeholder{{color:#b9adc9}}
@media(min-width:681px){{
#routerForm select,#routerGroupFilter{{color-scheme:dark}}
#routerForm select option,#routerGroupFilter option{{background-color:#221a34;color:#f7f2ff}}
#routerForm select option:checked,#routerGroupFilter option:checked{{background-color:#7c3aed;color:#fff}}
}}
.authMenu{{position:absolute;right:0;top:calc(100% + 10px);z-index:60;width:min(640px,calc(100vw - 44px));max-height:min(820px,calc(100svh - 120px));overflow:auto;padding:16px;background:linear-gradient(180deg,rgba(255,255,255,.09),rgba(255,255,255,.05)),rgba(19,14,32,.96);border:1px solid var(--line);border-radius:8px;box-shadow:0 24px 70px rgba(0,0,0,.36);backdrop-filter:blur(12px);scrollbar-width:thin}}.authMenu[hidden]{{display:none}}.authMenu>h2,.authMenu>p{{display:none}}.authMenuHead{{position:relative;display:block}}.authMenuHead>div{{min-width:0}}.authMenuHead p{{display:none}}.authMenu h2{{margin:0 0 4px;font-size:18px}}.authMenuClose{{display:none;position:absolute;top:14px;right:14px;min-height:34px;padding:7px 12px;border-radius:999px;white-space:nowrap}}.authMenu p{{margin:0 0 12px;color:var(--muted)}}.authGrid{{display:grid;grid-template-columns:1fr;gap:10px}}.authGrid .wide{{grid-column:1/-1}}.msg{{margin-top:10px;color:#bbf7d0;font-weight:750}}.msg.bad{{color:#fecdd3}}.formMsg{{margin:-8px 0 18px;padding:10px 12px;border:1px solid rgba(34,197,94,.34);border-radius:8px;background:rgba(34,197,94,.12);color:#bbf7d0;font-weight:800}}.formMsg.bad{{border-color:rgba(251,113,133,.4);background:rgba(251,113,133,.13);color:#fecdd3}} .authMenu{{position:absolute;right:0;top:calc(100% + 10px);z-index:60;width:min(640px,calc(100vw - 44px));max-height:min(820px,calc(100svh - 120px));overflow:auto;padding:16px;background:linear-gradient(180deg,rgba(255,255,255,.09),rgba(255,255,255,.05)),rgba(19,14,32,.96);border:1px solid var(--line);border-radius:8px;box-shadow:0 24px 70px rgba(0,0,0,.36);backdrop-filter:blur(12px);scrollbar-width:thin}}.authMenu[hidden]{{display:none}}.authMenu>h2,.authMenu>p{{display:none}}.authMenuHead{{position:relative;display:block}}.authMenuHead>div{{min-width:0}}.authMenuHead p{{display:none}}.authMenu h2{{margin:0 0 4px;font-size:18px}}.authMenuClose{{display:none;position:absolute;top:14px;right:14px;min-height:34px;padding:7px 12px;border-radius:999px;white-space:nowrap}}.authMenu p{{margin:0 0 12px;color:var(--muted)}}.authGrid{{display:grid;grid-template-columns:1fr;gap:10px}}.authGrid .wide{{grid-column:1/-1}}.msg{{margin-top:10px;color:#bbf7d0;font-weight:750}}.msg.bad{{color:#fecdd3}}.formMsg{{margin:-8px 0 18px;padding:10px 12px;border:1px solid rgba(34,197,94,.34);border-radius:8px;background:rgba(34,197,94,.12);color:#bbf7d0;font-weight:800}}.formMsg.bad{{border-color:rgba(251,113,133,.4);background:rgba(251,113,133,.13);color:#fecdd3}}
.authGroup{{margin-top:14px;border:1px solid rgba(34,211,238,.22);border-radius:14px;background:linear-gradient(180deg,rgba(255,255,255,.06),rgba(255,255,255,.03)),rgba(15,10,26,.78);box-shadow:inset 0 1px 0 rgba(255,255,255,.04)}}.authGroupSummary{{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:12px 14px;cursor:pointer;list-style:none}}.authGroupSummary::-webkit-details-marker{{display:none}}.authGroupTitle strong{{display:block;color:#f7f2ff;font-size:15px;font-weight:900;line-height:1.15}}.authGroupTitle span{{display:block;margin-top:4px;color:var(--muted);font-size:12px;line-height:1.35}}.authGroupChevron{{display:inline-flex;align-items:center;justify-content:center;flex:0 0 28px;width:28px;height:28px;border:1px solid rgba(34,211,238,.24);border-radius:999px;background:rgba(34,211,238,.08);color:#dbeafe;font-size:16px;line-height:1;transition:transform .18s ease,background .18s ease,border-color .18s ease}}.authGroup[open] .authGroupChevron{{transform:rotate(180deg);background:rgba(34,197,94,.12);border-color:rgba(34,197,94,.24)}}.authGroupBody{{padding:0 14px 14px;border-top:1px solid rgba(255,255,255,.08)}}.authGroupBody>.authPasswordSection{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.authOverview{{margin-top:0;padding-top:14px;border-top:0}}.authGroupBody>.sessionBox{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.notifyBox{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.backupBox{{margin-top:14px;padding-top:0;border-top:0}}.authOverview{{display:grid;gap:10px;margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.authPills{{display:flex;flex-wrap:wrap;gap:8px}}.authPill{{display:inline-flex;align-items:center;gap:7px;padding:7px 10px;border:1px solid rgba(34,211,238,.24);border-radius:999px;background:rgba(34,211,238,.08);color:#dbeafe;font-size:12px;font-weight:850}}.authPill.off{{border-color:rgba(255,255,255,.10);background:rgba(255,255,255,.06);color:#c4b5fd}}.authSection{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.authSectionHead{{display:flex;align-items:flex-start;justify-content:space-between;gap:10px;margin-bottom:8px}}.authSectionHead h3{{margin:0;font-size:15px}}.authSectionHead p{{margin:4px 0 0;color:var(--muted);font-size:12px;line-height:1.35}}.authSectionState{{display:inline-flex;align-items:center;justify-content:center;min-height:30px;padding:6px 10px;border:1px solid rgba(34,197,94,.30);border-radius:999px;background:rgba(34,197,94,.12);color:#bbf7d0;font-size:12px;font-weight:850;white-space:nowrap}}.authSectionState.off{{border-color:rgba(255,255,255,.10);background:rgba(255,255,255,.06);color:#c4b5fd}}.authSectionState.warn{{border-color:rgba(251,191,36,.28);background:rgba(251,191,36,.11);color:#fde68a}}.authFields{{display:grid;grid-template-columns:1fr 1fr;gap:8px}}.authFields .wide{{grid-column:1/-1}}.authFields input,.authFields textarea{{width:100%;min-width:0}}.authFields textarea{{min-height:92px;resize:vertical}}.authActions{{display:flex;gap:8px;flex-wrap:wrap;margin-top:8px}}.authActions .sessionBtn,.authActions .btn,.authActions button{{flex:1 1 180px}}.authHint{{margin:8px 0 0;color:var(--muted);font-size:12px;line-height:1.35}}.authRememberToggle{{display:inline-flex;align-items:center;gap:8px;min-height:34px;padding:7px 11px;border:1px solid rgba(255,255,255,.10);border-radius:999px;background:rgba(255,255,255,.05);color:var(--muted);font-size:12px;font-weight:750;cursor:pointer;user-select:none}}.authRememberToggle input{{width:14px;height:14px;margin:0;flex:0 0 14px;accent-color:#22c55e}}.authSecretBox{{margin-top:10px;padding:10px;border:1px solid rgba(34,211,238,.24);border-radius:8px;background:rgba(34,211,238,.08)}}.authSecretBox strong{{display:block;margin-bottom:6px;font-size:12px;color:#f7f2ff}}.authSecretValue{{margin:0;padding:9px 10px;border:1px solid rgba(255,255,255,.08);border-radius:8px;background:rgba(0,0,0,.18);color:#c4b5fd;font:12px/1.4 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;word-break:break-word}}.authList{{display:grid;gap:8px;margin-top:10px}}.authRow{{display:grid;grid-template-columns:1fr auto;gap:8px;align-items:start;border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px}}.authRowAll{{border-style:dashed;background:linear-gradient(135deg,rgba(34,211,238,.10),rgba(34,197,94,.08),rgba(255,255,255,.04))}}.authRowTitle{{display:flex;align-items:center;gap:8px;flex-wrap:wrap;font-weight:900}}.authRowMeta{{margin-top:4px;color:var(--muted);font-size:12px;line-height:1.35;word-break:break-word}}.authRowKey{{margin-top:7px;padding:8px;border:1px solid rgba(255,255,255,.08);border-radius:7px;background:rgba(0,0,0,.18);color:#c4b5fd;font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;word-break:break-all}}.authRouterFlags{{justify-content:flex-end}}.authFlagPill{{gap:6px;min-height:30px;padding:5px 8px;cursor:pointer;user-select:nonLine truncated .authGroup{{margin-top:14px;border:1px solid rgba(34,211,238,.22);border-radius:14px;background:linear-gradient(180deg,rgba(255,255,255,.06),rgba(255,255,255,.03)),rgba(15,10,26,.78);box-shadow:inset 0 1px 0 rgba(255,255,255,.04)}}.authGroupSummary{{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:12px 14px;cursor:pointer;list-style:none}}.authGroupSummary::-webkit-details-marker{{display:none}}.authGroupTitle strong{{display:block;color:#f7f2ff;font-size:15px;font-weight:900;line-height:1.15}}.authGroupTitle span{{display:block;margin-top:4px;color:var(--muted);font-size:12px;line-height:1.35}}.authGroupChevron{{display:inline-flex;align-items:center;justify-content:center;flex:0 0 28px;width:28px;height:28px;border:1px solid rgba(34,211,238,.24);border-radius:999px;background:rgba(34,211,238,.08);color:#dbeafe;font-size:16px;line-height:1;transition:transform .18s ease,background .18s ease,border-color .18s ease}}.authGroup[open] .authGroupChevron{{transform:rotate(180deg);background:rgba(34,197,94,.12);border-color:rgba(34,197,94,.24)}}.authGroupBody{{padding:0 14px 14px;border-top:1px solid rgba(255,255,255,.08)}}.authGroupBody>.authPasswordSection{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.authOverview{{margin-top:0;padding-top:14px;border-top:0}}.authGroupBody>.sessionBox{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.notifyBox{{margin-top:14px;padding-top:0;border-top:0}}.authGroupBody>.backupBox{{margin-top:14px;padding-top:0;border-top:0}}.authOverview{{display:grid;gap:10px;margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.authPills{{display:flex;flex-wrap:wrap;gap:8px}}.authPill{{display:inline-flex;align-items:center;gap:7px;padding:7px 10px;border:1px solid rgba(34,211,238,.24);border-radius:999px;background:rgba(34,211,238,.08);color:#dbeafe;font-size:12px;font-weight:850}}.authPill.off{{border-color:rgba(255,255,255,.10);background:rgba(255,255,255,.06);color:#c4b5fd}}.authSection{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.authSectionHead{{display:flex;align-items:flex-start;justify-content:space-between;gap:10px;margin-bottom:8px}}.authSectionHead h3{{margin:0;font-size:15px}}.authSectionHead p{{margin:4px 0 0;color:var(--muted);font-size:12px;line-height:1.35}}.authSectionState{{display:inline-flex;align-items:center;justify-content:center;min-height:30px;padding:6px 10px;border:1px solid rgba(34,197,94,.30);border-radius:999px;background:rgba(34,197,94,.12);color:#bbf7d0;font-size:12px;font-weight:850;white-space:nowrap}}.authSectionState.off{{border-color:rgba(255,255,255,.10);background:rgba(255,255,255,.06);color:#c4b5fd}}.authSectionState.warn{{border-color:rgba(251,191,36,.28);background:rgba(251,191,36,.11);color:#fde68a}}.authFields{{display:grid;grid-template-columns:1fr 1fr;gap:8px}}.authFields .wide{{grid-column:1/-1}}.authFields input,.authFields textarea{{width:100%;min-width:0}}.authFields textarea{{min-height:92px;resize:vertical}}.authActions{{display:flex;gap:8px;flex-wrap:wrap;margin-top:8px}}.authActions .sessionBtn,.authActions .btn,.authActions button{{flex:1 1 180px}}.authHint{{margin:8px 0 0;color:var(--muted);font-size:12px;line-height:1.35}}.authRememberToggle{{display:inline-flex;align-items:center;gap:8px;min-height:34px;padding:7px 11px;border:1px solid rgba(255,255,255,.10);border-radius:999px;background:rgba(255,255,255,.05);color:var(--muted);font-size:12px;font-weight:750;cursor:pointer;user-select:none}}.authRememberToggle input{{width:14px;height:14px;margin:0;flex:0 0 14px;accent-color:#22c55e}}.authSecretBox{{margin-top:10px;padding:10px;border:1px solid rgba(34,211,238,.24);border-radius:8px;background:rgba(34,211,238,.08)}}.authSecretBox strong{{display:block;margin-bottom:6px;font-size:12px;color:#f7f2ff}}.authSecretValue{{margin:0;padding:9px 10px;border:1px solid rgba(255,255,255,.08);border-radius:8px;background:rgba(0,0,0,.18);color:#c4b5fd;font:12px/1.4 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;word-break:break-word}}.authList{{display:grid;gap:8px;margin-top:10px}}.authRow{{display:grid;grid-template-columns:1fr auto;gap:8px;align-items:start;border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px}}.authRowAll{{border-style:dashed;background:linear-gradient(135deg,rgba(34,211,238,.10),rgba(34,197,94,.08),rgba(255,255,255,.04))}}.authRowTitle{{display:flex;align-items:center;gap:8px;flex-wrap:wrap;font-weight:900}}.authRowMeta{{margin-top:4px;color:var(--muted);font-size:12px;line-height:1.35;word-break:break-word}}.authRowKey{{margin-top:7px;padding:8px;border:1px solid rgba(255,255,255,.08);border-radius:7px;background:rgba(0,0,0,.18);color:#c4b5fd;font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;word-break:break-all}}.authRouterFlags{{justify-content:flex-end}}.authFlagPill{{gap:6px;min-height:30px;padding:5px 8px;cursor:pointer;user-select:nonLine truncated
.sessionBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.sessionHead{{display:flex;align-items:center;justify-content:space-between;gap:10px;margin-bottom:8px}}.sessionHead h3{{margin:0;font-size:15px}}.sessionList{{display:grid;gap:8px;max-height:260px;overflow:auto;padding-right:2px}}.sessionRow{{display:grid;grid-template-columns:1fr auto;gap:8px;align-items:center;border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px;text-align:left}}.sessionTitle{{display:flex;gap:7px;align-items:center;flex-wrap:wrap;font-weight:900}}.sessionMeta{{margin-top:3px;color:var(--muted);font-size:12px;line-height:1.35;word-break:break-word}}.sessionCurrent{{border:1px solid rgba(34,197,94,.38);border-radius:999px;padding:2px 7px;color:#bbf7d0;background:rgba(34,197,94,.13);font-size:11px}}.sessionBtn{{padding:7px 9px;font-size:12px;border-radius:999px}}.sessionEmpty{{padding:10px;border:1px dashed var(--line);border-radius:8px;color:var(--muted);text-align:center}} .sessionBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.sessionHead{{display:flex;align-items:center;justify-content:space-between;gap:10px;margin-bottom:8px}}.sessionHead h3{{margin:0;font-size:15px}}.sessionList{{display:grid;gap:8px;max-height:260px;overflow:auto;padding-right:2px}}.sessionRow{{display:grid;grid-template-columns:1fr auto;gap:8px;align-items:center;border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px;text-align:left}}.sessionTitle{{display:flex;gap:7px;align-items:center;flex-wrap:wrap;font-weight:900}}.sessionMeta{{margin-top:3px;color:var(--muted);font-size:12px;line-height:1.35;word-break:break-word}}.sessionCurrent{{border:1px solid rgba(34,197,94,.38);border-radius:999px;padding:2px 7px;color:#bbf7d0;background:rgba(34,197,94,.13);font-size:11px}}.sessionBtn{{padding:7px 9px;font-size:12px;border-radius:999px}}.sessionEmpty{{padding:10px;border:1px dashed var(--line);border-radius:8px;color:var(--muted);text-align:center}}
.notifyBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.notifyActions{{display:flex;gap:7px;align-items:center;justify-content:flex-end;flex-wrap:wrap}}.notifyHint{{margin:-4px 0 10px;color:var(--muted);font-size:12px;line-height:1.35}}.notifyList{{display:grid;gap:8px;max-height:260px;overflow:auto;padding-right:2px}}.notifyRow{{border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px;text-align:left}}.notifyTitle{{display:flex;align-items:center;justify-content:space-between;gap:8px;font-weight:950}}.notifyTitle span:first-child{{min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}}.notifyTime{{color:var(--muted);font-size:11px;font-weight:750;white-space:nowrap}}.notifyBody{{margin-top:4px;color:#ddd6fe;font-size:12px;line-height:1.35;word-break:break-word}}.notifyDetails{{margin:7px 0 0;padding:8px;border:1px solid rgba(255,255,255,.08);border-radius:7px;background:rgba(0,0,0,.18);color:#c4b5fd;font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;max-height:92px;overflow:auto}}.notifyRow.warn{{border-color:rgba(245,158,11,.34);background:rgba(245,158,11,.08)}}.notifyRow.bad{{border-color:rgba(251,113,133,.38);background:rgba(251,113,133,.09)}}.notifyBtn.on{{border-color:rgba(34,197,94,.36);background:rgba(34,197,94,.15);color:#bbf7d0}} .notifyBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.notifyActions{{display:flex;gap:7px;align-items:center;justify-content:flex-end;flex-wrap:wrap}}.notifyHint{{margin:-4px 0 10px;color:var(--muted);font-size:12px;line-height:1.35}}.notifyList{{display:grid;gap:8px;max-height:260px;overflow:auto;padding-right:2px}}.notifyRow{{border:1px solid var(--line);border-radius:8px;background:rgba(255,255,255,.055);padding:9px;text-align:left}}.notifyTitle{{display:flex;align-items:center;justify-content:space-between;gap:8px;font-weight:950}}.notifyTitle span:first-child{{min-width:0;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}}.notifyTime{{color:var(--muted);font-size:11px;font-weight:750;white-space:nowrap}}.notifyBody{{margin-top:4px;color:#ddd6fe;font-size:12px;line-height:1.35;word-break:break-word}}.notifyDetails{{margin:7px 0 0;padding:8px;border:1px solid rgba(255,255,255,.08);border-radius:7px;background:rgba(0,0,0,.18);color:#c4b5fd;font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap;max-height:92px;overflow:auto}}.notifyRow.warn{{border-color:rgba(245,158,11,.34);background:rgba(245,158,11,.08)}}.notifyRow.bad{{border-color:rgba(251,113,133,.38);background:rgba(251,113,133,.09)}}.notifyBtn.on{{border-color:rgba(34,197,94,.36);background:rgba(34,197,94,.15);color:#bbf7d0}}
.backupBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.backupGrid{{display:grid;grid-template-columns:1fr 1fr;gap:8px}}.backupGrid .wide{{grid-column:1/-1}}.backupGrid input[type=file]{{padding:8px;background:rgba(8,5,18,.72);border:1px solid var(--line);border-radius:8px;color:#ddd6fe;min-width:0}}.backupHint{{margin:0 0 10px;color:var(--muted);font-size:12px;line-height:1.35}}.backupCmds{{display:grid;gap:8px;margin-top:10px}}.backupCmd{{border:1px solid rgba(255,255,255,.10);border-radius:8px;background:rgba(0,0,0,.18);padding:9px}}.backupCmd strong{{display:block;margin-bottom:5px;color:#f7f2ff;font-size:12px}}.backupCmd pre{{margin:0;white-space:pre-wrap;word-break:break-word;color:#c4b5fd;font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace}}#backupRewrite{{border-color:rgba(251,191,36,.42);background:linear-gradient(135deg,rgba(251,191,36,.28),rgba(245,158,11,.22));color:#fff7cc;box-shadow:0 10px 24px rgba(245,158,11,.14),inset 0 1px 0 rgba(255,255,255,.12)}}#backupRewrite:hover{{border-color:rgba(251,191,36,.56);background:linear-gradient(135deg,rgba(251,191,36,.34),rgba(245,158,11,.28))}}.backupMsg{{margin-top:9px;color:#bbf7d0;font-size:12px;font-weight:800;line-height:1.35}}.backupMsg.bad{{color:#fecdd3}} .backupBox{{margin-top:14px;padding-top:12px;border-top:1px solid var(--line)}}.backupGrid{{display:grid;grid-template-columns:1fr 1fr;gap:8px}}.backupGrid .wide{{grid-column:1/-1}}.backupGrid input[type=file]{{padding:8px;background:rgba(8,5,18,.72);border:1px solid var(--line);border-radius:8px;color:#ddd6fe;min-width:0}}.backupHint{{margin:0 0 10px;color:var(--muted);font-size:12px;line-height:1.35}}#backupRewrite{{border-color:rgba(251,191,36,.42);background:linear-gradient(135deg,rgba(251,191,36,.28),rgba(245,158,11,.22));color:#fff7cc;box-shadow:0 10px 24px rgba(245,158,11,.14),inset 0 1px 0 rgba(255,255,255,.12)}}#backupRewrite:hover{{border-color:rgba(251,191,36,.56);background:linear-gradient(135deg,rgba(251,191,36,.34),rgba(245,158,11,.28))}}.backupMsg{{margin-top:9px;color:#bbf7d0;font-size:12px;font-weight:800;line-height:1.35}}.backupMsg.bad{{color:#fecdd3}}
input,select{{min-width:0;border:1px solid var(--line);border-radius:8px;padding:10px 11px;background:rgba(8,5,18,.72);color:var(--text)}}button,.btn{{border:1px solid rgba(255,255,255,.10);border-radius:8px;padding:10px 13px;background:rgba(255,255,255,.10);color:#f7f2ff;font-weight:850;text-decoration:none;cursor:pointer;display:inline-flex;justify-content:center;align-items:center}}.authToggle{{border-radius:999px;padding:8px 14px;background:rgba(255,255,255,.08);color:#f3e8ff}}button.primary,.btn.primary{{background:var(--blue);color:#fff;box-shadow:0 10px 22px rgba(124,58,237,.22)}}button.bad,.btn.bad{{background:rgba(251,113,133,.16);color:#fecdd3}}.btn.good{{background:rgba(34,197,94,.16);color:#bbf7d0}}.btn.disabled{{opacity:.45;cursor:not-allowed}} input,select{{min-width:0;border:1px solid var(--line);border-radius:8px;padding:10px 11px;background:rgba(8,5,18,.72);color:var(--text)}}button,.btn{{border:1px solid rgba(255,255,255,.10);border-radius:8px;padding:10px 13px;background:rgba(255,255,255,.10);color:#f7f2ff;font-weight:850;text-decoration:none;cursor:pointer;display:inline-flex;justify-content:center;align-items:center}}.authToggle{{border-radius:999px;padding:8px 14px;background:rgba(255,255,255,.08);color:#f3e8ff}}button.primary,.btn.primary{{background:var(--blue);color:#fff;box-shadow:0 10px 22px rgba(124,58,237,.22)}}button.bad,.btn.bad{{background:rgba(251,113,133,.16);color:#fecdd3}}.btn.good{{background:rgba(34,197,94,.16);color:#bbf7d0}}.btn.disabled{{opacity:.45;cursor:not-allowed}}
.summary{{display:flex;gap:10px;flex-wrap:wrap;justify-content:flex-end}}.miniStat{{display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;border:1px solid var(--line);border-radius:999px;background:rgba(255,255,255,.07);padding:8px 12px;color:#ddd6fe;font-weight:800;font-size:13px;line-height:1;white-space:nowrap}} .summary{{display:flex;gap:10px;flex-wrap:wrap;justify-content:flex-end}}.miniStat{{display:inline-flex;align-items:center;justify-content:center;gap:8px;min-height:36px;min-width:132px;border:1px solid var(--line);border-radius:999px;background:rgba(255,255,255,.07);padding:8px 12px;color:#ddd6fe;font-weight:800;font-size:13px;line-height:1;white-space:nowrap}}
.routerStats{{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));align-items:start;gap:8px;margin:0 0 10px}}.statCard{{position:relative;min-height:54px;overflow:hidden;border:1px solid var(--line);border-radius:8px;padding:8px 10px 7px;background:linear-gradient(180deg,rgba(255,255,255,.075),rgba(255,255,255,.04)),var(--panel);box-shadow:0 8px 20px rgba(0,0,0,.14);text-align:center;box-sizing:border-box}}.statCard::before{{content:"";position:absolute;inset:0 0 auto 0;height:2px;background:var(--cyan)}}.statCard span{{display:block;color:var(--muted);font-size:10px;font-weight:850;text-transform:uppercase;letter-spacing:.035em}}.statCard strong{{display:block;margin-top:0;font-size:22px;line-height:1;color:#f7f2ff}}.statCard em{{display:block;min-width:0;margin-top:2px;padding:0 2px;color:#c4b5fd;font-style:normal;font-size:10px;line-height:1.15}}.statCard.online::before{{background:var(--green)}}.statCard.online strong{{color:#bbf7d0}}.statCard.offline::before{{background:var(--red)}}.statCard.offline strong{{color:#fecdd3}}.statCard.total::before{{background:var(--cyan)}}.statCard.total strong{{color:#a5f3fc}}.statCardHead{{display:flex;align-items:center;justify-content:center;min-height:14px}}.statCard.has-popover{{overflow:visible;z-index:12}}.statValueRow,.offlineStatRow{{position:relative;display:flex;align-items:center;justify-content:center;min-height:20px;margin-top:3px}}.offlineMoreBtn{{position:absolute;right:6px;top:50%;transform:translateY(-50%);display:inline-flex;align-items:center;justify-content:center;min-height:22px;padding:0 8px;border:1px solid rgba(251,113,133,.34);border-radius:999px;background:rgba(251,113,133,.12);color:#fecdd3;font-size:10px;font-weight:900;line-height:1;cursor:pointer;white-space:nowrap}}.offlineMoreBtn[aria-expanded="true"]{{background:rgba(251,113,133,.18);border-color:rgba(251,113,133,.48)}}.offlinePopover{{position:absolute;top:calc(100% + 8px);right:6px;width:min(340px,calc(100vw - 30px));padding:10px;border:1px solid rgba(251,113,133,.28);border-radius:12px;background:linear-gradient(180deg,rgba(49,28,44,.98),rgba(30,18,40,.98));box-shadow:0 22px 50px rgba(0,0,0,.34);text-align:left;z-index:25}}.offlinePopover[hidden]{{display:none!important}}.offlineList{{display:grid;gap:6px;max-height:170px;overflow:auto;padding-right:2px}}.offlineItem{{border:1px solid rgba(251,113,133,.18);border-radius:8px;padding:7px 8px;background:rgba(251,113,133,.08)}}.offlineName{{display:block;color:#fdf2f8;font-size:11px;font-weight:900;line-height:1.25;text-transform:none;letter-spacing:0}} .routerStats{{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));align-items:start;gap:8px;margin:0 0 10px}}.statCard{{position:relative;min-height:54px;overflow:hidden;border:1px solid var(--line);border-radius:8px;padding:8px 10px 7px;background:linear-gradient(180deg,rgba(255,255,255,.075),rgba(255,255,255,.04)),var(--panel);box-shadow:0 8px 20px rgba(0,0,0,.14);text-align:center;box-sizing:border-box}}.statCard::before{{content:"";position:absolute;inset:0 0 auto 0;height:2px;background:var(--cyan)}}.statCard span{{display:block;color:var(--muted);font-size:10px;font-weight:850;text-transform:uppercase;letter-spacing:.035em}}.statCard strong{{display:block;margin-top:0;font-size:22px;line-height:1;color:#f7f2ff}}.statCard em{{display:block;min-width:0;margin-top:2px;padding:0 2px;color:#c4b5fd;font-style:normal;font-size:10px;line-height:1.15}}.statCard.online::before{{background:var(--green)}}.statCard.online strong{{color:#bbf7d0}}.statCard.offline::before{{background:var(--red)}}.statCard.offline strong{{color:#fecdd3}}.statCard.total::before{{background:var(--cyan)}}.statCard.total strong{{color:#a5f3fc}}.statCardHead{{display:flex;align-items:center;justify-content:center;min-height:14px}}.statCard.has-popover{{overflow:visible;z-index:12}}.statValueRow,.offlineStatRow{{position:relative;display:flex;align-items:center;justify-content:center;min-height:20px;margin-top:3px}}.offlineMoreBtn{{position:absolute;right:6px;top:50%;transform:translateY(-50%);display:inline-flex;align-items:center;justify-content:center;min-height:22px;padding:0 8px;border:1px solid rgba(251,113,133,.34);border-radius:999px;background:rgba(251,113,133,.12);color:#fecdd3;font-size:10px;font-weight:900;line-height:1;cursor:pointer;white-space:nowrap}}.offlineMoreBtn[aria-expanded="true"]{{background:rgba(251,113,133,.18);border-color:rgba(251,113,133,.48)}}.offlinePopover{{position:absolute;top:calc(100% + 8px);right:6px;width:min(340px,calc(100vw - 30px));padding:10px;border:1px solid rgba(251,113,133,.28);border-radius:12px;background:linear-gradient(180deg,rgba(49,28,44,.98),rgba(30,18,40,.98));box-shadow:0 22px 50px rgba(0,0,0,.34);text-align:left;z-index:25}}.offlinePopover[hidden]{{display:none!important}}.offlineList{{display:grid;gap:6px;max-height:170px;overflow:auto;padding-right:2px}}.offlineItem{{border:1px solid rgba(251,113,133,.18);border-radius:8px;padding:7px 8px;background:rgba(251,113,133,.08)}}.offlineName{{display:block;color:#fdf2f8;font-size:11px;font-weight:900;line-height:1.25;text-transform:none;letter-spacing:0}}
@@ -5022,7 +5065,7 @@ input,select{{min-width:0;border:1px solid var(--line);border-radius:8px;padding
<div class="brand"> <div class="brand">
<div class="desktopHeader"> <div class="desktopHeader">
<div class="desktopHeaderTop"> <div class="desktopHeaderTop">
<h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v116</span></span></h1> <h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v117</span></span></h1>
<div class="routerSearchDock" id="routerSearchDock"> <div class="routerSearchDock" id="routerSearchDock">
<button class="routerSearchToggle" id="routerSearchToggle" type="button" aria-expanded="false" aria-controls="routerSearchPanel" data-active="false"> <button class="routerSearchToggle" id="routerSearchToggle" type="button" aria-expanded="false" aria-controls="routerSearchPanel" data-active="false">
<span>Поиск роутеров</span> <span>Поиск роутеров</span>
@@ -5334,29 +5377,6 @@ input,select{{min-width:0;border:1px solid var(--line);border-radius:8px;padding
<a class="sessionBtn btn good" id="backupDownload" href="/api/backup/download">Скачать</a> <a class="sessionBtn btn good" id="backupDownload" href="/api/backup/download">Скачать</a>
</div> </div>
<div class="backupHint">Архив переносит БД роутеров, логин, токены Hub/агента, push-ключи и уведомления. Для переезда на другой IP укажи новый VPS host/Public URL перед восстановлением.</div> <div class="backupHint">Архив переносит БД роутеров, логин, токены Hub/агента, push-ключи и уведомления. Для переезда на другой IP укажи новый VPS host/Public URL перед восстановлением.</div>
<div class="backupCmds">
<div class="backupCmd">
<strong>1. Сделать backup на старой VPS</strong>
<pre>/opt/owrt-remote/owrt-remote-hub.py backup --out /root/owrt-hub-backup.tar.gz</pre>
</div>
<div class="backupCmd">
<strong>2. Поднять Hub на новой VPS</strong>
<pre>curl -fsSL "https://raw.githubusercontent.com/kzolotarev95/luci-app-owrt-remote/main/vps/install-vps.sh?v=$(date +%s)" | sh</pre>
</div>
<div class="backupCmd">
<strong>3. Перенести архив на новую VPS</strong>
<pre>scp /root/owrt-hub-backup.tar.gz root@NEW_IP:/root/owrt-hub-backup.tar.gz</pre>
</div>
<div class="backupCmd">
<strong>4. Восстановить на новой VPS с новым IP/domain</strong>
<pre>/opt/owrt-remote/owrt-remote-hub.py restore --file /root/owrt-hub-backup.tar.gz --vps-host NEW_IP_OR_DOMAIN --public-url https://NEW_DOMAIN</pre>
</div>
<div class="backupCmd">
<strong>5. Применить после restore</strong>
<pre>systemctl restart owrt-remote
systemctl restart owrt-remote-xray</pre>
</div>
</div>
<div class="backupGrid"> <div class="backupGrid">
<input class="wide" id="backupFile" type="file" accept=".gz,.tgz,.tar.gz,application/gzip,application/x-gzip"> <input class="wide" id="backupFile" type="file" accept=".gz,.tgz,.tar.gz,application/gzip,application/x-gzip">
<input id="backupVpsHost" placeholder="Новый VPS host/IP"> <input id="backupVpsHost" placeholder="Новый VPS host/IP">
@@ -5375,7 +5395,7 @@ systemctl restart owrt-remote-xray</pre>
</div> </div>
</div> </div>
<div class="mobileSearchDock" id="mobileRouterSearchDock"> <div class="mobileSearchDock" id="mobileRouterSearchDock">
<span class="mobileSearchVersion">v116</span> <span class="mobileSearchVersion">v117</span>
<button class="routerSearchToggle mobilePanelToggle primary" id="mobileRouterSearchToggle" type="button" aria-expanded="false" aria-controls="mobileRouterSearchPanel" data-active="false"> <button class="routerSearchToggle mobilePanelToggle primary" id="mobileRouterSearchToggle" type="button" aria-expanded="false" aria-controls="mobileRouterSearchPanel" data-active="false">
<span>Поиск роутеров</span> <span>Поиск роутеров</span>
</button> </button>
@@ -12203,10 +12223,10 @@ def router_logs_html(router):
@media(max-width:680px),(pointer:coarse){body{background-attachment:scroll;background-image:linear-gradient(145deg,#07040f,#120a24 54%,#05030a)}.legend{justify-content:center;text-align:center;row-gap:8px}} @media(max-width:680px),(pointer:coarse){body{background-attachment:scroll;background-image:linear-gradient(145deg,#07040f,#120a24 54%,#05030a)}.legend{justify-content:center;text-align:center;row-gap:8px}}
</style></head><body><main class="wrap"><section class="panel"> </style></head><body><main class="wrap"><section class="panel">
<div class="head"><h1>Глобал логи</h1><a class="btn" href="/">Вернуться в Hub</a></div><h2>__TITLE__</h2> <div class="head"><h1>Глобал логи</h1><a class="btn" href="/">Вернуться в Hub</a></div><h2>__TITLE__</h2>
<blockquote class="note">Действия панели, Heartbeat, WAN, предупреждения и ошибки Xray этого роутера. Копия доставляется на VPS каждые 3 секунды, пока доступна связь. Если роутер оффлайн, доступны последние доставленные записи. На роутере журнал хранится только в ОЗУ и очищается после перезагрузки и каждые 24 часа. Архив VPS сохраняется отдельно: до 2 МБ на роутер. Ключи и пароли скрываются.</blockquote> <blockquote class="note">Действия панели, Heartbeat, WAN, предупреждения и ошибки Xray этого роутера. Доставка проверяется каждые 3 секунды; новые записи обычно появляются после пакетной записи на Flash, до 60 секунд. На обновлённом роутере журнал сохраняется на проверенной Flash, временный буфер ограничен 16 КиБ. Если роутер оффлайн, доступны последние доставленные записи. Архив VPS хранится в SQLite на диске: до 2 МиБ на роутер. Ключи и пароли скрываются.</blockquote>
<div class="legend" aria-label="Цвета журнала"><span class="ok">● Успешно</span><span class="warn">● Предупреждение</span><span class="error">● Ошибка</span></div> <div class="legend" aria-label="Цвета журнала"><span class="ok">● Успешно</span><span class="warn">● Предупреждение</span><span class="error">● Ошибка</span></div>
<div class="actions"><a class="btn" href="__DOWNLOAD__">Скачать логи</a><button class="btn" id="logsLive" type="button">Запустить в реальном времени</button><button class="btn bad" id="logsClear" type="button">Очистить логи</button></div> <div class="actions"><a class="btn" href="__DOWNLOAD__">Скачать логи</a><button class="btn" id="logsLive" type="button">Запустить в реальном времени</button><button class="btn bad" id="logsClear" type="button">Очистить логи</button></div>
<div id="logsDiagnosis" class="warn" role="status">Получаю состояние роутера…</div><div id="logsStatus" role="status">Загружаю сохранённые записи с VPS…</div><pre id="logsText" tabindex="0">Загрузка записей журнала…</pre> <div id="logsDiagnosis" class="warn" role="status">Получаю состояние роутера…</div><div id="logsStorage" role="status"></div><div id="logsStatus" role="status">Загружаю сохранённые записи с VPS…</div><pre id="logsText" tabindex="0">Загрузка записей журнала…</pre>
</section></main><script> </section></main><script>
(function(){ (function(){
const endpoint=__ENDPOINT__, output=document.getElementById('logsText'), status=document.getElementById('logsStatus'), diagnosis=document.getElementById('logsDiagnosis'), live=document.getElementById('logsLive'), clear=document.getElementById('logsClear'); const endpoint=__ENDPOINT__, output=document.getElementById('logsText'), status=document.getElementById('logsStatus'), diagnosis=document.getElementById('logsDiagnosis'), live=document.getElementById('logsLive'), clear=document.getElementById('logsClear');
@@ -12231,6 +12251,8 @@ def router_logs_html(router):
const data=await response.json();if(current!==generation)return; const data=await response.json();if(current!==generation)return;
render(data.text||''); render(data.text||'');
showDiagnosis(data); showDiagnosis(data);
const storage=data.storage, storageLabel=document.getElementById('logsStorage');
if(storage){storageLabel.className=storage.persistent===false?'error':storage.persistent===true?'ok':'warn';storageLabel.textContent=(storage.persistent===false?'Внимание: архив VPS находится в RAM и исчезнет после перезагрузки.':storage.persistent===true?'Архив VPS: постоянный диск.':'Архив VPS: тип хранилища не подтверждён.')+' '+storage.path+' · '+storage.filesystem+' · Лимит: 2 МиБ на роутер.';}
status.textContent=(data.online?'Роутер онлайн':'Роутер оффлайн — показан архив VPS')+' · Последняя доставка: '+stamp(data.last_sync)+' · Объём архива: '+Math.ceil(data.bytes/1024)+' КБ'+(data.bytes>65536?' · На экране последние 64 КБ; полный журнал доступен через «Скачать логи».':'')+(data.pending_clear?' · Очистка журнала роутера ожидает связи.':''); status.textContent=(data.online?'Роутер онлайн':'Роутер оффлайн — показан архив VPS')+' · Последняя доставка: '+stamp(data.last_sync)+' · Объём архива: '+Math.ceil(data.bytes/1024)+' КБ'+(data.bytes>65536?' · На экране последние 64 КБ; полный журнал доступен через «Скачать логи».':'')+(data.pending_clear?' · Очистка журнала роутера ожидает связи.':'');
}catch(error){if(current===generation)status.textContent='Не удалось получить логи: '+error.message+(running?'. Повтор через 3 секунды.':'');} }catch(error){if(current===generation)status.textContent='Не удалось получить логи: '+error.message+(running?'. Повтор через 3 секунды.':'');}
finally{clearTimeout(timeout);if(controller===request)controller=null;if(current===generation&&running&&!document.hidden)timer=setTimeout(refresh,3000);} finally{clearTimeout(timeout);if(controller===request)controller=null;if(current===generation&&running&&!document.hidden)timer=setTimeout(refresh,3000);}
@@ -12815,7 +12837,7 @@ button:hover{{filter:brightness(1.06)}}
<form class="login" method="post" action="/login"> <form class="login" method="post" action="/login">
{error_html} {error_html}
<span class="brand"> <span class="brand">
<h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v116</span></span></h1> <h1 class="appBanner"><span>OpenWrt Remote Hub <span class="appBannerVersion">v117</span></span></h1>
</span> </span>
<label for="hubUsername">Логин</label> <label for="hubUsername">Логин</label>
<input id="hubUsername" name="username" autocomplete="off" autofocus required> <input id="hubUsername" name="username" autocomplete="off" autofocus required>
@@ -13287,7 +13309,7 @@ body::after{{content:"";position:fixed;inset:0;pointer-events:none;background:li
<circle cx="65" cy="59" r="3" fill="#E5F2FF"/> <circle cx="65" cy="59" r="3" fill="#E5F2FF"/>
</svg> </svg>
</div> </div>
<h2 class="brandTitle">OpenWrt Remote Hub <span class="brandVersion">v116</span></h2> <h2 class="brandTitle">OpenWrt Remote Hub <span class="brandVersion">v117</span></h2>
</div> </div>
</div> </div>
<div class="brandBottom"> <div class="brandBottom">
@@ -15852,7 +15874,8 @@ a{{display:inline-flex;margin-top:18px;color:#93c5fd}}
self.send_bytes(200, raw, 'text/plain; charset=utf-8', self.send_bytes(200, raw, 'text/plain; charset=utf-8',
[('Cache-Control', 'no-store'), ('Content-Disposition', f'attachment; filename="{filename}"')]) [('Cache-Control', 'no-store'), ('Content-Disposition', f'attachment; filename="{filename}"')])
else: else:
payload = {'ok': True, 'router_id': router_id, 'online': router['online'], **snapshot} payload = {'ok': True, 'router_id': router_id, 'online': router['online'],
'storage': router_log_storage(self.app.db_path), **snapshot}
if mode == 'clear': if mode == 'clear':
payload['message'] = 'Архив VPS очищен. Журнал роутера будет очищен при следующем соединении.' payload['message'] = 'Архив VPS очищен. Журнал роутера будет очищен при следующем соединении.'
self.send_json(200, payload) self.send_json(200, payload)