Fix SQLite connection leaks and resource cleanup
This commit is contained in:
1 parent
e677fd2391
commit
f01f9c799a
5 files changed
+1177
-173
No files matched your search
@@ -0,0 +1,136 @@
|
||||
# Аудит ресурсов OpenWrt Remote Hub v109
|
||||
|
||||
Исправления внесены поверх `7f809fd`. Версия панели остаётся v109.
|
||||
|
||||
## SQLite
|
||||
|
||||
В исходном коде нативный `sqlite3.Connection.__exit__` управлял транзакцией,
|
||||
но не закрывал соединение. Это затрагивало мониторинг, API и CLI.
|
||||
|
||||
Теперь `connect()` — единственный владелец создаваемого SQLite connection:
|
||||
|
||||
```python
|
||||
with connect(db_path) as conn:
|
||||
# Работа с БД только внутри этого блока.
|
||||
...
|
||||
```
|
||||
|
||||
Внутренний `with conn` сохраняет штатную семантику SQLite commit/rollback,
|
||||
включая rollback после неудачного commit и `BaseException`. Внешний `finally`
|
||||
всегда вызывает `close()`, в том числе при ошибке установки `row_factory`,
|
||||
инициализации схемы, выходе через `return` и ошибке завершения транзакции.
|
||||
Ошибки SQLite продолжают передаваться вызывающему коду.
|
||||
|
||||
`App.conn()` инициализирует схему внутри того же защищённого контекста.
|
||||
Аудит AST проверил один прямой `sqlite3.connect`, 13 вызовов общего `connect`,
|
||||
30 вызовов `self.app.conn`, один `self.conn` и один `app.conn`.
|
||||
Все потребители контекстных менеджеров используют `with`.
|
||||
|
||||
Проверены мониторинг, heartbeat, группы, заметки, выдача конфигураций,
|
||||
HTTP handlers, CLI, backup/restore и обновление Xray. Отдельных долгоживущих
|
||||
cursor нет; результаты извлекаются внутри операции, connection закрывает
|
||||
оставшиеся SQLite ресурсы. Backup закрывает source даже при ошибке открытия
|
||||
destination. Возвращаемые `sqlite3.Row` остаются доступны после закрытия БД.
|
||||
|
||||
## Другие ресурсы
|
||||
|
||||
Исправлены найденные пути с неполной очисткой:
|
||||
|
||||
* HTTPError OAuth закрывается после чтения тела ошибки.
|
||||
* Ответы Web Push закрываются при успехе и при ответе с ошибкой.
|
||||
* Ответ proxy закрывается контекстным менеджером, backend — через `finally`.
|
||||
* HTTP сервер освобождает listening socket при ошибке TLS; главный и
|
||||
дополнительные серверы закрываются при выходе или ошибке запуска.
|
||||
* Монитор останавливается с ожиданием потока; неудачный запуск потока
|
||||
не ломает cleanup. Ошибка первого снимка БД логируется, монитор повторяет
|
||||
попытки с обычной паузой.
|
||||
* HTTP terminal reader владеет PTY и сбором дочернего процесса. При
|
||||
неудачном запуске reader или первой записи WebSocket ресурсы освобождаются.
|
||||
Завершение ребёнка ожидается; при необходимости применяются HUP/TERM/KILL.
|
||||
* Запись в HTTP terminal использует собственный временный дубликат FD,
|
||||
который закрывается в `finally`, чтобы избежать гонки с reader.
|
||||
* Закрытые terminal sessions удаляются спустя 60 секунд. HTTP sessions без
|
||||
обращений браузера завершаются после 15 минут; живой polling продлевает их.
|
||||
Буфер вывода ограничен 256 порциями по максимуму 4096 байт исходных данных.
|
||||
При остановке сервера sessions помечаются закрытыми, reader threads ожидаются.
|
||||
* Прерванный chunked request вызывает ошибку вместо бесконечного цикла на EOF.
|
||||
Для сетевого чтения/записи установлен timeout 90 секунд. Долгий terminal
|
||||
без ввода не истекает по этому timeout: его цикл ждёт данные через `select`.
|
||||
* Атомарная запись и создание архива используют уникальные временные файлы
|
||||
с очисткой при ошибке. SSH askpass очищается при неудачной подготовке
|
||||
и после subprocess, включая timeout и ошибку подготовки аргументов.
|
||||
|
||||
Проверенные прежние безопасные участки: `urlopen` успеха и TCP-проверки
|
||||
используют `with`; `TemporaryDirectory` в подписи SSH, backup/restore и
|
||||
upload/download очищаются контекстом; `subprocess.run` использует конечные
|
||||
timeout и собирает child/PIPE после завершения или timeout. Web Push worker
|
||||
имеет конечный список подписок и timeout HTTP; notification long polling
|
||||
ждёт Condition с конечным deadline. Эти механизмы не отключались.
|
||||
Лимит FD и конфигурация systemd не изменялись.
|
||||
|
||||
## Автоматическая проверка
|
||||
|
||||
```powershell
|
||||
python -m unittest discover -s tests -v
|
||||
python tests/test_resource_lifecycle.py --soak-seconds 600
|
||||
```
|
||||
|
||||
Тесты используют отдельные временные state/DB и loopback fixture роутера.
|
||||
Они удерживают ссылки на connection, чтобы GC не маскировал утечку.
|
||||
Проверяются 500 snapshots, commit, rollback, ранний return, init failure,
|
||||
ошибка commit, setup/transaction exit failure, backup и остальные
|
||||
исправленные пути очистки. AST-тест запрещает вызовы DB factory вне `with`.
|
||||
|
||||
В Windows два POSIX-теста пропускаются: реальный PTY/child и сигналы.
|
||||
В Linux установщик запускает их до замены рабочего файла на VPS.
|
||||
CI в этом репозитории отсутствует; тесты подготовлены для его последующего
|
||||
подключения без сторонних Python зависимостей.
|
||||
|
||||
В исходном `7f809fd` тест с удержанными connection показал 101 незакрытое
|
||||
соединение и рост Windows handles с 227 до 327 после 100 snapshots.
|
||||
После фикса 500 snapshots не оставляют открытых connection.
|
||||
|
||||
Первый локальный длительный прогон: 601.1 секунды, 116 циклов
|
||||
панель/API/heartbeat/proxy/notifications/snapshot, handles 279 → 277,
|
||||
максимум 281, CPU time 8.39 секунды. Это Windows handles, а не Linux FD.
|
||||
Второй прогон после изменений сетевого cleanup: 601.2 секунды, 116 циклов,
|
||||
handles 279 → 277, максимум 281, CPU time 8.73 секунды.
|
||||
Локальный тест подтверждает fixture, а не реальные туннели/доставку push/nginx.
|
||||
|
||||
## Установка и откат для проверки пользователем
|
||||
|
||||
Из PowerShell Windows 11:
|
||||
|
||||
```powershell
|
||||
$Vps = "root@YOUR_VPS_IP"
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix.ps1" -Vps $Vps -Action Install
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix.ps1" -Vps $Vps -Action Monitor -Minutes 10
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "vps\deploy-resource-fix.ps1" -Vps $Vps -Action Rollback
|
||||
```
|
||||
|
||||
Первые две команды выполняются последовательно. Откат — отдельная команда
|
||||
только при необходимости. Рабочая директория — корень репозитория.
|
||||
Вместо `YOUR_VPS_IP` укажите адрес своего VPS.
|
||||
Для длительного мониторинга можно использовать `-Minutes 120`.
|
||||
|
||||
Установщик загружает локальный файл без Git, запускает изолированные тесты,
|
||||
сохраняет исходный серверный файл:
|
||||
`/opt/owrt-remote/owrt-remote-hub.py.bak-before-resource-fix-v109`,
|
||||
атомарно заменяет Python файл, перезапускает `owrt-remote` и проверяет health.
|
||||
Существующая резервная копия не перезаписывается. При ошибке установки
|
||||
восстанавливается файл, работавший непосредственно перед текущей попыткой.
|
||||
База и конфигурация роутеров этим скриптом не заменяются.
|
||||
|
||||
Монитор читает `/proc/MainPID/fd`, hub.db FD, CPU за интервал, RSS, threads,
|
||||
health и journal. Отчёт остаётся в `/tmp/owrt-resource-monitor-*.jsonl`.
|
||||
Во время мониторинга нужно пользоваться реальной панелью и proxy роутера.
|
||||
Значения FD могут кратковременно расти из-за активных запросов; важны
|
||||
отсутствие накопления hub.db FD и устойчивого восходящего тренда между ними.
|
||||
|
||||
## Что ещё требуется на реальном VPS
|
||||
|
||||
Сам агент VPS не обновлял и длинный серверный тест не проводил. До финального
|
||||
закрытия инцидента нужны минимум 10 минут Linux мониторинга, желательно 1–2
|
||||
часа с реальными роутерами, проверка online/offline уведомлений, Web Push,
|
||||
heartbeat, SSH, API и nginx error log. Ошибки upstream/504 могут иметь
|
||||
дополнительные причины; их отсутствие нельзя подтвердить Windows fixture.
|
||||
@@ -0,0 +1,101 @@
|
||||
"""Read-only Linux monitoring of owrt-remote; no credentials/DB writes needed.
|
||||
|
||||
Usage on VPS: python3 monitor_vps_resources.py 10
|
||||
FD samples can briefly include active requests or SQLite transactions.
|
||||
The report records trends; real router/proxy/browser actions should run alongside.
|
||||
"""
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
|
||||
def main(minutes):
|
||||
started = time.monotonic()
|
||||
since = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%d-%H%M%S")
|
||||
report_path = Path(f"/tmp/owrt-resource-monitor-{stamp}.jsonl")
|
||||
ticks_per_second = os.sysconf("SC_CLK_TCK")
|
||||
samples = []
|
||||
previous = None
|
||||
print(f"Read-only monitoring for {minutes} minute(s). Report: {report_path}", flush=True)
|
||||
with report_path.open("x", encoding="utf-8") as report:
|
||||
while True:
|
||||
pid = int(subprocess.check_output(["systemctl", "show", "owrt-remote", "-p", "MainPID", "--value"], text=True, timeout=10).strip())
|
||||
if pid <= 0:
|
||||
raise RuntimeError("owrt-remote has no running MainPID")
|
||||
proc = Path(f"/proc/{pid}")
|
||||
links = []
|
||||
for item in (proc / "fd").iterdir():
|
||||
try:
|
||||
links.append(os.readlink(item))
|
||||
except FileNotFoundError:
|
||||
pass # a live request can close a descriptor while sampled
|
||||
fields = (proc / "stat").read_text().rsplit(")", 1)[1].split()
|
||||
ticks = int(fields[11]) + int(fields[12])
|
||||
sample_time = time.monotonic()
|
||||
cpu = 0.0
|
||||
if previous and previous[0] == pid:
|
||||
cpu = 100 * (ticks - previous[1]) / ticks_per_second / (sample_time - previous[2])
|
||||
previous = pid, ticks, sample_time
|
||||
status = (proc / "status").read_text()
|
||||
memory = re.search(r"^VmRSS:\s+(\d+)", status, re.MULTILINE)
|
||||
threads = re.search(r"^Threads:\s+(\d+)", status, re.MULTILINE)
|
||||
try:
|
||||
with urllib.request.urlopen("http://127.0.0.1:8088/health", timeout=5) as response:
|
||||
health = response.status == 200 and bool(json.load(response).get("ok"))
|
||||
except Exception as exc:
|
||||
health = f"error: {exc}"
|
||||
sample = {"time": datetime.datetime.now().isoformat(timespec="seconds"),
|
||||
"elapsed": round(sample_time - started, 1), "pid": pid, "fd": len(links),
|
||||
"hub_db_fd": sum("/hub.db" in link for link in links),
|
||||
"cpu_percent": round(cpu, 2), "rss_kib": int(memory[1]) if memory else None,
|
||||
"threads": int(threads[1]) if threads else None, "health": health}
|
||||
samples.append(sample)
|
||||
line = json.dumps(sample, ensure_ascii=False)
|
||||
print(line, flush=True)
|
||||
report.write(line + "\n")
|
||||
report.flush()
|
||||
if time.monotonic() - started >= minutes * 60:
|
||||
break
|
||||
time.sleep(min(10, max(0, minutes * 60 - (time.monotonic() - started))))
|
||||
|
||||
journal = subprocess.check_output(["journalctl", "-u", "owrt-remote", "--since", since, "--no-pager"], text=True, errors="replace", timeout=30)
|
||||
pattern = re.compile(r"unable to open database|database is locked|sqlite|traceback|exception|monitor.*error|connection reset|remote end closed", re.IGNORECASE)
|
||||
errors = [line for line in journal.splitlines() if pattern.search(line)]
|
||||
window = max(1, min(10, len(samples) // 3))
|
||||
trend = {
|
||||
key: {"first_median": statistics.median(s[key] for s in samples[:window]),
|
||||
"last_median": statistics.median(s[key] for s in samples[-window:])}
|
||||
for key in ("fd", "hub_db_fd", "rss_kib", "threads")
|
||||
if all(s[key] is not None for s in samples)
|
||||
}
|
||||
# Concurrent requests can briefly raise FD counts. Compare windows and
|
||||
# flag sustained growth; the report still needs workload-aware review.
|
||||
fd_growth = trend["fd"]["last_median"] > trend["fd"]["first_median"] + 8
|
||||
db_growth = trend["hub_db_fd"]["last_median"] > trend["hub_db_fd"]["first_median"] + 2
|
||||
summary = {"type": "summary", "seconds": samples[-1]["elapsed"], "first": samples[0], "last": samples[-1],
|
||||
"fd_min": min(s["fd"] for s in samples), "fd_max": max(s["fd"] for s in samples),
|
||||
"hub_db_fd_max": max(s["hub_db_fd"] for s in samples),
|
||||
"pid_changed": len({s["pid"] for s in samples}) != 1,
|
||||
"health_failures": sum(s["health"] is not True for s in samples), "matching_log_lines": errors,
|
||||
"trend": trend, "sustained_fd_growth": fd_growth, "sustained_db_growth": db_growth}
|
||||
report.write(json.dumps(summary, ensure_ascii=False) + "\n")
|
||||
print(json.dumps(summary, ensure_ascii=False), flush=True)
|
||||
print(f"Report saved: {report_path}", flush=True)
|
||||
print("Check nginx and exercise the panel, a real router proxy, heartbeat and notifications alongside this report.", flush=True)
|
||||
if summary["pid_changed"] or summary["health_failures"] or errors or fd_growth or db_growth:
|
||||
raise RuntimeError("Monitoring found failures; inspect the saved report")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
minutes = int(sys.argv[1]) if len(sys.argv) > 1 else 10
|
||||
if not 1 <= minutes <= 1440:
|
||||
raise ValueError("minutes must be between 1 and 1440")
|
||||
main(minutes)
|
||||
@@ -0,0 +1,481 @@
|
||||
"""Resource regression tests; only temporary state and loopback servers are used.
|
||||
|
||||
Run: python -m unittest discover -s tests -v
|
||||
Soak: python tests/test_resource_lifecycle.py --soak-seconds 600
|
||||
Linux measures /proc/self/fd (including hub.db); Windows measures process handles.
|
||||
Connections are deliberately retained so GC cannot conceal missing close().
|
||||
"""
|
||||
import argparse
|
||||
import ast
|
||||
import contextlib
|
||||
import ctypes
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from unittest import mock
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
|
||||
SOURCE = Path(os.environ.get("OWRT_RESOURCE_HUB_SOURCE", Path(__file__).resolve().parents[1] / "vps" / "owrt-remote-hub.py"))
|
||||
STATE = tempfile.TemporaryDirectory(prefix="owrt-resource-tests-")
|
||||
spec = importlib.util.spec_from_file_location("resource_test_hub", SOURCE)
|
||||
hub = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = hub
|
||||
with mock.patch.dict(os.environ, {"OWRT_REMOTE_STATE_DIR": STATE.name, "OWRT_REMOTE_ROUTER_NOTIFY_POLL": "3"}):
|
||||
spec.loader.exec_module(hub)
|
||||
REAL_CONNECT = sqlite3.connect
|
||||
|
||||
|
||||
class TrackedConnection(sqlite3.Connection):
|
||||
def close(self):
|
||||
super().close()
|
||||
self.closed = True
|
||||
|
||||
|
||||
def track_connections(retained):
|
||||
def factory(*args, **kwargs):
|
||||
conn = REAL_CONNECT(*args, factory=TrackedConnection, **kwargs)
|
||||
conn.closed = False
|
||||
retained.append(conn)
|
||||
return conn
|
||||
return mock.patch.object(hub.sqlite3, "connect", side_effect=factory)
|
||||
|
||||
|
||||
def resource_counts():
|
||||
if sys.platform.startswith("linux"):
|
||||
links = []
|
||||
for path in Path("/proc/self/fd").iterdir():
|
||||
try:
|
||||
links.append(os.readlink(path))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return {"fd": len(links), "hub_db_fd": sum("hub.db" in item for item in links)}
|
||||
if sys.platform == "win32":
|
||||
kernel = ctypes.WinDLL("kernel32", use_last_error=True)
|
||||
kernel.GetCurrentProcess.restype = ctypes.c_void_p
|
||||
kernel.GetProcessHandleCount.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong)]
|
||||
count = ctypes.c_ulong()
|
||||
if not kernel.GetProcessHandleCount(kernel.GetCurrentProcess(), ctypes.byref(count)):
|
||||
raise ctypes.WinError(ctypes.get_last_error())
|
||||
return {"handles": count.value}
|
||||
return {}
|
||||
|
||||
|
||||
class QuietHandler(hub.Handler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
class RouterBackend(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
body = b"router proxy fixture"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
|
||||
class HubFixture:
|
||||
def __enter__(self):
|
||||
self.tmp = tempfile.TemporaryDirectory(prefix="owrt-db-", dir=STATE.name)
|
||||
self.app = hub.App(Path(self.tmp.name) / "hub.db", "test-session", "test-agent", "")
|
||||
hub.write_json_private(hub.AUTH_FILE, {"username": "test-owner", "password_hash": "unused"})
|
||||
self.backend = ThreadingHTTPServer(("127.0.0.1", 0), RouterBackend)
|
||||
with self.app.conn() as conn:
|
||||
hub.upsert_router(conn, {
|
||||
"id": "test-router", "name": "Test Router", "entry_port": self.backend.server_port,
|
||||
"vps_host": "127.0.0.1", "public_url": "http://127.0.0.1",
|
||||
})
|
||||
self.server = hub.HubHTTPServer(("127.0.0.1", 0), QuietHandler)
|
||||
self.server.app = self.app
|
||||
self.server.is_tls = False
|
||||
self.threads = []
|
||||
self.push_patch = mock.patch.object(hub, "queue_web_push_payload")
|
||||
self.push_patch.start()
|
||||
for server in (self.backend, self.server):
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
self.threads.append(thread)
|
||||
self.app.start_router_state_monitor()
|
||||
return self
|
||||
|
||||
def request(self, path, payload=None):
|
||||
headers = {"Cookie": "owrt_remote_session=test-session", "Authorization": "Bearer test-agent"}
|
||||
body = None
|
||||
if payload is not None:
|
||||
headers["Content-Type"] = "application/json"
|
||||
body = json.dumps(payload).encode()
|
||||
request = urllib.request.Request(f"http://127.0.0.1:{self.server.server_port}{path}", data=body, headers=headers)
|
||||
with urllib.request.urlopen(request, timeout=5) as response:
|
||||
return response.read()
|
||||
|
||||
def exercise(self):
|
||||
assert json.loads(self.request("/health"))["ok"]
|
||||
assert json.loads(self.request("/api/heartbeat", {"id": "test-router", "hostname": "test-router"}))["ok"]
|
||||
assert len(json.loads(self.request("/api/routers"))["routers"]) == 1
|
||||
assert "groups" in json.loads(self.request("/api/router-groups"))
|
||||
assert "notifications" in json.loads(self.request("/api/notifications"))
|
||||
assert self.request("/access/test-router/") == b"router proxy fixture"
|
||||
assert b"v109" in self.request("/")
|
||||
self.app.snapshot_router_states()
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.app.stop_router_state_monitor()
|
||||
for server in (self.server, self.backend):
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
for thread in self.threads:
|
||||
thread.join(timeout=5)
|
||||
self.push_patch.stop()
|
||||
self.tmp.cleanup()
|
||||
|
||||
|
||||
class SQLiteLifecycleTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory(dir=STATE.name)
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.db = Path(self.tmp.name) / "hub.db"
|
||||
self.app = hub.App(self.db, "test-session", "test-agent", "")
|
||||
self.retained = []
|
||||
self.patch = track_connections(self.retained)
|
||||
self.patch.start()
|
||||
self.addCleanup(self.patch.stop)
|
||||
|
||||
def assert_closed(self):
|
||||
self.assertTrue(self.retained)
|
||||
for conn in self.retained:
|
||||
self.assertTrue(conn.closed)
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
conn.execute("select 1")
|
||||
|
||||
def test_success_return_and_commit(self):
|
||||
def operation():
|
||||
with self.app.conn() as conn:
|
||||
conn.execute("create table sample(value integer)")
|
||||
conn.execute("insert into sample values (7)")
|
||||
return 7
|
||||
self.assertEqual(operation(), 7)
|
||||
with self.app.conn() as conn:
|
||||
self.assertEqual(conn.execute("select value from sample").fetchone()[0], 7)
|
||||
self.assert_closed()
|
||||
|
||||
def test_exception_and_base_exception_rollback(self):
|
||||
with self.app.conn() as conn:
|
||||
conn.execute("create table sample(value integer)")
|
||||
for error in (ValueError("body failed"), KeyboardInterrupt()):
|
||||
with self.assertRaises(type(error)):
|
||||
with self.app.conn() as conn:
|
||||
conn.execute("insert into sample values (9)")
|
||||
raise error
|
||||
with self.app.conn() as conn:
|
||||
self.assertEqual(conn.execute("select count(*) from sample").fetchone()[0], 0)
|
||||
self.assert_closed()
|
||||
|
||||
def test_init_failure_closes_and_propagates(self):
|
||||
with mock.patch.object(hub, "init_db", side_effect=sqlite3.OperationalError("init failed")):
|
||||
with self.assertRaisesRegex(sqlite3.OperationalError, "init failed"):
|
||||
with self.app.conn():
|
||||
self.fail("must not enter body")
|
||||
self.assert_closed()
|
||||
|
||||
def test_commit_failure_rolls_back_and_closes(self):
|
||||
with hub.connect(self.db) as conn:
|
||||
conn.executescript("pragma foreign_keys=on; create table parent(id primary key); "
|
||||
"create table child(id references parent(id) deferrable initially deferred);")
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
with hub.connect(self.db) as conn:
|
||||
conn.execute("pragma foreign_keys=on")
|
||||
conn.execute("insert into child values (1)")
|
||||
with hub.connect(self.db) as conn:
|
||||
self.assertEqual(conn.execute("select count(*) from child").fetchone()[0], 0)
|
||||
self.assert_closed()
|
||||
|
||||
def test_setup_and_transaction_exit_failure_close(self):
|
||||
for setup_error in (True, False):
|
||||
connection = mock.MagicMock()
|
||||
if setup_error:
|
||||
type(connection).row_factory = mock.PropertyMock(side_effect=RuntimeError("setup failed"))
|
||||
else:
|
||||
connection.__exit__.side_effect = sqlite3.OperationalError("rollback failed")
|
||||
with mock.patch.object(hub.sqlite3, "connect", return_value=connection):
|
||||
with self.assertRaises((RuntimeError, sqlite3.OperationalError)):
|
||||
with hub.connect(self.db):
|
||||
pass
|
||||
connection.close.assert_called_once()
|
||||
|
||||
def test_backup_destination_open_failure_closes_source(self):
|
||||
with self.app.conn():
|
||||
pass
|
||||
with self.assertRaises(sqlite3.OperationalError):
|
||||
hub.copy_sqlite_backup(self.db, self.tmp.name) # destination is a directory
|
||||
self.assert_closed()
|
||||
|
||||
def test_backup_success(self):
|
||||
with self.app.conn() as conn:
|
||||
hub.upsert_router(conn, {"id": "backup-router", "name": "backup", "entry_port": 0, "vps_host": "localhost"})
|
||||
copy = Path(self.tmp.name) / "copy.db"
|
||||
hub.copy_sqlite_backup(self.db, copy)
|
||||
with hub.connect(copy) as conn:
|
||||
self.assertEqual(hub.list_router_rows(conn)[0]["id"], "backup-router")
|
||||
self.assert_closed()
|
||||
|
||||
def test_snapshot_fd_regression_without_gc(self):
|
||||
with self.app.conn():
|
||||
pass
|
||||
before = resource_counts()
|
||||
for _ in range(500):
|
||||
self.assertEqual(self.app.snapshot_router_states(), {})
|
||||
after = resource_counts()
|
||||
self.assert_closed()
|
||||
self.assertGreaterEqual(len(self.retained), 501)
|
||||
for key, value in before.items():
|
||||
self.assertLessEqual(after[key], value + (2 if key != "hub_db_fd" else 0), (before, after))
|
||||
|
||||
def test_all_sqlite_creation_uses_managed_factory(self):
|
||||
tree = ast.parse(SOURCE.read_text(encoding="utf-8"))
|
||||
parents = {child: parent for parent in ast.walk(tree) for child in ast.iter_child_nodes(parent)}
|
||||
direct = []
|
||||
for node in ast.walk(tree):
|
||||
if not isinstance(node, ast.Call):
|
||||
continue
|
||||
if isinstance(node.func, ast.Attribute) and isinstance(node.func.value, ast.Name) and node.func.value.id == "sqlite3" and node.func.attr == "connect":
|
||||
direct.append(node)
|
||||
elif isinstance(node.func, ast.Name) and node.func.id == "connect" or isinstance(node.func, ast.Attribute) and node.func.attr == "conn":
|
||||
self.assertIsInstance(parents[node], ast.withitem, f"unmanaged DB operation at line {node.lineno}")
|
||||
self.assertEqual(len(direct), 1)
|
||||
|
||||
|
||||
class OtherResourceTests(unittest.TestCase):
|
||||
def handler(self, data=b"", headers=None):
|
||||
handler = object.__new__(hub.Handler)
|
||||
handler.rfile = io.BytesIO(data)
|
||||
handler.headers = headers or {}
|
||||
return handler
|
||||
|
||||
def test_chunked_valid_and_truncated_bodies(self):
|
||||
headers = {"Transfer-Encoding": "chunked"}
|
||||
self.assertEqual(self.handler(b"3\r\nabc\r\n0\r\n\r\n", headers).read_body(), b"abc")
|
||||
for data in (b"", b"3\r\nab", b"3\r\nabc", b"0\r\n", b"-1\r\n", b"\r\n"):
|
||||
with self.subTest(data=data), self.assertRaises(ValueError):
|
||||
self.handler(data, headers).read_body()
|
||||
with self.assertRaises(ValueError):
|
||||
self.handler(b"short", {"Content-Length": "100"}).read_body()
|
||||
|
||||
def test_oauth_http_error_closes_response(self):
|
||||
body = io.BytesIO(b'{"error":"denied"}')
|
||||
error = urllib.error.HTTPError("http://test", 403, "denied", {}, body)
|
||||
with mock.patch.object(hub.urllib.request, "urlopen", side_effect=error):
|
||||
with self.assertRaisesRegex(ValueError, "denied"):
|
||||
hub.oauth_fetch_json("http://test")
|
||||
self.assertTrue(body.closed)
|
||||
|
||||
def test_push_success_and_failure_close_response(self):
|
||||
response = mock.Mock(status_code=410, text="gone")
|
||||
error = RuntimeError("push failed")
|
||||
error.response = response
|
||||
with mock.patch.object(hub, "web_push_ready", return_value=True), mock.patch.object(hub, "webpush", return_value=response) as push:
|
||||
self.assertEqual(hub.send_web_push({"endpoint": "https://test"}, {}), "ok")
|
||||
response.close.assert_called_once()
|
||||
response.reset_mock()
|
||||
push.side_effect = error
|
||||
self.assertEqual(hub.send_web_push({"endpoint": "https://test"}, {}), "gone")
|
||||
response.close.assert_called_once()
|
||||
|
||||
def test_atomic_failure_removes_temp_and_preserves_target(self):
|
||||
with tempfile.TemporaryDirectory(dir=STATE.name) as name:
|
||||
target = Path(name) / "original.txt"
|
||||
target.write_text("original")
|
||||
with mock.patch.object(hub.os, "replace", side_effect=OSError("replace failed")):
|
||||
with self.assertRaises(OSError):
|
||||
hub.atomic_write_text(target, "new")
|
||||
self.assertEqual(target.read_text(), "original")
|
||||
self.assertEqual(list(Path(name).iterdir()), [target])
|
||||
|
||||
def test_askpass_setup_failure_removes_directory(self):
|
||||
with tempfile.TemporaryDirectory(dir=STATE.name) as name:
|
||||
directory = Path(name) / "askpass"
|
||||
directory.mkdir()
|
||||
with mock.patch.object(hub.tempfile, "mkdtemp", return_value=str(directory)), mock.patch.object(hub.os, "chmod", side_effect=OSError("chmod failed")):
|
||||
with self.assertRaises(OSError):
|
||||
self.handler().prepare_ssh_askpass({}, "fake-password")
|
||||
self.assertFalse(directory.exists())
|
||||
|
||||
def test_https_failure_releases_listening_port(self):
|
||||
server = hub.make_http_server(None, "127.0.0.1", 0)
|
||||
port = server.server_port
|
||||
server.server_close()
|
||||
with self.assertRaises(OSError):
|
||||
hub.make_http_server(None, "127.0.0.1", port, "missing-cert.pem", "missing-key.pem")
|
||||
replacement = hub.make_http_server(None, "127.0.0.1", port)
|
||||
replacement.server_close()
|
||||
|
||||
def test_terminal_reader_start_failure_cleans_process_and_map(self):
|
||||
session = {"id": "failed-thread", "pid": 123, "fd": 456}
|
||||
with mock.patch.object(hub.threading.Thread, "start", side_effect=RuntimeError("thread failed")), mock.patch.object(hub, "close_terminal_process") as close:
|
||||
with self.assertRaises(RuntimeError):
|
||||
self.handler().launch_ssh_http_reader(session)
|
||||
close.assert_called_once_with(123, 456)
|
||||
self.assertNotIn("failed-thread", hub.SSH_HTTP_SESSIONS)
|
||||
|
||||
def test_initial_websocket_failure_cleans_terminal(self):
|
||||
handler = self.handler()
|
||||
handler.connection = object()
|
||||
with mock.patch.object(handler, "spawn_terminal_pty", return_value=(123, 456)), mock.patch.object(hub, "ws_send_frame", side_effect=BrokenPipeError()), mock.patch.object(hub, "close_terminal_process") as close:
|
||||
handler.run_terminal_session({}, ["sh"], "unavailable", "open", "sh", "terminal error")
|
||||
close.assert_called_once_with(123, 456)
|
||||
|
||||
def test_terminal_input_failure_closes_owned_descriptor(self):
|
||||
handler = self.handler()
|
||||
handler.send_json = mock.Mock()
|
||||
session = {"alive": True, "fd": 456, "last_seen": 0, "lock": threading.Lock()}
|
||||
with mock.patch.dict(hub.SSH_HTTP_SESSIONS, {"input-test": session}, clear=True), mock.patch.object(hub.os, "dup", return_value=789), mock.patch.object(hub.os, "close") as close, mock.patch.object(hub, "write_pty_all", side_effect=TimeoutError("write failed")):
|
||||
handler.ssh_http_write("input-test", {"data": "hello"})
|
||||
close.assert_called_once_with(789)
|
||||
self.assertEqual(handler.send_json.call_args.args[0], 500)
|
||||
|
||||
@unittest.skipUnless(hasattr(hub.signal, "SIGHUP"), "POSIX terminal signals")
|
||||
def test_terminal_shutdown_escalates_and_reaps(self):
|
||||
# One unsuccessful poll per signal, followed by the final blocking reap.
|
||||
with mock.patch.object(hub.os, "close") as close, mock.patch.object(hub.os, "waitpid", create=True, side_effect=[(0, 0)] * 3 + [(123, 0)]) as wait, mock.patch.object(hub.os, "kill") as kill, mock.patch.object(hub.time, "monotonic", side_effect=[0, 2, 0, 2, 0, 2]):
|
||||
hub.close_terminal_process(123, 456)
|
||||
close.assert_called_once_with(456)
|
||||
self.assertEqual([c.args[1] for c in kill.call_args_list], [hub.signal.SIGHUP, hub.signal.SIGTERM, hub.signal.SIGKILL])
|
||||
self.assertEqual(wait.call_args.args, (123, 0))
|
||||
|
||||
def test_terminal_session_expiry_and_retained_close_notice(self):
|
||||
now = hub.now_ts()
|
||||
sessions = {
|
||||
"expired": {"alive": True, "last_seen": now - 901, "lock": threading.Lock()},
|
||||
"closed": {"alive": False, "last_seen": now, "closed_at": now - 61, "lock": threading.Lock()},
|
||||
"recent": {"alive": False, "last_seen": now, "closed_at": now, "lock": threading.Lock()},
|
||||
"active": {"alive": True, "last_seen": now, "lock": threading.Lock()},
|
||||
}
|
||||
with mock.patch.dict(hub.SSH_HTTP_SESSIONS, sessions, clear=True):
|
||||
hub.expire_ssh_http_sessions()
|
||||
self.assertEqual(set(hub.SSH_HTTP_SESSIONS), {"recent", "active"})
|
||||
|
||||
def test_monitor_recovers_from_initial_database_error(self):
|
||||
app = hub.App("unused", "session", "agent", "")
|
||||
with mock.patch.object(app, "prime_router_state_snapshot", side_effect=sqlite3.OperationalError("initial DB error")), mock.patch.object(app, "check_router_state_changes") as check, mock.patch.object(app.router_monitor_stop, "wait", side_effect=[False, True]), contextlib.redirect_stderr(io.StringIO()) as errors:
|
||||
app.router_state_monitor_loop()
|
||||
check.assert_called_once()
|
||||
self.assertIn("initial DB error", errors.getvalue())
|
||||
|
||||
def test_monitor_start_failure_does_not_break_cleanup(self):
|
||||
app = hub.App("unused", "session", "agent", "")
|
||||
with mock.patch.object(hub.threading.Thread, "start", side_effect=RuntimeError("start failed")):
|
||||
with self.assertRaises(RuntimeError):
|
||||
app.start_router_state_monitor()
|
||||
app.stop_router_state_monitor()
|
||||
|
||||
def test_serve_start_failure_closes_main_server(self):
|
||||
app = hub.App(Path(STATE.name) / "serve-failure.db", "session", "agent", "")
|
||||
server = mock.Mock()
|
||||
args = argparse.Namespace(db=app.db_path, public_url="", host="127.0.0.1", port=0)
|
||||
with mock.patch.object(hub, "App", return_value=app), mock.patch.object(hub, "session_token", return_value="session"), mock.patch.object(hub, "agent_token", return_value="agent"), mock.patch.object(hub, "record_hub_start_event"), mock.patch.object(hub, "load_auth", return_value={}), mock.patch.object(hub, "make_http_server", return_value=server), mock.patch.object(app, "start_router_state_monitor", side_effect=RuntimeError("monitor failed")):
|
||||
with self.assertRaises(RuntimeError):
|
||||
hub.cmd_serve(args)
|
||||
server.server_close.assert_called_once()
|
||||
|
||||
def test_serve_stop_closes_main_and_extra_servers(self):
|
||||
app = hub.App(Path(STATE.name) / "serve-stop.db", "session", "agent", "")
|
||||
main = mock.Mock()
|
||||
extra = mock.Mock()
|
||||
main.serve_forever.side_effect = KeyboardInterrupt()
|
||||
args = argparse.Namespace(db=app.db_path, public_url="", host="127.0.0.1", port=8088,
|
||||
extra_ports="18088", tls_ports="", tls_cert="", tls_key="")
|
||||
with mock.patch.object(hub, "App", return_value=app), mock.patch.object(hub, "session_token", return_value="session"), mock.patch.object(hub, "agent_token", return_value="agent"), mock.patch.object(hub, "record_hub_start_event"), mock.patch.object(hub, "load_auth", return_value={}), mock.patch.object(hub, "make_http_server", side_effect=[main, extra]), contextlib.redirect_stdout(io.StringIO()):
|
||||
hub.cmd_serve(args)
|
||||
main.server_close.assert_called_once()
|
||||
extra.shutdown.assert_called_once()
|
||||
extra.server_close.assert_called_once()
|
||||
self.assertFalse(app.router_monitor_thread.is_alive())
|
||||
|
||||
def test_incoming_socket_has_finite_io_timeout(self):
|
||||
server = hub.make_http_server(None, "127.0.0.1", 0)
|
||||
self.addCleanup(server.server_close)
|
||||
with socket.create_connection(server.server_address) as client:
|
||||
incoming, _ = server.get_request()
|
||||
try:
|
||||
self.assertEqual(incoming.gettimeout(), 90)
|
||||
finally:
|
||||
incoming.close()
|
||||
|
||||
@unittest.skipUnless(sys.platform.startswith("linux"), "real PTY/child reaping needs Linux")
|
||||
def test_real_terminal_child_is_reaped_and_fd_closed(self):
|
||||
handler = self.handler()
|
||||
pid, fd = handler.spawn_terminal_pty(dict(os.environ), ["/bin/sh", "-c", "sleep 60"], "pty", "open", "sh")
|
||||
hub.close_terminal_process(pid, fd)
|
||||
with self.assertRaises(OSError):
|
||||
os.fstat(fd)
|
||||
with self.assertRaises(ChildProcessError):
|
||||
os.waitpid(pid, os.WNOHANG)
|
||||
|
||||
|
||||
class IntegrationTests(unittest.TestCase):
|
||||
def test_panel_api_proxy_heartbeat_and_monitor(self):
|
||||
with HubFixture() as fixture:
|
||||
for _ in range(5):
|
||||
fixture.exercise()
|
||||
self.assertTrue(fixture.app.router_monitor_thread.is_alive())
|
||||
# Verify the monitor still emits the online -> offline transition.
|
||||
fixture.app.prime_router_state_snapshot()
|
||||
with fixture.app.conn() as conn:
|
||||
conn.execute("update routers set last_seen = ?", (hub.now_ts() - 3600,))
|
||||
with mock.patch.object(hub, "notify_router_offline") as notify:
|
||||
fixture.app.check_router_state_changes()
|
||||
notify.assert_called_once()
|
||||
self.assertFalse(fixture.app.router_monitor_thread.is_alive())
|
||||
|
||||
|
||||
def soak(seconds):
|
||||
started = time.monotonic()
|
||||
cpu_start = time.process_time()
|
||||
samples = []
|
||||
operations = 0
|
||||
with HubFixture() as fixture:
|
||||
fixture.exercise() # warm up auth, sessions, notifications and HTTP workers
|
||||
while True:
|
||||
fixture.exercise()
|
||||
operations += 1
|
||||
# Measure between requests, after worker sockets have closed.
|
||||
time.sleep(0.05)
|
||||
counts = resource_counts()
|
||||
sample = {"elapsed": round(time.monotonic() - started, 1), **counts}
|
||||
samples.append(sample)
|
||||
print(json.dumps(sample), flush=True)
|
||||
if time.monotonic() - started >= seconds:
|
||||
break
|
||||
time.sleep(min(5, max(0, seconds - (time.monotonic() - started))))
|
||||
assert fixture.app.router_monitor_thread.is_alive(), "router monitor stopped"
|
||||
for key in samples[0]:
|
||||
if key == "elapsed":
|
||||
continue
|
||||
assert samples[-1][key] <= samples[0][key] + (4 if key != "hub_db_fd" else 0), samples
|
||||
result = {"seconds": round(time.monotonic() - started, 1), "cpu_seconds": round(time.process_time() - cpu_start, 2),
|
||||
"cycles": operations, "first": samples[0], "last": samples[-1], "peak": {key: max(s[key] for s in samples) for key in counts}}
|
||||
print("SOAK_OK " + json.dumps(result), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if "--soak-seconds" in sys.argv:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--soak-seconds", type=int, required=True)
|
||||
soak(parser.parse_args().soak_seconds)
|
||||
else:
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user