Files
Aemeath ac90575b0b
Build packages / CI (push) Blocked by required conditions
Build packages / Version (push) Waiting to run
CI / Web (format, types, tests, bundles) (push) Blocked by required conditions
CI / ShellCheck (push) Blocked by required conditions
CI / Backend (Go: vet, tests, a real ubusd and uhttpd) (push) Blocked by required conditions
CI / Package files (push) Blocked by required conditions
Dawnline 0.20.0
- **Backend rewritten in Go.** One daemon, `dawnlined`, replaces the whole ucode backend: the rpcd plugin, the agent, the CLI and the updater. The package no longer depends on ucode.
- **Per-architecture builds.** The release carries the backend for arm64, ARMv7, mips, mipsle and amd64; the router fetches its own after the package is installed and checks its sha256. Dawnline does not run on other architectures (older ARM, 32-bit x86, RISC-V).
- **Faster on the router.** A firewall sync takes 0.06 s instead of 0.9 s; a full apply takes about 300 ms. The portal is served by the daemon itself, without a CGI process per request.
- **Live pages.** The admin and the portal follow a live stream from the router instead of re-fetching the whole state every 30 seconds.
- **Admin redesigned in the portal's style.** Three sections (Overview · Log · Settings); a server is a page of its own with People, Places and Settings; a device opens as a dialog over its person.
- **Changes wait for «Apply».** Every setting, and a person's access, is staged in one dock before it reaches the router.
- **Configuration rollout.** After a change that makes devices fetch their configuration again, the admin shows how many have fetched it and who has not. Every setting shows beforehand (⟳N) how many devices it will touch.
- **Waiting for a decision.** Overview and People put first the people who need one (access ended, limit reached, a device that does not connect), each with its one action a tap away.
- **Bulk actions on people.** «Select» (or a long press on a card) picks several people to extend by a month, pause or resume at once.
- **Undo.** An action done from a card or a request is undone from its toast.
- **Update dialogs redesigned.** An update is one view from its question to its result; Dawnline's own update runs in it like every other, and the page reloads itself once the new version answers. Other versions are a list, each with its notes and Install on a page of its own.
- **Site to site: networks behind a router.** A router connected as a device can have networks behind it (up to eight, /16 to /30): the server routes them and applies the router's access to them. A network that shares addresses with a tunnel, the router's own networks or another site is refused by name.
- **Routers from the portal.** People can add a router and name its networks on their own page; the admin accepts or declines with a reason, or lets them through without asking.
- **Check access and Health know the networks behind routers.**
- **Usage limits reworked.** A monthly limit no longer follows the calendar. A person's month starts with their first traffic and ends a month later; the next one starts only when they connect again, so time away uses up none of it. Before, everyone's limit reset on the 1st.
- Extra data lasts until that person's month ends; «Reset usage» starts the month anew.
- A limit is for a month or for the whole term; the admin and the portal show when it resets.
- Charts that said «this month» show the last 30 days.
- After the update nobody loses what was counted: the month that was running becomes the person's first and ends on the 1st.
- **Access by time of day removed.** The Hours limit is gone from the admin, the portal and the backend; saved hours are dropped.
- **Expiry simplified.** An end date always stops access, with a heads-up three days before; the per-person choice is gone.
- **No false disconnects from sleeping phones.** A phone or a tablet stays online through ten minutes of silence.
- **Mobile Safari fixes.** Pressing a control no longer sends the page to its top; tab tracks that do not fit scroll; switches animate after a press.
2026-11-13 09:43:50 +03:00

363 lines
12 KiB
Go

// Package watchdog checks, once a minute, what keeps people connected and repairs what it can, waiting
// longer after each repair that did not help (dawnline.watchdog). What it finds goes to the service log
// (English, as logged); its state to watchdog.json for the Health page.
package watchdog
import (
"fmt"
"os"
"strings"
"github.com/tickcount/luci-app-dawnline/backend/files"
"github.com/tickcount/luci-app-dawnline/backend/model"
"github.com/tickcount/luci-app-dawnline/backend/servicelog"
)
// wait is the seconds before the next repair, by the number of repairs so far.
var wait = []int64{0, 120, 600, 1800, 3600}
// settled is the good minutes in a row before a repaired target's count of repairs is forgotten: one good
// minute between two failures is still flapping.
const settled = 10
// Repair is how a target is put right: what is said of it ("restarting it"), how many times at most
// (0: no limit), and the doing (n: which try this is).
type Repair struct {
What string
Max int
Run func(n int)
}
// Probe is one thing looked at. ID is the Health check's id; Name the chip (stored in English); Who the
// log's subject. The log reads "<who> <problem> - <what>".
type Probe struct {
ID, Src, Name, Who string
OK bool
Problem, Back string
Detail func() string
Repair *Repair
}
// Target is what is kept of a target between passes.
type Target struct {
Bad int `json:"bad"`
Tries int `json:"tries"`
Next int64 `json:"next"`
Name string `json:"name"`
OK bool `json:"ok"`
Reported bool `json:"reported"`
Stuck bool `json:"stuck"`
Good int `json:"good"`
Last int64 `json:"last,omitempty"`
}
// Fixed is a repair that was made.
type Fixed struct {
TS int64 `json:"ts"`
ID string `json:"id"`
Name string `json:"name"`
}
// State is watchdog.json.
type State struct {
Last int64 `json:"last"`
Targets map[string]*Target `json:"targets"`
Repairs []Fixed `json:"repairs"`
}
// Pass is one pass over the probes. With fix off it only reports. busy: an update is replacing packages
// or restarting tunnels; what looks broken meanwhile is being worked on, and repairs would only get in
// its way.
func Pass(path string, probes []Probe, fix, busy bool, now int64, log *servicelog.Log) {
say := func(src, level, msg string, detail func() string, who string) {
if log == nil {
return
}
d := ""
if detail != nil {
d = detail()
}
log.Write(src, level, msg, d, who, servicelog.Extra{})
}
var st State
files.ReadJSON(path, &st)
if st.Targets == nil {
st.Targets = map[string]*Target{}
}
repairs := []Fixed{}
for _, r := range st.Repairs {
if now-r.TS < 86400 {
repairs = append(repairs, r)
}
}
seen := map[string]bool{}
for _, p := range probes {
seen[p.ID] = true
x := st.Targets[p.ID]
if x == nil {
x = &Target{}
st.Targets[p.ID] = x
}
x.Name, x.OK = p.Name, p.OK
if p.OK {
if x.Reported {
msg := p.Back
switch {
case x.Stuck:
msg = p.Back + " · restored"
case x.Tries > 1:
msg = fmt.Sprintf("%s (after %d repairs)", p.Back, x.Tries)
case x.Tries == 1:
msg = fmt.Sprintf("%s (after %d repair)", p.Back, x.Tries)
}
say(p.Src, "up", msg, nil, p.Who)
}
x.Bad, x.Next, x.Reported, x.Stuck = 0, 0, false, false
// The repairs so far are forgotten only once it has stayed up a while.
if x.Good++; x.Good >= settled {
x.Tries, x.Good = 0, 0
}
continue
}
x.Good = 0
// one bad minute can be a restart in progress
if x.Bad++; x.Bad < 2 {
continue
}
if busy {
continue
}
max := 0
if p.Repair != nil {
max = p.Repair.Max
}
if max > 0 && x.Tries >= max && !x.Reported {
say(p.Src, "error", fmt.Sprintf("%s - not repaired again (%d repairs didn't help)", p.Problem, max), p.Detail, p.Who)
x.Reported = true
continue
}
if !fix || p.Repair == nil || (max > 0 && x.Tries >= max) {
if !x.Reported {
say(p.Src, "error", p.Problem, p.Detail, p.Who)
}
x.Reported = true
continue
}
// Three repairs did not help: said once, and from then on it keeps trying quietly until it works
// again (that is logged).
if x.Tries >= 3 && !x.Stuck && now-x.Last >= 50 {
x.Stuck = true
say(p.Src, "error", fmt.Sprintf("%s - %d repairs didn't help", p.Problem, x.Tries), p.Detail, p.Who)
}
if now < x.Next {
continue
}
x.Tries++
x.Last = now
w := x.Tries
if w > len(wait)-1 {
w = len(wait) - 1
}
x.Next = now + wait[w]
x.Reported = true
if !x.Stuck {
say(p.Src, "repair", p.Problem+" - "+p.Repair.What, p.Detail, p.Who)
}
func() {
defer func() {
if e := recover(); e != nil && !x.Stuck {
say(p.Src, "error", fmt.Sprintf("couldn't be repaired: %v", e), nil, p.Who)
}
}()
p.Repair.Run(x.Tries)
}()
if !x.Stuck {
repairs = append(repairs, Fixed{TS: now, ID: p.ID, Name: p.Name})
}
}
for id := range st.Targets {
if !seen[id] {
delete(st.Targets, id)
}
}
st.Last, st.Repairs = now, repairs
if i := strings.LastIndex(path, "/"); i > 0 {
os.MkdirAll(path[:i], 0o755)
}
files.WriteJSON(path, st, 0o644)
}
// Iface is netifd's word on an interface.
type Iface struct {
Exists, Up bool
}
// Facts is what the probes are made from: the router as it is now, and what repairs it.
type Facts struct {
// AmneziaWG: its tool and protocol script, its module (installed, loaded), netifd's handler for it,
// the packages that are missing.
Tools, Kmod, Loaded, Handler bool
Missing []string
// Owner is whose an interface of the network configuration is: has false when there is none, id ""
// when it is not Dawnline's.
Owner func(ifname string) (id string, has bool)
Iface func(ifname string) Iface
// IfaceErrors is why a tunnel is down, from netifd and the last related syslog lines.
IfaceErrors func(ifname string) string
UDPListening func(port int) bool
// Drift is what of each running server's interface differs from what Dawnline writes, by option name.
Drift map[string][]string
// NftPresent: the access table is loaded. Firewall: "" fine, "off", "changed" or "not loaded".
NftPresent bool
Firewall string
// The portal: whether it can run at all here (HTTPS is there, nobody else holds its port), and listens.
PortalCan, PortalListening bool
// The Home-only resolver: its configuration is there, and it runs.
ResolverConf, ResolverRunning bool
// Said is the last lines of syslog about something ("uhttpd", "rpcd"), for a problem's detail.
Said func(what string) string
// Reapply is the full apply; Run a command; StartServices starts the service's missing instances.
Reapply func()
Run func(name string, args ...string)
StartServices func()
// BlockedNote and BlockedFix word why a server is kept down.
BlockedNote, BlockedFix func(why map[string]any) string
}
// Probes is what to look at for these settings on this router.
func Probes(m *model.Model, f Facts) []Probe {
var out []Probe
var on []*model.Server
keyed := false
for _, id := range m.ServerIDs() {
srv := m.Servers[id]
keyed = keyed || srv.PrivateKey != ""
if srv.Enabled && srv.PrivateKey != "" {
on = append(on, srv)
}
}
// The tunnels check themselves: AmneziaWG first (without it no tunnel can run), then each server's interface.
awgOK := false
tunnels := func() {
if len(on) == 0 {
return
}
base := Probe{ID: "awg", Src: "awg", Name: "AmneziaWG", Who: "AmneziaWG", Back: "is ready again"}
if !f.Tools || !f.Kmod {
p := base
p.Problem = "isn't installed: no tunnel can run - install it in Dawnline → Components"
if f.Tools {
p.Problem = "kernel module isn't installed: no tunnel can run - install it in Dawnline → Components"
}
p.Detail = func() string { return "Missing: " + strings.Join(f.Missing, ", ") + "." }
out = append(out, p)
return
}
p := base
p.OK = f.Loaded && f.Handler
awgOK = p.OK
if !f.Loaded {
p.Problem = "kernel module isn't loaded"
p.Detail = func() string { return f.Said("modprobe") }
p.Repair = &Repair{What: "loading it", Run: func(int) { f.Run("modprobe", "amneziawg") }}
} else {
p.Problem = "isn't known to the network service - restart the network or reboot"
p.Detail = func() string {
return "netifd reads its protocol handlers when it starts; it was running before AmneziaWG was installed."
}
}
out = append(out, p)
// without the protocol handler ifup cannot work: the tunnels are not probed (or restarted) one by one
if f.Loaded && !f.Handler {
return
}
for _, srv := range on {
srv := srv
t := Probe{ID: "up_" + srv.ID, Src: "awg", Name: srv.Name + " · AmneziaWG", Who: srv.Name}
// A tunnel Dawnline may not set up is reported, never restarted: an interface of the same name
// that isn't Dawnline's may be your LAN.
why, _ := srv.Blocked.(map[string]any)
if why == nil {
if !model.ValidIfname(srv.Ifname) {
why = map[string]any{"code": "ifname", "ifname": srv.Ifname}
} else if owner, has := f.Owner(srv.Ifname); has && owner != srv.ID {
why = map[string]any{"code": "taken", "ifname": srv.Ifname}
}
}
if why != nil {
t.Problem, t.Back = f.BlockedNote(why), "AmneziaWG is running again"
t.Detail = func() string { return f.BlockedFix(why) }
out = append(out, t)
continue
}
s := f.Iface(srv.Ifname)
t.OK = s.Up && f.UDPListening(srv.Port)
switch {
case !s.Exists:
t.Problem = "AmneziaWG is missing from the router"
case !s.Up:
t.Problem = "AmneziaWG isn't running"
default:
t.Problem = fmt.Sprintf("AmneziaWG isn't accepting connections on port %d", srv.Port)
}
t.Back = fmt.Sprintf("AmneziaWG is running again · accepting connections on port %d", srv.Port)
t.Detail = func() string { return f.IfaceErrors(srv.Ifname) }
if s.Exists {
t.Repair = &Repair{What: "restarting it", Run: func(int) { f.Run("ifup", srv.Ifname) }}
} else {
t.Repair = &Repair{What: "setting it up again", Run: func(int) { f.Reapply() }}
}
out = append(out, t)
}
}
tunnels()
// A tunnel's interface edited outside Dawnline keeps that until the next apply: set up again with
// Dawnline's settings. Only while AmneziaWG itself works (its target comes first, and is fixed first).
if awgOK {
for _, id := range m.ServerIDs() {
diff, has := f.Drift[id]
if !has {
continue
}
srv := m.Servers[id]
out = append(out, Probe{ID: "cfg_" + id, Src: "awg", Name: srv.Name + " · interface settings", Who: srv.Name, OK: len(diff) == 0,
Problem: fmt.Sprintf("interface %s was changed outside Dawnline", srv.Ifname),
Back: fmt.Sprintf("interface %s has Dawnline's settings again", srv.Ifname),
// option names only: their values hold the server's private key
Detail: func() string { return "Changed: " + strings.Join(diff, ", ") + "." },
Repair: &Repair{What: "setting it up again", Run: func(int) { f.Reapply() }}})
}
}
if len(m.Servers) > 0 {
out = append(out, Probe{ID: "nft", Src: "dawnline", Name: "Access rules", Who: "Access rules", OK: f.NftPresent,
Problem: "are missing", Back: "are loaded again", Repair: &Repair{What: "loading them again", Run: func(int) { f.Reapply() }}})
}
if keyed {
p := Probe{ID: "fw", Src: "dawnline", Name: "Firewall", Who: "Firewall", OK: f.Firewall == "", Back: "rules for Dawnline are in place again"}
switch f.Firewall {
case "off":
// fw4 stopped on purpose is the admin's call: report it, do not start it
p.Problem = "isn't running on this router (fw4): tunnels and the portal are closed"
case "changed":
p.Problem = "rules for Dawnline were changed or removed"
default:
p.Problem = "rules for Dawnline aren't loaded"
}
if f.Firewall != "off" {
p.Repair = &Repair{What: "restoring them", Run: func(int) { f.Reapply(); f.Run("fw4", "-q", "reload") }}
}
out = append(out, p)
}
if s := m.Settings; s.PortalEnabled && f.PortalCan {
out = append(out, Probe{ID: "portal", Src: "portal", Name: "Portal", Who: "Personal portal", OK: f.PortalListening,
Problem: fmt.Sprintf("isn't answering on port %d", s.PortalPort), Back: "answers again",
Detail: func() string { return f.Said("portal") }, Repair: &Repair{What: "restarting it", Run: func(int) { f.StartServices() }}})
}
// only while a server is on: with all of them off the resolver idles on loopback
if f.ResolverConf && len(on) > 0 {
out = append(out, Probe{ID: "resolver", Src: "dawnline", Name: "Resolver", Who: "DNS for Home-only devices", OK: f.ResolverRunning,
Problem: "isn't running", Back: "runs again", Repair: &Repair{What: "restarting it", Run: func(int) { f.StartServices() }}})
}
return out
}