Dawnline 0.20.0
Build packages / CI (push) Blocked by required conditions
Build packages / Version (push) Waiting to run
CI / Web (format, types, tests, bundles) (push) Blocked by required conditions
CI / ShellCheck (push) Blocked by required conditions
CI / Backend (Go: vet, tests, a real ubusd and uhttpd) (push) Blocked by required conditions
CI / Package files (push) Blocked by required conditions

- **Backend rewritten in Go.** One daemon, `dawnlined`, replaces the whole ucode backend: the rpcd plugin, the agent, the CLI and the updater. The package no longer depends on ucode.
- **Per-architecture builds.** The release carries the backend for arm64, ARMv7, mips, mipsle and amd64; the router fetches its own after the package is installed and checks its sha256. Dawnline does not run on other architectures (older ARM, 32-bit x86, RISC-V).
- **Faster on the router.** A firewall sync takes 0.06 s instead of 0.9 s; a full apply takes about 300 ms. The portal is served by the daemon itself, without a CGI process per request.
- **Live pages.** The admin and the portal follow a live stream from the router instead of re-fetching the whole state every 30 seconds.
- **Admin redesigned in the portal's style.** Three sections (Overview · Log · Settings); a server is a page of its own with People, Places and Settings; a device opens as a dialog over its person.
- **Changes wait for «Apply».** Every setting, and a person's access, is staged in one dock before it reaches the router.
- **Configuration rollout.** After a change that makes devices fetch their configuration again, the admin shows how many have fetched it and who has not. Every setting shows beforehand (⟳N) how many devices it will touch.
- **Waiting for a decision.** Overview and People put first the people who need one (access ended, limit reached, a device that does not connect), each with its one action a tap away.
- **Bulk actions on people.** «Select» (or a long press on a card) picks several people to extend by a month, pause or resume at once.
- **Undo.** An action done from a card or a request is undone from its toast.
- **Update dialogs redesigned.** An update is one view from its question to its result; Dawnline's own update runs in it like every other, and the page reloads itself once the new version answers. Other versions are a list, each with its notes and Install on a page of its own.
- **Site to site: networks behind a router.** A router connected as a device can have networks behind it (up to eight, /16 to /30): the server routes them and applies the router's access to them. A network that shares addresses with a tunnel, the router's own networks or another site is refused by name.
- **Routers from the portal.** People can add a router and name its networks on their own page; the admin accepts or declines with a reason, or lets them through without asking.
- **Check access and Health know the networks behind routers.**
- **Usage limits reworked.** A monthly limit no longer follows the calendar. A person's month starts with their first traffic and ends a month later; the next one starts only when they connect again, so time away uses up none of it. Before, everyone's limit reset on the 1st.
- Extra data lasts until that person's month ends; «Reset usage» starts the month anew.
- A limit is for a month or for the whole term; the admin and the portal show when it resets.
- Charts that said «this month» show the last 30 days.
- After the update nobody loses what was counted: the month that was running becomes the person's first and ends on the 1st.
- **Access by time of day removed.** The Hours limit is gone from the admin, the portal and the backend; saved hours are dropped.
- **Expiry simplified.** An end date always stops access, with a heads-up three days before; the per-person choice is gone.
- **No false disconnects from sleeping phones.** A phone or a tablet stays online through ten minutes of silence.
- **Mobile Safari fixes.** Pressing a control no longer sends the page to its top; tab tracks that do not fit scroll; switches animate after a press.
This commit is contained in:
Aemeath committed 2026-11-13 09:43:50 +03:00
1 parent f8130e39a8
commit ac90575b0b
788 files changed
+139244 -41720

No files matched your search

+708
View File
@@ -0,0 +1,708 @@
// Package daemon is the running backend: it joins the parts and keeps them
// going until it is told to stop. cmd/dawnlined gives it the router's paths;
// a test gives it a stand's.
package daemon
import (
"context"
"fmt"
"net/http"
"os"
"os/exec"
"sync"
"syscall"
"time"
backend "github.com/tickcount/luci-app-dawnline/backend"
"github.com/tickcount/luci-app-dawnline/backend/api"
"github.com/tickcount/luci-app-dawnline/backend/apply"
"github.com/tickcount/luci-app-dawnline/backend/awg"
"github.com/tickcount/luci-app-dawnline/backend/bridge"
minute "github.com/tickcount/luci-app-dawnline/backend/chores"
"github.com/tickcount/luci-app-dawnline/backend/files"
"github.com/tickcount/luci-app-dawnline/backend/geo"
"github.com/tickcount/luci-app-dawnline/backend/github"
"github.com/tickcount/luci-app-dawnline/backend/hub"
"github.com/tickcount/luci-app-dawnline/backend/i18n"
"github.com/tickcount/luci-app-dawnline/backend/journal"
"github.com/tickcount/luci-app-dawnline/backend/methods"
"github.com/tickcount/luci-app-dawnline/backend/migrate"
"github.com/tickcount/luci-app-dawnline/backend/model"
"github.com/tickcount/luci-app-dawnline/backend/netview"
"github.com/tickcount/luci-app-dawnline/backend/portal"
"github.com/tickcount/luci-app-dawnline/backend/resolve"
"github.com/tickcount/luci-app-dawnline/backend/roster"
"github.com/tickcount/luci-app-dawnline/backend/servicelog"
"github.com/tickcount/luci-app-dawnline/backend/store"
"github.com/tickcount/luci-app-dawnline/backend/ubus"
"github.com/tickcount/luci-app-dawnline/backend/uci"
"github.com/tickcount/luci-app-dawnline/backend/usage"
"github.com/tickcount/luci-app-dawnline/backend/view"
"github.com/tickcount/luci-app-dawnline/backend/watch"
"github.com/tickcount/luci-app-dawnline/backend/watchdog"
"github.com/tickcount/luci-app-dawnline/backend/zone"
)
// Options is where the backend finds the router.
type Options struct {
// Socket is ubusd's.
Socket string
// RunDir holds runtime state (tmpfs), DataDir kept state (flash), Config the settings file.
RunDir, DataDir, Config string
// Stream is how pages follow: "sse" or "poll" (api.Info).
Stream string
// Watch: the backend looks at the tunnels and keeps the router to the settings (always, on a router; a
// test stand that only wants the bus and the stream leaves it off).
Watch bool
// Tunnels reads the tunnels the watcher looks at; nil: the router's own (awg.System). A test stand's are
// in its hands.
Tunnels *awg.Reader
// Portal: the backend serves the personal portal itself (package portal); nil: its own web server does.
Portal *Portal
// Log gets the lines for the system log.
Log func(format string, args ...any)
}
// Portal is what the portal's web server runs with (the init script works it out).
type Portal struct {
// Port is the HTTPS port, HTTP the plain one beside it (0: none); Cert and Key the certificate's files.
Port, HTTP int
Cert, Key string
// Procs is how many of the ucode portal run at once.
Procs int
}
// Run works until ctx ends, then waits for its parts.
func Run(ctx context.Context, o Options) {
// The router's zone, before anything counts a day in it (OpenWrt keeps it where Go does not look).
zone.Load(zone.File)
h := hub.New()
audience := api.NewAudience()
var b *bridge.Bridge
// The watcher, when this backend runs it: what the methods it answers itself read.
var w *watch.Watcher
var more []ubus.Method
nets := &homes{}
// the portal's own answers, where this backend keeps the router (set with the watcher)
var native http.Handler
// set when the watcher runs: the topics Go makes itself, told again as the router is at that moment
// (after a change of the settings)
var publish func()
if o.Watch {
tunnels := awg.System()
if o.Tunnels != nil {
tunnels = *o.Tunnels
}
w = &watch.Watcher{
Config: o.Config, RunDir: o.RunDir, DataDir: o.DataDir,
DHCP: "/etc/config/dhcp", Leases: "/tmp/dhcp.leases", Zone: zone.File,
Tunnels: tunnels, Hub: h, Audience: audience, Log: o.Log,
}
names := &resolve.Resolver{}
// The router is this backend's to keep: its own apply, sync, keeper, chores and watchdog. One file gives
// all of it back to the ucode backend, for as long as that one is still shipped: /etc/dawnline/backend-off
// (the init script then starts this process without its watcher, and the agent does the work).
mine := func() bool {
_, err := os.Stat(o.DataDir + "/backend-off")
return err != nil
}
logs := &servicelog.Log{RunDir: o.RunDir, Level: func() string {
if world := w.World(); world != nil {
return world.M.Settings.LogLevel
}
return "info"
}}
started(o, logs)
defer ended(o, logs)
syncer := &apply.Router{RunDir: o.RunDir, HostsFile: "/tmp/hosts/dawnline", DHCP: "/etc/config/dhcp", Log: logs,
Network: "/etc/config/network", Firewall: "/etc/config/firewall",
Remember: func(address string) {
(store.Store{Config: o.Config, RunDir: o.RunDir}).Transact(func(c *uci.Config, _ *model.Model) error {
if c.Section("settings") == nil {
c.Add("settings", "settings")
}
c.Set("settings", "lan_address", address)
return nil
})
},
Snapshot: func() *netview.Snapshot { return netview.Take(netview.System(nets.logical)) },
Exec: func(name string, args ...string) (string, int) {
out, err := exec.Command(name, args...).CombinedOutput()
if err == nil {
return string(out), 0
}
if ee, isExit := err.(*exec.ExitError); isExit {
return string(out), ee.ExitCode()
}
return string(out), 127
}}
// The keeper: once a minute, unless the router is given back to the ucode backend (mine), the runtime
// sync, the home-like path's upkeep and the watch on the router's networks. The ucode agent leaves
// those steps alone while this process is alive (its beat).
keeper := &apply.Keeper{R: syncer, Config: o.Config, Words: i18n.Use(i18n.Dir, ""),
Load: func() (*model.Model, string, error) {
c, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read()
if err != nil {
return nil, "", err
}
saved := ""
if s := c.Section("settings"); s != nil {
saved, _ = s.Get("lan_address")
}
return m, saved, nil
},
Usage: func() *usage.Usage {
if world := w.World(); world != nil {
return world.Usage
}
return nil
},
Dump: func() (map[string]awg.Peer, map[string]string) {
if world := w.World(); world != nil {
return world.Peers, world.Reads
}
return nil, nil
},
Tools: func() bool {
_, tool := os.Stat("/usr/bin/awg")
_, proto := os.Stat("/lib/netifd/proto/amneziawg.sh")
return tool == nil && proto == nil
},
Caps: func() awg.Caps { return awg.SystemCaps(o.DataDir) }}
// The other things done once a minute (package chores), with the same marker: what happens to people on
// its own goes to the activity log, and their pages are told.
var chores struct {
deps methods.Deps
run *minute.Chores
}
chores.run = &minute.Chores{RunDir: o.RunDir, DataDir: o.DataDir, Config: o.Config, Log: logs,
PortalCert: func(s model.Settings) (string, bool) { return methods.PortalCert(s, o.DataDir) },
WanAddress: func(srv *model.Server) string { return netview.Take(netview.System(nets.logical)).WanAddress(srv) },
Backup: func(kind string) (string, error) { return methods.BackupCreate(chores.deps, kind) },
Reload: func() {
cmd := exec.Command("sh", "-c", "/etc/init.d/dawnline reload >/dev/null 2>&1")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Start()
},
Updating: func() bool { return methods.Updating(o.RunDir) },
Update: func() {
exec.Command("start-stop-daemon", "-S", "-b", "-x", "/usr/libexec/dawnline/updater", "--", "auto").Run()
}}
// The places for the map: lookups run by themselves, a few at a time; what they find goes onto the
// connections logged before it was known, and those people's pages are told.
poke := func(people []string) {
if len(people) == 0 {
return
}
var waiting []string
files.ReadJSON(o.RunDir+"/presence.poke", &waiting)
files.WriteJSON(o.RunDir+"/presence.poke", append(waiting, people...), 0o600)
}
places := &geo.Locator{Store: geo.Store{Run: o.RunDir, Data: o.DataDir}, Get: geo.HTTP,
Usage: func() *usage.Usage {
if world := w.World(); world != nil {
return world.Usage
}
return nil
},
Found: func(where map[string]string, devices []string) {
journal.Log{Run: o.RunDir, Data: o.DataDir}.SetPlaces(where, time.Now().Unix()-3600)
world := w.World()
if world == nil {
return
}
var people []string
for _, id := range devices {
if d := world.M.Devices[id]; d != nil {
people = append(people, d.Person)
}
}
poke(people)
}}
chores.run.FlushPlaces = func() { places.Store.Flush() }
go func() {
// the addresses the watcher saw connect or move: located at once
for {
time.Sleep(time.Second)
if !mine() {
continue
}
var ips []string
if !files.ReadJSON(o.RunDir+"/geo.poke", &ips) {
continue
}
os.Remove(o.RunDir + "/geo.poke")
if world := w.World(); world != nil && world.M.Settings.GeoEnabled {
places.Ask(world.M.Settings.GeoProvider, ips, nil)
}
}
}()
others := func() {
world := w.World()
if world == nil {
return
}
_, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read()
if err != nil {
return
}
events := chores.run.Tick(m, world.Usage)
wan := ""
if st := chores.deps.Bus; st != nil {
list, _ := st("network.interface.wan", "status", nil)["ipv4-address"].([]any)
if len(list) > 0 {
first, _ := list[0].(map[string]any)
wan, _ = first["address"].(string)
}
}
places.Minute(m.Settings.GeoEnabled, m.Settings.GeoProvider, wan)
if len(events) == 0 {
return
}
log := journal.Log{Run: o.RunDir, Data: o.DataDir}
if err := log.Add(events, journal.PolicyOf(m.Settings.Retention), journal.NamesOf(m), time.Now().Unix()); err != nil {
logs.Say("error", "chores", "the activity log was not written · "+err.Error())
}
// the watcher moves these people's revisions with its next look, so their pages follow
var people []string
for _, e := range events {
if p, _ := e["person"].(string); p != "" {
people = append(people, p)
}
}
poke(people)
}
go func() {
// the first look once the watcher has its own, then every minute
for wait := 15 * time.Second; ; wait = time.Minute {
time.Sleep(wait)
if mine() {
beatNow(o)
keeper.Tick()
others()
// the watchdog: what keeps people connected, looked at and repaired
func() {
defer func() {
if e := recover(); e != nil {
logs.Say("error", "watchdog", fmt.Sprint(e))
}
}()
m, _, err := keeper.Load()
if err != nil {
return
}
facts := methods.WatchdogFacts(chores.deps, m, syncer, func() {
if world := w.World(); world != nil {
syncer.Apply(keeper.Load, world.Usage, keeper.Words, keeper.Tools(), keeper.Caps())
}
})
watchdog.Pass(o.RunDir+"/watchdog.json", watchdog.Probes(m, facts), m.Settings.Watchdog, methods.Updating(o.RunDir), time.Now().Unix(), logs)
}()
}
}
}()
deps := methods.Deps{World: func() *view.World { return roster.Complete(w.World(), nets.of, o.DataDir) }, Events: func() []map[string]any { return w.Recent(0) }, Catalogs: i18n.Dir,
Home: func(srv *model.Server, p *model.Person) []string { return nets.of(srv, p) }, HostIP: names.HostIP, RunDir: o.RunDir, DataDir: o.DataDir, Config: o.Config, Addresses: nets.addresses,
Snapshot: func() *netview.Snapshot { return netview.Take(netview.System(nets.logical)) },
Handler: nets.handler,
Resolve: names.Addresses,
ResetUsage: w.ResetUsage,
Teardown: func(m *model.Model) {
syncer.Teardown(m)
migrate.Router{Store: store.Store{Config: o.Config, RunDir: o.RunDir}, Uhttpd: "/etc/config/uhttpd", CGI: "/www/cgi-bin/dawnline-portal", Log: logs,
ReloadUhttpd: func() { exec.Command("/etc/init.d/uhttpd", "reload").Run() }}.UhttpdOnly()
// the agent's heartbeat check
exec.Command("sh", "-c", "grep -qs /usr/libexec/dawnline/heartbeat /etc/crontabs/root && { sed -i '\\#/usr/libexec/dawnline/heartbeat#d' /etc/crontabs/root; /etc/init.d/cron restart >/dev/null 2>&1; }").Run()
},
// after the answer is out: this process is one of the service's own
Stop: func() {
cmd := exec.Command("sh", "-c", "sleep 1; /etc/init.d/dawnline stop >/dev/null 2>&1; /etc/init.d/dawnline disable >/dev/null 2>&1")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Start()
},
ApplyPlan: func() any {
c, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read()
if err != nil {
return nil
}
saved := ""
if s := c.Section("settings"); s != nil {
saved, _ = s.Get("lan_address")
}
_, tool := os.Stat("/usr/bin/awg")
_, proto := os.Stat("/lib/netifd/proto/amneziawg.sh")
return syncer.ApplyPlan(m, saved, tool == nil && proto == nil, awg.SystemCaps(o.DataDir))
},
SyncPlan: func() any {
world := w.World()
_, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read()
if world == nil || err != nil {
return nil
}
return syncer.Plan(m, world.Usage, time.Now().Unix())
},
Background: func(script string) {
cmd := exec.Command("sh", "-c", "( "+script+" ) >/dev/null 2>&1 </dev/null")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Start()
},
Ifup: func(name string) { exec.Command("ifup", name).Run() },
Updater: func(lang string, args []string) {
cmd := exec.Command("start-stop-daemon", append([]string{"-S", "-b", "-x", "/usr/libexec/dawnline/updater", "--"}, args...)...)
if lang != "" {
cmd.Env = append(os.Environ(), "DAWNLINE_LANG="+lang)
}
cmd.Run()
},
Restart: func(network bool) {
script := "/etc/init.d/dawnline enabled || /etc/init.d/dawnline enable; sleep 1; "
if network {
script += "/etc/init.d/network restart; "
}
cmd := exec.Command("sh", "-c", script+"/etc/init.d/dawnline restart >/dev/null 2>&1")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Start()
},
// the full apply
Apply: func(lang string) map[string]any {
// Go's own full apply (unless the router is given back to ucode): the one-time steps,
// then network, firewall, the home-like path, the resolver, the reply table and the sync
if mine() {
if world := w.World(); world != nil {
st := store.Store{Config: o.Config, RunDir: o.RunDir}
migrate.Router{Store: st, NewCode: methods.NewCode, Uhttpd: "/etc/config/uhttpd", CGI: "/www/cgi-bin/dawnline-portal", Log: logs,
ReloadUhttpd: func() { exec.Command("/etc/init.d/uhttpd", "reload").Run() }}.Run()
_, tool := os.Stat("/usr/bin/awg")
_, proto := os.Stat("/lib/netifd/proto/amneziawg.sh")
full := syncer.Apply(func() (*model.Model, string, error) {
c, m, err := st.Read()
if err != nil {
return nil, "", err
}
saved := ""
if s := c.Section("settings"); s != nil {
saved, _ = s.Get("lan_address")
}
return m, saved, nil
}, world.Usage, i18n.Use(i18n.Dir, lang), tool == nil && proto == nil, awg.SystemCaps(o.DataDir))
out := map[string]any{"network": full.Network, "firewall": full.Firewall, "like_home": full.LikeHome, "peers": full.Peers, "warning": nil}
if full.Warning != "" {
out["warning"] = full.Warning
}
return out
}
}
// switched off while it runs (/etc/dawnline/backend-off without a restart): the ucode backend's
// own service applies when it starts
return map[string]any{"warning": "The change is saved. Dawnline's backend is switched off: restart the service to put it into effect (/etc/init.d/dawnline restart)."}
},
// after the answer is out: the reload may restart this very process (its portal's port is in its arguments)
Reload: func() {
cmd := exec.Command("sh", "-c", "sleep 1; /etc/init.d/dawnline reload >/dev/null 2>&1")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
cmd.Start()
},
AWG: func() map[string]any { return roster.AwgState(nets.handler) },
Store: &store.Store{Config: o.Config, RunDir: o.RunDir},
Told: func() {
if publish != nil {
publish()
}
},
// the runtime sync, and its warning for the admin
Sync: func(lang string) any {
// Go's own sync (unless the router is given back to ucode): the access sets, the devices
// into the tunnels and the hostnames, with no call to ucode
if mine() {
if world := w.World(); world != nil {
if _, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read(); err == nil {
res := syncer.Sync(m, world.Usage, time.Now().Unix(), false, i18n.Use(i18n.Dir, lang))
if res.Warning == "" {
return nil
}
return res.Warning
}
}
}
return "The change is saved. Dawnline's backend is switched off: restart the service to put it into effect (/etc/init.d/dawnline restart)."
},
Log: &servicelog.Log{RunDir: o.RunDir, Level: func() string {
if world := w.World(); world != nil {
return world.M.Settings.LogLevel
}
return "info"
}},
RouterWAN: nets.routerWAN,
Bus: func(object, method string, args map[string]any) map[string]any {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := nets.bus.Call(ctx, object, method, args)
if err != nil {
return nil
}
return res
},
WANOf: nets.wanOf,
GitHub: &github.Client{RunDir: o.RunDir},
Board: func() map[string]any {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, _ := nets.bus.Call(ctx, "system", "board", nil)
return res
}}
more = methods.List(deps)
chores.deps = deps
native = methods.PortalHandler(deps, true)
}
object := api.Object(h, audience, api.Info{Version: backend.Version, Stream: o.Stream, Refresh: func(ctx context.Context) { b.Look(ctx) }, More: more})
bus := ubus.NewBus(o.Socket, object)
bus.Log = o.Log
b = bridge.New(bridge.Bridge{
Hub: h, Audience: audience, Log: o.Log,
// what the topics are made of besides what the watcher tells of itself: the settings, the
// configurations handed out, the servers kept down, the places found
Watch: []string{o.Config, o.DataDir + "/issued.json", o.RunDir + "/netifd-blocked.json", o.RunDir + "/geo.json"},
})
run := []func(){
func() { bus.Run(ctx) },
func() { api.Broadcast(ctx, h, bus, object) },
func() { b.Run(ctx) },
}
if o.Watch {
run = append(run, func() { w.Run(ctx) })
// The people as Go makes them, held against each status ucode answers (package roster).
nets.bus = bus
ro := &roster.Roster{World: w.World, Home: nets.of, DataDir: o.DataDir, RunDir: o.RunDir, Hub: h, Log: o.Log, Keyed: bridge.Split, Recent: w.Recent, Own: nets.own, Iface: nets.ifaceState, Homes: nets.serverHomes, Handler: nets.handler, WAN: nets.routerWAN}
// The topics as the router is now: the watcher looks first, so what is told is never what it knew a
// second ago. Without that look a page that asked in the backend's first second was told nothing
// until Refresh (the watcher had no world yet), and settings changed on disk were told as they were
// before (the bridge's second comes before the watcher's).
tell := func() bool {
w.Look()
return ro.Publish()
}
publish = func() { tell() }
// told again when the settings change, when somebody starts to look, and with the minute
b.Tell = tell
// the moment the watcher saw someone come, go or move, the pages have Go's own people
w.Changed = func() { go ro.Publish() }
}
if p := o.Portal; p != nil {
bell := &portal.Bell{}
srv := &portal.Server{
TLS: []string{fmt.Sprintf(":%d", p.Port)}, Cert: p.Cert, Key: p.Key,
People: &portal.Settings{Path: o.Config}, Bell: bell, Log: o.Log,
// the portal's answers (methods.Portal)
Native: native,
}
if p.HTTP > 0 {
srv.Plain = []string{fmt.Sprintf(":%d", p.HTTP)}
}
run = append(run, func() { srv.Run(ctx) }, func() { portal.Watch(ctx, bell, o.RunDir, o.DataDir, o.Config) })
}
var parts sync.WaitGroup
for _, part := range run {
parts.Add(1)
go func() {
defer parts.Done()
part()
}()
}
parts.Wait()
}
// homes answers which home networks a person reaches, from a look at the router's interfaces that is kept
// for a few seconds (every person of a status asks).
type homes struct {
bus *ubus.Bus
mu sync.Mutex
snap *netview.Snapshot
at time.Time
}
func (h *homes) of(srv *model.Server, p *model.Person) []string {
h.mu.Lock()
defer h.mu.Unlock()
if h.snap == nil || time.Since(h.at) > 5*time.Second {
h.snap = netview.Take(netview.System(h.logical))
h.at = time.Now()
}
return h.snap.PersonHome(srv, p)
}
// own is the router's own public-side addresses: each server's WAN address, and the wan interface's.
func (h *homes) own(m *model.Model) map[string]bool {
h.mu.Lock()
if h.snap == nil || time.Since(h.at) > 5*time.Second {
h.snap = netview.Take(netview.System(h.logical))
h.at = time.Now()
}
snap := h.snap
h.mu.Unlock()
out := map[string]bool{}
for _, srv := range m.Servers {
a := snap.WanAddress(srv)
if a == "" {
a = h.ifaceAddr(srv.WanZone)
}
if a != "" {
out[a] = true
}
}
if a := h.ifaceAddr("wan"); a != "" {
out[a] = true
}
return out
}
// addresses is every IPv4 address of netifd's interfaces, in its order.
func (h *homes) addresses() []methods.Address {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := h.bus.Call(ctx, "network.interface", "dump", nil)
if err != nil {
return nil
}
var out []methods.Address
list, _ := res["interface"].([]any)
for _, i := range list {
m, _ := i.(map[string]any)
name, _ := m["interface"].(string)
addrs, _ := m["ipv4-address"].([]any)
for _, a := range addrs {
am, _ := a.(map[string]any)
out = append(out, methods.Address{Iface: name, CIDR: fmt.Sprintf("%v/%v", am["address"], am["mask"])})
}
}
return out
}
// wanOf is the IPv4 address of a server's WAN (its interface's, else its zone's netifd interface's); of
// netifd's wan for no server.
func (h *homes) wanOf(srv *model.Server) string {
if srv == nil {
return h.ifaceAddr("wan")
}
if a := netview.Take(netview.System(h.logical)).WanAddress(srv); a != "" {
return a
}
return h.ifaceAddr(srv.WanZone)
}
// handler tells whether netifd lists the amneziawg protocol.
func (h *homes) handler() bool {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := h.bus.Call(ctx, "network", "get_proto_handlers", nil)
_, has := res["amneziawg"]
return err == nil && has
}
// routerWAN is the router's WAN address: the first server's with keys, else its zone's interface's.
func (h *homes) routerWAN(m *model.Model) string {
h.mu.Lock()
if h.snap == nil || time.Since(h.at) > 5*time.Second {
h.snap = netview.Take(netview.System(h.logical))
h.at = time.Now()
}
snap := h.snap
h.mu.Unlock()
for _, id := range m.ServerIDs() {
srv := m.Servers[id]
if srv.PrivateKey == "" {
continue
}
if a := snap.WanAddress(srv); a != "" {
return a
}
if a := h.ifaceAddr(srv.WanZone); a != "" {
return a
}
}
return h.ifaceAddr("wan")
}
// ifaceState is whether a netifd interface is up, and for how long.
func (h *homes) ifaceState(name string) (bool, int64) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := h.bus.Call(ctx, "network.interface."+name, "status", nil)
if err != nil {
return false, 0
}
up, _ := res["up"].(bool)
var uptime int64
switch n := res["uptime"].(type) {
case int64:
uptime = n
case int32:
uptime = int64(n)
case float64:
uptime = int64(n)
case int:
uptime = int64(n)
}
return up, uptime
}
// serverHomes is a server's home interfaces with what each reaches.
func (h *homes) serverHomes(srv *model.Server) []view.HomeNet {
h.mu.Lock()
defer h.mu.Unlock()
if h.snap == nil || time.Since(h.at) > 5*time.Second {
h.snap = netview.Take(netview.System(h.logical))
h.at = time.Now()
}
home, _ := h.snap.ServerHome(srv)
out := []view.HomeNet{}
for _, x := range home {
out = append(out, view.HomeNet{Dev: x.Dev, Nets: x.Nets})
}
return out
}
// ifaceAddr is a netifd interface's first IPv4 address, or "".
func (h *homes) ifaceAddr(name string) string {
if name == "" {
return ""
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := h.bus.Call(ctx, "network.interface."+name, "status", nil)
if err != nil {
return ""
}
list, _ := res["ipv4-address"].([]any)
if len(list) == 0 {
return ""
}
first, _ := list[0].(map[string]any)
a, _ := first["address"].(string)
return a
}
// logical asks netifd which interface stands on which device.
func (h *homes) logical() map[string]netview.Logical {
out := map[string]netview.Logical{}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
res, err := h.bus.Call(ctx, "network.interface", "dump", nil)
if err != nil {
return out
}
list, _ := res["interface"].([]any)
for _, i := range list {
m, _ := i.(map[string]any)
dev, _ := m["l3_device"].(string)
if dev == "" {
dev, _ = m["device"].(string)
}
if _, seen := out[dev]; dev == "" || seen {
continue
}
name, _ := m["interface"].(string)
proto, _ := m["proto"].(string)
out[dev] = netview.Logical{Name: name, Proto: proto}
}
return out
}
+368
View File
@@ -0,0 +1,368 @@
//go:build e2e
// The whole path a page takes, on a real OpenWrt userland: uhttpd's /ubus,
// rpcd's sessions and access lists (the package's own ACL file), ubusd and the
// backend with its own watcher and topics, over a stand's settings and tunnel.
// scripts/backend-container.sh starts the daemons and runs this.
package daemon
import (
"bufio"
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
"github.com/tickcount/luci-app-dawnline/backend/awg"
"github.com/tickcount/luci-app-dawnline/backend/hub"
"github.com/tickcount/luci-app-dawnline/backend/ubus"
)
const web = "http://127.0.0.1:8099/ubus"
// rpc is a JSON-RPC call through uhttpd, as LuCI's rpc.js makes it. It
// returns ubus's status and the answer.
func rpc(t *testing.T, sid, object, method string, args map[string]any) (int, map[string]any) {
t.Helper()
body, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "call", "params": []any{sid, object, method, args}})
res, err := http.Post(web, "application/json", bytes.NewReader(body))
if err != nil {
t.Fatal(err)
}
defer res.Body.Close()
var out struct {
Result []json.RawMessage
Error *struct {
Code int
Message string
}
}
if err := json.NewDecoder(res.Body).Decode(&out); err != nil {
t.Fatal(err)
}
if out.Error != nil {
return out.Error.Code, nil
}
var code int
json.Unmarshal(out.Result[0], &code)
var data map[string]any
if len(out.Result) > 1 {
json.Unmarshal(out.Result[1], &data)
}
return code, data
}
func at(v any, path ...string) any {
for _, k := range path {
m, _ := v.(map[string]any)
v = m[k]
}
return v
}
// The stand's settings: one server, one person, one device.
const (
standServerKey = "yAnz5TF+lXXJte14tji3zlMNq+hd2rYUIgJBgB3fBmk="
standServerPub = "xTIBA5rboUvnH4htodjb6e697QjLERt1NAB4mZqp8Dg="
standDeviceKey = "HIgo9xNzJMWLKASShiTqIybxZ0U3wGLiUeJ1PKf8ykw="
)
func standSettings(person string) string {
return `
config settings 'settings'
config server 's1'
option name 'Home'
option ifname 'dawn0'
option subnet '10.77.0.0/24'
option private_key '` + standServerKey + `'
option public_key '` + standServerPub + `'
config person 'p1'
option server 's1'
option name '` + person + `'
config device 'd1'
option person 'p1'
option name 'Phone'
option kind 'phone'
option address '10.77.0.2'
option public_key '` + standDeviceKey + `'
`
}
// tunnel is the stand's one tunnel: its device is there, with the handshake the test gave it.
type tunnel struct {
mu sync.Mutex
handshake int64
}
func (tn *tunnel) reader() *awg.Reader {
return &awg.Reader{
// as `awg show <ifname> dump` prints: the interface's line, then a peer a line
Show: func(string) (string, bool) {
tn.mu.Lock()
defer tn.mu.Unlock()
return standServerKey + "\t" + standServerPub + "\t4500\toff\n" +
fmt.Sprintf("%s\t(none)\t203.0.113.7:41821\t10.77.0.2/32\t%d\t0\t0\toff\n", standDeviceKey, tn.handshake), true
},
Index: func(string) (int, bool) { return 7, true },
}
}
func (tn *tunnel) shake(at int64) {
tn.mu.Lock()
tn.handshake = at
tn.mu.Unlock()
}
// follow asks with wait until the documents satisfy `done`, taking the events into them, and returns the
// number the page is at and the events of the last answer. The topics are the page's whole snapshot.
func follow(t *testing.T, sid, epoch string, seq float64, docs map[string]any, topics []any, what string, done func() bool) (float64, []any) {
t.Helper()
deadline := time.Now().Add(25 * time.Second)
for {
args := map[string]any{"epoch": epoch, "seq": seq, "timeout": 10}
if topics != nil {
args["topics"] = topics
}
code, got := rpc(t, sid, "dawnline", "wait", args)
events, _ := got["events"].([]any)
if code != 0 || got["docs"] != nil || (len(events) == 0 && time.Now().After(deadline)) {
t.Fatalf("wait for %s: %d %v", what, code, got)
}
take(docs, events)
seq = got["seq"].(float64)
if done() {
return seq, events
}
}
}
// take applies events to a page's documents, as a page does.
func take(docs map[string]any, events []any) {
for _, e := range events {
topic, _ := at(e, "topic").(string)
switch at(e, "op") {
case "put":
docs[topic] = at(e, "data")
case "patch":
docs[topic] = hub.Merge(docs[topic], at(e, "data"))
}
}
}
func TestThroughUhttpd(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
dir := t.TempDir()
config := filepath.Join(dir, "config")
settings := func(person string) {
if err := os.WriteFile(config, []byte(standSettings(person)), 0o600); err != nil {
t.Fatal(err)
}
}
settings("Myata")
// The stand is no router to keep: the minute's work (the access rules, the watchdog) stays away, the
// watcher and the topics are the backend's own.
if err := os.WriteFile(filepath.Join(dir, "backend-off"), nil, 0o600); err != nil {
t.Fatal(err)
}
// A backend that was restarted: its earlier run left who was online, and nobody came or went since, so
// the watcher's first look finds nothing to tell of by itself.
run := filepath.Join(dir, "run")
os.MkdirAll(run, 0o755)
if err := os.WriteFile(filepath.Join(run, "presence.json"), []byte(`{"online":{"d1":false},"endpoint":{"d1":"203.0.113.7"},"rev":1,"people":{"p1":1}}`), 0o600); err != nil {
t.Fatal(err)
}
tn := &tunnel{}
began := time.Now()
// The backend is stopped, and has stopped, before the test ends: it writes into the stand's folders (its beat,
// its presence) and logs through t, and a test that ended under it failed at removing a folder still written to.
stopped := make(chan struct{})
go func() {
defer close(stopped)
Run(ctx, Options{Socket: ubus.Socket, RunDir: run, DataDir: dir, Config: config, Stream: "sse", Watch: true, Tunnels: tn.reader(), Log: t.Logf})
}()
defer func() {
cancel()
select {
case <-stopped:
case <-time.After(10 * time.Second):
t.Error("the backend did not stop")
}
}()
for exec.CommandContext(ctx, "ubus", "list", "dawnline").Run() != nil {
if ctx.Err() != nil {
t.Fatal("the object never appeared on the bus")
}
time.Sleep(20 * time.Millisecond)
}
// Without a session rpcd refuses; with one, the read ACL lets the page in.
if code, _ := rpc(t, "00000000000000000000000000000000", "dawnline", "hello", nil); code == 0 {
t.Fatal("answered without a session")
}
code, login := rpc(t, "00000000000000000000000000000000", "session", "login", map[string]any{"username": "root", "password": ""})
if code != 0 {
t.Fatalf("login: %d", code)
}
sid := login["ubus_rpc_session"].(string)
code, hello := rpc(t, sid, "dawnline", "hello", map[string]any{})
if code != 0 || hello["stream"] != "sse" || hello["epoch"] == "" {
t.Fatalf("hello: %d %v", code, hello)
}
// A page that was open while the backend restarted asks at once, before the watcher's first look: its
// first question starts the bridge, and the documents come with the bridge's next second. (They once
// came only with the next Refresh, half a minute later: the bridge took topics nobody could make yet
// for told.)
asked := time.Now()
code, snap := rpc(t, sid, "dawnline", "sync", map[string]any{})
if code != 0 {
t.Fatalf("sync: %d", code)
}
epoch, seq := snap["epoch"].(string), snap["seq"].(float64)
docs, _ := snap["docs"].(map[string]any)
if docs == nil {
docs = map[string]any{}
}
whole := func() bool {
return at(docs, "people", "items", "p1", "name") == "Myata" && at(docs, "servers", "items", "s1") != nil && at(docs, "backend", "ok") == true
}
if !whole() {
seq, _ = follow(t, sid, epoch, seq, docs, nil, "the first documents", whole)
}
took := time.Since(asked)
t.Logf("the page asked %v after the backend started and had its first documents %v later", asked.Sub(began).Round(time.Millisecond), took.Round(time.Millisecond))
if took > 3*time.Second {
t.Fatalf("the first documents took %v", took)
}
if at(docs, "people", "items", "p1", "devices", "items", "d1", "online") != false || at(docs, "servers", "order").([]any)[0] != "s1" {
t.Fatalf("documents: %v", docs)
}
// An event stream, as a page opens it; on a web server that cannot hold one this part is skipped.
stream, lines := openStream(t, ctx, sid)
// A device comes online: the long poll returns with a patch that says so, and not the people whole.
online := func() bool { return at(docs, "people", "items", "p1", "devices", "items", "d1", "online") == true }
type result struct {
seq float64
events []any
}
waited := make(chan result, 1)
go func() {
s, events := follow(t, sid, epoch, seq, docs, []any{"people"}, "the device to come online", online)
waited <- result{s, events}
}()
time.Sleep(300 * time.Millisecond)
changed := time.Now()
tn.shake(time.Now().Unix())
var r result
select {
case r = <-waited:
case <-time.After(40 * time.Second):
t.Fatal("the device never came online for the page")
}
var patch any
for _, e := range r.events {
if at(e, "op") == "patch" && at(e, "data", "items", "p1", "devices", "items", "d1", "online") == true {
patch = e
}
}
if patch == nil || at(patch, "data", "items", "p1", "name") != nil || at(patch, "data", "order") != nil {
t.Fatalf("patch: %v", r.events)
}
t.Logf("a device that came online reached the page %v later, as %d bytes", time.Since(changed).Round(time.Millisecond), len(fmt.Sprint(at(patch, "data"))))
if stream {
// The same event came down the stream, with the same number.
deadline := time.After(10 * time.Second)
for found := false; !found; {
select {
case l := <-lines:
var e map[string]any
if strings.HasPrefix(l, "data: ") && json.Unmarshal([]byte(l[6:]), &e) == nil && e["seq"] == at(patch, "seq") {
if e["topic"] != "people" || e["op"] != "patch" || e["epoch"] != epoch || !reflect.DeepEqual(e["data"], at(patch, "data")) {
t.Fatalf("the stream's event differs: %v", e)
}
found = true
}
case <-deadline:
t.Fatal("the event never came down the stream")
}
}
}
// The meter's points come as they are made, to a page that names it.
if code, _ := rpc(t, sid, "dawnline", "watch", map[string]any{"topics": []any{"meter"}}); code != 0 {
t.Fatalf("watch: %d", code)
}
code, got := rpc(t, sid, "dawnline", "wait", map[string]any{"epoch": epoch, "seq": r.seq, "topics": []any{"meter"}, "timeout": 10})
events, _ := got["events"].([]any)
appended := false
for _, e := range events {
appended = appended || at(e, "op") == "append"
}
if code != 0 || !appended {
t.Fatalf("meter: %d %v", code, got)
}
// A page reads its own edit: once refresh answers, the stream holds what is on disk, with no waiting
// for the watcher's second.
settings("Mint")
code, stands := rpc(t, sid, "dawnline", "refresh", map[string]any{})
if code != 0 || stands["epoch"] != epoch {
t.Fatalf("refresh: %d %v", code, stands)
}
code, got = rpc(t, sid, "dawnline", "sync", map[string]any{"epoch": epoch, "seq": r.seq, "topics": []any{"people"}})
events, _ = got["events"].([]any)
take(docs, events)
if code != 0 || at(docs, "people", "items", "p1", "name") != "Mint" || got["seq"].(float64) < stands["seq"].(float64) {
t.Fatalf("after refresh: %d %v (the stream stands at %v)", code, got, stands["seq"])
}
// Arguments are checked at the door.
if code, _ := rpc(t, sid, "dawnline", "sync", map[string]any{"seq": -1}); code != int(ubus.StatusInvalidArgument) {
t.Fatalf("a negative number was accepted: %d", code)
}
if code, _ := rpc(t, sid, "dawnline", "sync", map[string]any{"topics": "people"}); code != int(ubus.StatusInvalidArgument) {
t.Fatalf("topics that are no list were accepted: %d", code)
}
}
// openStream subscribes through uhttpd and returns the stream's lines.
func openStream(t *testing.T, ctx context.Context, sid string) (bool, <-chan string) {
req, _ := http.NewRequestWithContext(ctx, "GET", web+"/subscribe/dawnline", nil)
req.Header.Set("Authorization", "Bearer "+sid)
res, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
if ct := res.Header.Get("Content-Type"); !strings.HasPrefix(ct, "text/event-stream") {
t.Logf("no event stream from this web server (%s): %s", res.Status, ct)
res.Body.Close()
return false, nil
}
lines := make(chan string, 64)
go func() {
defer res.Body.Close()
sc := bufio.NewScanner(res.Body)
sc.Buffer(make([]byte, 0, 64<<10), 2<<20)
for sc.Scan() {
select {
case lines <- sc.Text():
default:
}
}
}()
return true, lines
}
+93
View File
@@ -0,0 +1,93 @@
package daemon
import (
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"time"
backend "github.com/tickcount/luci-app-dawnline/backend"
"github.com/tickcount/luci-app-dawnline/backend/geo"
"github.com/tickcount/luci-app-dawnline/backend/journal"
"github.com/tickcount/luci-app-dawnline/backend/servicelog"
"github.com/tickcount/luci-app-dawnline/backend/store"
"github.com/tickcount/luci-app-dawnline/backend/updater"
)
// The beat is the time of the backend's last minute: the Health page calls it fine while it is fresh, and
// a beat left behind by a backend that is gone tells the next start that the last one did not end well.
const beatFile = "/agent.beat"
func readTrim(path string) string {
raw, _ := os.ReadFile(path)
return strings.Trim(string(raw), " \t\r\n\v\f")
}
// started says in the service log why this start: an update that was rolled back, a new version, a crash
// (the beat outlives a backend that died), a restart (the init script notes a stop), a boot, or a plain
// start. It also puts back the service log a reboot took from memory, and remembers the kernel an
// AmneziaWG module Dawnline did not install is for.
func started(o Options, logs *servicelog.Log) {
os.MkdirAll(o.RunDir, 0o755)
_, beat := os.Stat(o.RunDir + beatFile)
crashed := beat == nil
stopped, _ := strconv.ParseInt(readTrim(o.RunDir+"/agent.stopped"), 10, 64)
os.Remove(o.RunDir + "/agent.stopped")
was := readTrim(o.DataDir + "/version")
v := backend.Version
if was != v {
os.MkdirAll(o.DataDir, 0o700)
os.WriteFile(o.DataDir+"/version", []byte(v), 0o644)
}
// the service log lives in memory: after a reboot, what flash kept of it
if _, err := os.Stat(o.RunDir + "/service.log"); err != nil {
if raw, err := os.ReadFile(o.DataDir + "/service.log"); err == nil {
os.WriteFile(o.RunDir+"/service.log", raw, 0o600)
}
}
say := func(level, msg, detail string) {
logs.Write("dawnline", level, msg, detail, "Dawnline", servicelog.Extra{})
}
rolled := readTrim(o.RunDir + "/update-rolled-back")
os.Remove(o.RunDir + "/update-rolled-back")
uptime := 0.0
fmt.Sscan(readTrim("/proc/uptime"), &uptime)
switch {
// postupdate put this (kept) version back because the new one did not come up
case rolled != "":
say("error", fmt.Sprintf("update to %s failed its check - rolled back to %s", rolled, v),
"The new version did not answer in LuCI or its agent did not start. Try again from Components, or install it with install.sh to see why.")
case was != "" && was != v:
say("up", fmt.Sprintf("updated · %s → %s", was, v), "")
case crashed:
// why, when syslog knows: procd, the kernel (out of memory, a fault)
why, _ := exec.Command("sh", "-c", `logread -l 400 2>/dev/null | grep -iE "kernel:.*(out of memory|killed process|oom-kill|segfault)|procd:.*dawnline|dawnlined" | tail -n 6`).Output()
say("repair", "stopped unexpectedly - started again", strings.Trim(string(why), " \t\r\n\v\f"))
case stopped != 0 && time.Now().Unix()-stopped < 300:
say("info", "restarted · version "+v, "")
case uptime < 600:
say("info", "started with the router · version "+v, "")
default:
say("info", "started · version "+v, "")
}
updater.Adopt(o.DataDir)
beatNow(o)
}
func beatNow(o Options) {
os.WriteFile(o.RunDir+beatFile, []byte(strconv.FormatInt(time.Now().Unix(), 10)), 0o644)
}
// ended is a clean stop: what lives in memory goes to flash (the activity log, the service log, the places
// found), and the beat goes, so the next start is not taken for a crash. The counters are the watcher's.
func ended(o Options, logs *servicelog.Log) {
now := time.Now().Unix()
if _, m, err := (store.Store{Config: o.Config, RunDir: o.RunDir}).Read(); err == nil {
journal.Log{Run: o.RunDir, Data: o.DataDir}.Persist(journal.PolicyOf(m.Settings.Retention), now)
}
logs.Trim(o.DataDir, true)
geo.Store{Run: o.RunDir, Data: o.DataDir}.Flush()
os.Remove(o.RunDir + beatFile)
}