Dawnline 0.20.0
Build packages / CI (push) Blocked by required conditions
Build packages / Version (push) Waiting to run
CI / Web (format, types, tests, bundles) (push) Blocked by required conditions
CI / ShellCheck (push) Blocked by required conditions
CI / Backend (Go: vet, tests, a real ubusd and uhttpd) (push) Blocked by required conditions
CI / Package files (push) Blocked by required conditions
Build packages / CI (push) Blocked by required conditions
Build packages / Version (push) Waiting to run
CI / Web (format, types, tests, bundles) (push) Blocked by required conditions
CI / ShellCheck (push) Blocked by required conditions
CI / Backend (Go: vet, tests, a real ubusd and uhttpd) (push) Blocked by required conditions
CI / Package files (push) Blocked by required conditions
- **Backend rewritten in Go.** One daemon, `dawnlined`, replaces the whole ucode backend: the rpcd plugin, the agent, the CLI and the updater. The package no longer depends on ucode. - **Per-architecture builds.** The release carries the backend for arm64, ARMv7, mips, mipsle and amd64; the router fetches its own after the package is installed and checks its sha256. Dawnline does not run on other architectures (older ARM, 32-bit x86, RISC-V). - **Faster on the router.** A firewall sync takes 0.06 s instead of 0.9 s; a full apply takes about 300 ms. The portal is served by the daemon itself, without a CGI process per request. - **Live pages.** The admin and the portal follow a live stream from the router instead of re-fetching the whole state every 30 seconds. - **Admin redesigned in the portal's style.** Three sections (Overview · Log · Settings); a server is a page of its own with People, Places and Settings; a device opens as a dialog over its person. - **Changes wait for «Apply».** Every setting, and a person's access, is staged in one dock before it reaches the router. - **Configuration rollout.** After a change that makes devices fetch their configuration again, the admin shows how many have fetched it and who has not. Every setting shows beforehand (⟳N) how many devices it will touch. - **Waiting for a decision.** Overview and People put first the people who need one (access ended, limit reached, a device that does not connect), each with its one action a tap away. - **Bulk actions on people.** «Select» (or a long press on a card) picks several people to extend by a month, pause or resume at once. - **Undo.** An action done from a card or a request is undone from its toast. - **Update dialogs redesigned.** An update is one view from its question to its result; Dawnline's own update runs in it like every other, and the page reloads itself once the new version answers. Other versions are a list, each with its notes and Install on a page of its own. - **Site to site: networks behind a router.** A router connected as a device can have networks behind it (up to eight, /16 to /30): the server routes them and applies the router's access to them. A network that shares addresses with a tunnel, the router's own networks or another site is refused by name. - **Routers from the portal.** People can add a router and name its networks on their own page; the admin accepts or declines with a reason, or lets them through without asking. - **Check access and Health know the networks behind routers.** - **Usage limits reworked.** A monthly limit no longer follows the calendar. A person's month starts with their first traffic and ends a month later; the next one starts only when they connect again, so time away uses up none of it. Before, everyone's limit reset on the 1st. - Extra data lasts until that person's month ends; «Reset usage» starts the month anew. - A limit is for a month or for the whole term; the admin and the portal show when it resets. - Charts that said «this month» show the last 30 days. - After the update nobody loses what was counted: the month that was running becomes the person's first and ends on the 1st. - **Access by time of day removed.** The Hours limit is gone from the admin, the portal and the backend; saved hours are dropped. - **Expiry simplified.** An end date always stops access, with a heads-up three days before; the per-person choice is gone. - **No false disconnects from sleeping phones.** A phone or a tablet stays online through ten minutes of silence. - **Mobile Safari fixes.** Pressing a control no longer sends the page to its top; tab tracks that do not fit scroll; switches animate after a press.
This commit is contained in:
1 parent
f8130e39a8
commit
ac90575b0b
788 files changed
+139244
-41720
No files matched your search
@@ -0,0 +1,341 @@
|
||||
// Package geo is the city-level places for the admin's map (dawnline.geo). Client addresses are looked up
|
||||
// once with a third-party service (ipwho.is, ipapi.co as the other) and kept: the live copy in the run
|
||||
// directory (tmpfs), put on flash about hourly, like the traffic counters. The pages read the cache.
|
||||
package geo
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/tickcount/luci-app-dawnline/backend/files"
|
||||
"github.com/tickcount/luci-app-dawnline/backend/ipnet"
|
||||
)
|
||||
|
||||
const (
|
||||
ttl = 30 * 86400
|
||||
retry = 86400
|
||||
// limit is the entries kept at least; addresses still shown are never pushed out.
|
||||
limit = 500
|
||||
)
|
||||
|
||||
// Place is what is kept of an address, as the cache's file has it (lat, lon, city, country, cc, asn, org,
|
||||
// ts; or fail alone: the moment a lookup found nothing).
|
||||
type Place = map[string]any
|
||||
|
||||
// Cache is the file: the places by address, and the router's own.
|
||||
type Cache struct {
|
||||
IPs map[string]Place `json:"ips"`
|
||||
Home Place `json:"home"`
|
||||
}
|
||||
|
||||
// Store is where the cache lives.
|
||||
type Store struct {
|
||||
Run, Data string
|
||||
}
|
||||
|
||||
func (s Store) Load() *Cache {
|
||||
c := &Cache{}
|
||||
if !files.ReadJSON(s.Run+"/geo.json", c) {
|
||||
*c = Cache{}
|
||||
files.ReadJSON(s.Data+"/geo.json", c)
|
||||
}
|
||||
if c.IPs == nil {
|
||||
c.IPs = map[string]Place{}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Save writes the live copy; Flush puts it on flash.
|
||||
func (s Store) Save(c *Cache) {
|
||||
os.MkdirAll(s.Run, 0o755)
|
||||
files.WriteJSON(s.Run+"/geo.json", c, 0o600)
|
||||
}
|
||||
|
||||
// Flush copies the live cache to flash when it is newer; whether it did.
|
||||
func (s Store) Flush() bool {
|
||||
live, err := os.Stat(s.Run + "/geo.json")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if kept, err := os.Stat(s.Data + "/geo.json"); err == nil && !live.ModTime().After(kept.ModTime()) {
|
||||
return false
|
||||
}
|
||||
var c Cache
|
||||
if !files.ReadJSON(s.Run+"/geo.json", &c) {
|
||||
return false
|
||||
}
|
||||
os.MkdirAll(s.Data, 0o700)
|
||||
return files.WriteJSON(s.Data+"/geo.json", c, 0o600) == nil
|
||||
}
|
||||
|
||||
var private4 = []ipnet.Net{}
|
||||
|
||||
func init() {
|
||||
for _, n := range []string{"0.0.0.0/8", "10.0.0.0/8", "100.64.0.0/10", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.168.0.0/16", "198.18.0.0/15", "224.0.0.0/3"} {
|
||||
if c, ok := ipnet.Any(n); ok {
|
||||
private4 = append(private4, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
v6 = regexp.MustCompile(`^[0-9a-f:]+$`)
|
||||
localV6 = regexp.MustCompile(`^(fe[89ab]|f[cd]|::1$|::$)`)
|
||||
)
|
||||
|
||||
// IsPublic tells an address a location service can say something about.
|
||||
func IsPublic(ip string) bool {
|
||||
if ip == "" {
|
||||
return false
|
||||
}
|
||||
if n, ok := ipnet.IP(ip); ok {
|
||||
for _, c := range private4 {
|
||||
if c.Holds(n) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
l := strings.ToLower(ip)
|
||||
return v6.MatchString(l) && !localV6.MatchString(l)
|
||||
}
|
||||
|
||||
func num(v any) (float64, bool) {
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
return x, true
|
||||
case string:
|
||||
f, err := strconv.ParseFloat(x, 64)
|
||||
return f, err == nil
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func str(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
// Stale is the public addresses of ips the cache does not know or knows from too long ago (a failed lookup
|
||||
// is tried again after a day), once each and in order.
|
||||
func Stale(c *Cache, ips []string, now int64) []string {
|
||||
out := []string{}
|
||||
seen := map[string]bool{}
|
||||
for _, ip := range ips {
|
||||
if seen[ip] || !IsPublic(ip) {
|
||||
continue
|
||||
}
|
||||
seen[ip] = true
|
||||
if p := c.IPs[ip]; p != nil {
|
||||
if fail, failed := num(p["fail"]); failed && fail != 0 {
|
||||
if float64(now)-fail < retry {
|
||||
continue
|
||||
}
|
||||
} else if ts, _ := num(p["ts"]); p["asn"] != nil && float64(now)-ts < ttl {
|
||||
continue
|
||||
}
|
||||
}
|
||||
out = append(out, ip)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Fetch gets a URL's JSON (nil: not there, or not JSON).
|
||||
type Fetch func(url string) map[string]any
|
||||
|
||||
// HTTP is Fetch over the network, five seconds at most.
|
||||
func HTTP(url string) map[string]any {
|
||||
req, err := http.NewRequest("GET", url, nil)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
res, err := (&http.Client{Timeout: 5 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer res.Body.Close()
|
||||
body, err := io.ReadAll(io.LimitReader(res.Body, 1<<20))
|
||||
if err != nil || res.StatusCode != 200 {
|
||||
return nil
|
||||
}
|
||||
var out map[string]any
|
||||
if json.Unmarshal(body, &out) != nil {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ipwho(get Fetch, ip string) Place {
|
||||
j := get("https://ipwho.is/" + ip + "?fields=success,ip,latitude,longitude,city,country,country_code,connection")
|
||||
if j == nil || j["success"] != true || j["latitude"] == nil {
|
||||
return nil
|
||||
}
|
||||
lat, _ := num(j["latitude"])
|
||||
lon, _ := num(j["longitude"])
|
||||
cn, _ := j["connection"].(map[string]any)
|
||||
asn, _ := num(cn["asn"])
|
||||
org := str(cn["isp"])
|
||||
if org == "" {
|
||||
org = str(cn["org"])
|
||||
}
|
||||
return Place{"lat": lat, "lon": lon, "city": str(j["city"]), "country": str(j["country"]), "cc": str(j["country_code"]), "ip": j["ip"], "asn": asn, "org": org}
|
||||
}
|
||||
|
||||
var asPrefix = regexp.MustCompile(`(?i)^AS`)
|
||||
|
||||
func ipapi(get Fetch, ip string) Place {
|
||||
url := "https://ipapi.co/json/"
|
||||
if ip != "" {
|
||||
url = "https://ipapi.co/" + ip + "/json/"
|
||||
}
|
||||
j := get(url)
|
||||
if j == nil || (j["error"] != nil && j["error"] != false) || j["latitude"] == nil {
|
||||
return nil
|
||||
}
|
||||
lat, _ := num(j["latitude"])
|
||||
lon, _ := num(j["longitude"])
|
||||
asn, _ := strconv.ParseFloat(asPrefix.ReplaceAllString(fmt.Sprint(j["asn"]), ""), 64)
|
||||
return Place{"lat": lat, "lon": lon, "city": str(j["city"]), "country": str(j["country_name"]), "cc": str(j["country_code"]), "ip": j["ip"], "asn": asn, "org": str(j["org"])}
|
||||
}
|
||||
|
||||
// Lookup is one lookup with the chosen provider and the other as fallback (ip "": whoever asks).
|
||||
func Lookup(get Fetch, provider, ip string) Place {
|
||||
order := []func(Fetch, string) Place{ipwho, ipapi}
|
||||
if provider == "ipapi.co" {
|
||||
order = []func(Fetch, string) Place{ipapi, ipwho}
|
||||
}
|
||||
for _, fn := range order {
|
||||
if p := fn(get, ip); p != nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LookupAll looks up at most max of ips, within budget seconds (each lookup may wait on the network): a
|
||||
// place, or nil where the services had nothing, for the ones tried. Network only, no cache.
|
||||
func LookupAll(get Fetch, provider string, ips []string, max int, budget time.Duration) map[string]Place {
|
||||
started := time.Now()
|
||||
out := map[string]Place{}
|
||||
for _, ip := range ips {
|
||||
if len(out) >= max || time.Since(started) >= budget {
|
||||
break
|
||||
}
|
||||
p := Lookup(get, provider, ip)
|
||||
if p != nil {
|
||||
delete(p, "ip")
|
||||
}
|
||||
out[ip] = p
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func when(p Place) float64 {
|
||||
if ts, ok := num(p["ts"]); ok {
|
||||
return ts
|
||||
}
|
||||
fail, _ := num(p["fail"])
|
||||
return fail
|
||||
}
|
||||
|
||||
// Merge puts LookupAll's results into the cache; true when it changed. shown is what is still on the map:
|
||||
// the cache is kept bounded by dropping other entries only, oldest first, so a lookup is never repeated
|
||||
// for an address that is still on the map.
|
||||
func Merge(c *Cache, found map[string]Place, shown []string, now int64) bool {
|
||||
changed := false
|
||||
for ip, p := range found {
|
||||
if !IsPublic(ip) {
|
||||
continue
|
||||
}
|
||||
if p != nil {
|
||||
kept := Place{}
|
||||
for k, v := range p {
|
||||
kept[k] = v
|
||||
}
|
||||
kept["ts"] = float64(now)
|
||||
c.IPs[ip] = kept
|
||||
} else {
|
||||
c.IPs[ip] = Place{"fail": float64(now)}
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
if over := len(c.IPs) - limit; over > 0 {
|
||||
keep := map[string]bool{}
|
||||
for _, ip := range shown {
|
||||
keep[ip] = true
|
||||
}
|
||||
var other []string
|
||||
for ip := range c.IPs {
|
||||
if !keep[ip] {
|
||||
other = append(other, ip)
|
||||
}
|
||||
}
|
||||
sort.Slice(other, func(i, j int) bool {
|
||||
a, b := when(c.IPs[other[i]]), when(c.IPs[other[j]])
|
||||
if a != b {
|
||||
return a < b
|
||||
}
|
||||
return other[i] < other[j]
|
||||
})
|
||||
for i := 0; i < over && i < len(other); i++ {
|
||||
delete(c.IPs, other[i])
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
return changed
|
||||
}
|
||||
|
||||
// ResolveHome locates the router itself (the arcs' origin), at most every six hours; true when the cache changed.
|
||||
func ResolveHome(get Fetch, provider string, c *Cache, wan string, now int64) bool {
|
||||
h := c.Home
|
||||
if h != nil && h["asn"] != nil {
|
||||
if ts, _ := num(h["ts"]); float64(now)-ts < 6*3600 && (wan == "" || !IsPublic(wan) || h["wan"] == wan) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
// behind NAT, the service is asked for our public address instead
|
||||
ask := ""
|
||||
if IsPublic(wan) {
|
||||
ask = wan
|
||||
}
|
||||
next := Place{}
|
||||
if p := Lookup(get, provider, ask); p != nil {
|
||||
next = p
|
||||
next["wan"] = wan
|
||||
if wan == "" {
|
||||
next["wan"] = nil
|
||||
}
|
||||
} else {
|
||||
for k, v := range h {
|
||||
next[k] = v
|
||||
}
|
||||
}
|
||||
next["ts"] = float64(now)
|
||||
c.Home = next
|
||||
return true
|
||||
}
|
||||
|
||||
// Label is "City · ISP" for the activity log; "" for no place (none known, or a failed lookup).
|
||||
func Label(p Place) string {
|
||||
if p == nil {
|
||||
return ""
|
||||
}
|
||||
if fail, failed := num(p["fail"]); failed && fail != 0 {
|
||||
return ""
|
||||
}
|
||||
var parts []string
|
||||
for _, x := range []string{str(p["city"]), str(p["org"])} {
|
||||
if x != "" {
|
||||
parts = append(parts, x)
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, " · ")
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/tickcount/luci-app-dawnline/backend/usage"
|
||||
)
|
||||
|
||||
const now = 1790000000
|
||||
|
||||
func place() Place {
|
||||
return Place{"lat": 45.04, "lon": 38.98, "city": "Krasnodar", "country": "Russia", "cc": "RU", "asn": 12389.0, "org": "Rostelecom"}
|
||||
}
|
||||
|
||||
func stamped(ts float64) Place { p := place(); p["ts"] = ts; return p }
|
||||
|
||||
// As tests/ucode/geo.test.uc holds the ucode module to.
|
||||
func TestStaleAndMerge(t *testing.T) {
|
||||
c := &Cache{IPs: map[string]Place{"37.78.36.76": stamped(now - 86400), "212.233.85.1": stamped(now - 31*86400), "185.135.150.35": {"fail": float64(now - 3600)},
|
||||
"185.135.150.28": {"fail": float64(now - 2*86400)}, "83.239.111.148": {"lat": 1.0, "lon": 2.0, "city": "Old", "ts": float64(now)}}}
|
||||
for _, k := range []struct {
|
||||
ips []string
|
||||
want string
|
||||
}{{[]string{"37.78.36.76"}, ""}, {[]string{"212.233.85.1"}, "212.233.85.1"}, {[]string{"185.135.150.35"}, ""}, {[]string{"185.135.150.28"}, "185.135.150.28"},
|
||||
{[]string{"83.239.111.148"}, "83.239.111.148"}, {[]string{"10.0.1.164", "100.64.1.1", "", "fe80::1"}, ""}, {[]string{"1.1.1.1", "8.8.8.8", "1.1.1.1"}, "1.1.1.1 8.8.8.8"}} {
|
||||
if got := strings.Join(Stale(c, k.ips, now), " "); got != k.want {
|
||||
t.Errorf("stale of %v: %q, not %q", k.ips, got, k.want)
|
||||
}
|
||||
}
|
||||
c = &Cache{IPs: map[string]Place{}}
|
||||
if !Merge(c, map[string]Place{"1.1.1.1": place(), "8.8.8.8": nil}, nil, now) {
|
||||
t.Error("results changed nothing")
|
||||
}
|
||||
if c.IPs["1.1.1.1"]["ts"] != float64(now) || c.IPs["1.1.1.1"]["city"] != "Krasnodar" || len(c.IPs["8.8.8.8"]) != 1 || c.IPs["8.8.8.8"]["fail"] != float64(now) {
|
||||
t.Errorf("after the merge: %v", c.IPs)
|
||||
}
|
||||
if Merge(c, nil, nil, now) || Merge(c, map[string]Place{"10.0.0.1": place()}, nil, now) || c.IPs["10.0.0.1"] != nil {
|
||||
t.Error("nothing, or a local address, changed the cache")
|
||||
}
|
||||
if Label(c.IPs["1.1.1.1"]) != "Krasnodar · Rostelecom" || Label(c.IPs["8.8.8.8"]) != "" || Label(nil) != "" || Label(Place{"city": "X"}) != "X" {
|
||||
t.Error("the labels")
|
||||
}
|
||||
// bounded: the oldest go, never what is still shown
|
||||
c = &Cache{IPs: map[string]Place{}}
|
||||
for i := 0; i < 500; i++ {
|
||||
c.IPs[fmt.Sprintf("1.0.%d.%d", i/250, i%250+1)] = stamped(float64(now - 1000 + i))
|
||||
}
|
||||
Merge(c, map[string]Place{"9.9.9.9": place(), "9.9.9.10": place()}, []string{"1.0.0.1"}, now)
|
||||
if len(c.IPs) != 500 || c.IPs["1.0.0.1"] == nil || c.IPs["1.0.0.2"] != nil || c.IPs["1.0.0.3"] != nil || c.IPs["1.0.0.4"] == nil || c.IPs["9.9.9.9"] == nil {
|
||||
t.Errorf("the bound: %d entries", len(c.IPs))
|
||||
}
|
||||
for _, ip := range []string{"8.8.8.8", "2a00:1450::1"} {
|
||||
if !IsPublic(ip) {
|
||||
t.Errorf("%s is not public", ip)
|
||||
}
|
||||
}
|
||||
for _, ip := range []string{"", "192.168.1.1", "172.20.0.1", "224.0.0.1", "fe80::1", "fd00::1", "::1", "nonsense"} {
|
||||
if IsPublic(ip) {
|
||||
t.Errorf("%s is public", ip)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookups(t *testing.T) {
|
||||
var asked []string
|
||||
get := func(url string) map[string]any {
|
||||
asked = append(asked, url)
|
||||
switch {
|
||||
case strings.HasPrefix(url, "https://ipwho.is/1.1.1.1?"):
|
||||
return map[string]any{"success": true, "ip": "1.1.1.1", "latitude": 45.04, "longitude": 38.98, "city": "Krasnodar", "country": "Russia", "country_code": "RU",
|
||||
"connection": map[string]any{"asn": 12389.0, "isp": "Rostelecom", "org": "Other"}}
|
||||
case url == "https://ipapi.co/8.8.8.8/json/":
|
||||
return map[string]any{"ip": "8.8.8.8", "latitude": 37.4, "longitude": -122.0, "city": "Mountain View", "country_name": "United States", "country_code": "US", "asn": "AS15169", "org": "GOOGLE"}
|
||||
case url == "https://ipapi.co/json/":
|
||||
return map[string]any{"ip": "203.0.113.7", "latitude": 55.7, "longitude": 37.6, "city": "Moscow", "country_name": "Russia", "country_code": "RU", "asn": "AS1", "org": "ISP"}
|
||||
case strings.HasPrefix(url, "https://ipwho.is/"):
|
||||
return map[string]any{"success": false}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
found := LookupAll(get, "ipwho.is", []string{"1.1.1.1", "8.8.8.8", "4.4.4.4", "5.5.5.5"}, 3, time.Minute)
|
||||
if len(found) != 3 || found["1.1.1.1"]["org"] != "Rostelecom" || found["1.1.1.1"]["ip"] != nil || found["8.8.8.8"]["asn"] != 15169.0 || found["8.8.8.8"]["country"] != "United States" || found["4.4.4.4"] != nil {
|
||||
t.Errorf("the lookups: %v", found)
|
||||
}
|
||||
asked = nil
|
||||
Lookup(get, "ipapi.co", "1.1.1.1")
|
||||
if len(asked) != 2 || !strings.HasPrefix(asked[0], "https://ipapi.co/") {
|
||||
t.Errorf("the chosen provider is not asked first: %v", asked)
|
||||
}
|
||||
// the router's own place: behind NAT the service is asked who we are; kept six hours, or until the address moves
|
||||
c := &Cache{IPs: map[string]Place{}}
|
||||
if !ResolveHome(get, "ipwho.is", c, "192.168.1.2", now) || c.Home["city"] != "Moscow" || c.Home["wan"] != "192.168.1.2" {
|
||||
t.Errorf("home: %v", c.Home)
|
||||
}
|
||||
if ResolveHome(get, "ipwho.is", c, "192.168.1.2", now+3600) || !ResolveHome(get, "ipwho.is", c, "192.168.1.2", now+7*3600) {
|
||||
t.Error("home is asked too often, or never again")
|
||||
}
|
||||
c.Home["wan"] = "1.1.1.1"
|
||||
if ResolveHome(get, "ipwho.is", c, "1.1.1.1", now+7*3600+60) || !ResolveHome(get, "ipwho.is", c, "8.8.8.8", now+7*3600+120) || c.Home["city"] != "Mountain View" {
|
||||
t.Errorf("home after the address moved: %v", c.Home)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocator(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
u := usage.Empty()
|
||||
u.Devices["d1"] = &usage.Record{IPs: map[string]float64{"1.1.1.1": 1, "10.0.0.9": 1}}
|
||||
u.Devices["d2"] = &usage.Record{IPs: map[string]float64{"2.2.2.2": 1, "3.3.3.3": 1, "4.4.4.4": 1}}
|
||||
var mu sync.Mutex
|
||||
asked := 0
|
||||
var where map[string]string
|
||||
var devices []string
|
||||
done := make(chan bool, 8)
|
||||
l := &Locator{Store: Store{Run: dir + "/run", Data: dir + "/data"}, Usage: func() *usage.Usage { return u },
|
||||
Get: func(url string) map[string]any {
|
||||
mu.Lock()
|
||||
asked++
|
||||
mu.Unlock()
|
||||
if strings.Contains(url, "1.1.1.1") {
|
||||
return map[string]any{"success": true, "latitude": 1.0, "longitude": 2.0, "city": "Krasnodar", "connection": map[string]any{"asn": 1.0, "isp": "ISP"}}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Found: func(w map[string]string, d []string) { where, devices = w, d; done <- true }}
|
||||
l.Ask("ipwho.is", []string{"1.1.1.1"}, nil)
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("nothing was found")
|
||||
}
|
||||
if where["1.1.1.1"] != "Krasnodar · ISP" || len(devices) != 1 || devices[0] != "d1" {
|
||||
t.Errorf("found %v for %v", where, devices)
|
||||
}
|
||||
if c := l.Store.Load(); c.IPs["1.1.1.1"]["city"] != "Krasnodar" {
|
||||
t.Errorf("the cache: %v", c.IPs)
|
||||
}
|
||||
// the minute: what is shown and unknown, three a run; what is known is not asked again
|
||||
mu.Lock()
|
||||
asked = 0
|
||||
mu.Unlock()
|
||||
l.Minute(true, "ipwho.is", "")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
c := l.Store.Load()
|
||||
if c.IPs["1.1.1.1"]["city"] != "Krasnodar" || c.IPs["2.2.2.2"]["fail"] == nil || c.IPs["3.3.3.3"]["fail"] == nil || c.IPs["4.4.4.4"]["fail"] == nil || c.IPs["10.0.0.9"] != nil {
|
||||
t.Errorf("after the minute: %v", c.IPs)
|
||||
}
|
||||
if !l.Store.Flush() || l.Store.Flush() {
|
||||
t.Error("the cache is not put on flash once")
|
||||
}
|
||||
l.Minute(false, "ipwho.is", "")
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package geo
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/tickcount/luci-app-dawnline/backend/usage"
|
||||
)
|
||||
|
||||
// A lookup run: this many addresses at most, within this long.
|
||||
const (
|
||||
perRun = 3
|
||||
budget = 12 * time.Second
|
||||
)
|
||||
|
||||
// Locator finds the places of addresses for the map, a few at a time and never in the way of who asks: a
|
||||
// lookup waits on the network for seconds. Addresses that just connected go first; the minute adds
|
||||
// anything still shown that is missing or stale. One run at a time; what did not fit waits for the next.
|
||||
type Locator struct {
|
||||
Store Store
|
||||
Get Fetch
|
||||
// Usage is the counters now: they tell which addresses are still shown, and whose they are.
|
||||
Usage func() *usage.Usage
|
||||
// Found is told the places found ("City · ISP" by address) and the devices seen from those addresses.
|
||||
Found func(where map[string]string, devices []string)
|
||||
|
||||
mu sync.Mutex
|
||||
pending []string
|
||||
running bool
|
||||
lastHome int64
|
||||
}
|
||||
|
||||
func (l *Locator) shown(u *usage.Usage) []string {
|
||||
var ips []string
|
||||
seen := map[string]bool{}
|
||||
if u == nil {
|
||||
return ips
|
||||
}
|
||||
for _, rec := range u.Devices {
|
||||
for ip := range rec.IPs {
|
||||
if !seen[ip] {
|
||||
seen[ip] = true
|
||||
ips = append(ips, ip)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ips
|
||||
}
|
||||
|
||||
// Ask has these addresses located now (provider: the settings' choice), and with them whatever of more is
|
||||
// missing or stale. It returns at once; the run goes on by itself.
|
||||
func (l *Locator) Ask(provider string, ips, more []string) {
|
||||
l.mu.Lock()
|
||||
for _, ip := range ips {
|
||||
known := false
|
||||
for _, q := range l.pending {
|
||||
known = known || q == ip
|
||||
}
|
||||
if !known {
|
||||
l.pending = append(l.pending, ip)
|
||||
}
|
||||
}
|
||||
if l.running {
|
||||
l.mu.Unlock()
|
||||
return
|
||||
}
|
||||
todo := Stale(l.Store.Load(), append(append([]string{}, l.pending...), more...), time.Now().Unix())
|
||||
want := todo
|
||||
if len(want) > perRun {
|
||||
want = want[:perRun]
|
||||
}
|
||||
// what just connected and did not fit waits for the next run
|
||||
var rest []string
|
||||
for _, ip := range todo[len(want):] {
|
||||
for _, q := range l.pending {
|
||||
if q == ip {
|
||||
rest = append(rest, ip)
|
||||
}
|
||||
}
|
||||
}
|
||||
l.pending = rest
|
||||
if len(want) == 0 {
|
||||
l.mu.Unlock()
|
||||
return
|
||||
}
|
||||
l.running = true
|
||||
l.mu.Unlock()
|
||||
go func() {
|
||||
found := LookupAll(l.Get, provider, want, perRun, budget)
|
||||
l.done(found)
|
||||
l.mu.Lock()
|
||||
l.running = false
|
||||
again := len(l.pending) > 0
|
||||
l.mu.Unlock()
|
||||
if again {
|
||||
l.Ask(provider, nil, nil)
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// done puts a run's results into the cache and tells who waits for them.
|
||||
func (l *Locator) done(found map[string]Place) {
|
||||
u := l.Usage()
|
||||
c := l.Store.Load()
|
||||
if !Merge(c, found, l.shown(u), time.Now().Unix()) {
|
||||
return
|
||||
}
|
||||
l.Store.Save(c)
|
||||
where := map[string]string{}
|
||||
for ip, p := range found {
|
||||
if s := Label(p); s != "" {
|
||||
where[ip] = s
|
||||
}
|
||||
}
|
||||
if len(where) == 0 || l.Found == nil {
|
||||
return
|
||||
}
|
||||
var devices []string
|
||||
if u != nil {
|
||||
for id, rec := range u.Devices {
|
||||
for ip := range where {
|
||||
if _, has := rec.IPs[ip]; has {
|
||||
devices = append(devices, id)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
l.Found(where, devices)
|
||||
}
|
||||
|
||||
// Minute is the minute's part: addresses still shown that are missing or stale get a run, and the router's
|
||||
// own place is looked at about hourly (wan: its public address as the router has it, "" when unknown).
|
||||
func (l *Locator) Minute(enabled bool, provider, wan string) {
|
||||
if !enabled {
|
||||
l.mu.Lock()
|
||||
l.pending = nil
|
||||
l.mu.Unlock()
|
||||
return
|
||||
}
|
||||
l.Ask(provider, nil, l.shown(l.Usage()))
|
||||
now := time.Now().Unix()
|
||||
l.mu.Lock()
|
||||
due := now-l.lastHome > 3600
|
||||
if due {
|
||||
l.lastHome = now
|
||||
}
|
||||
l.mu.Unlock()
|
||||
if due {
|
||||
go func() {
|
||||
c := l.Store.Load()
|
||||
if ResolveHome(l.Get, provider, c, wan, time.Now().Unix()) {
|
||||
// the places found meanwhile are the file's: only the router's own is put in
|
||||
fresh := l.Store.Load()
|
||||
fresh.Home = c.Home
|
||||
l.Store.Save(fresh)
|
||||
}
|
||||
}()
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user