Files
dawnline/backend/github/github.go
T
Aemeath ac90575b0b
Build packages / CI (push) Blocked by required conditions
Build packages / Version (push) Waiting to run
CI / Web (format, types, tests, bundles) (push) Blocked by required conditions
CI / ShellCheck (push) Blocked by required conditions
CI / Backend (Go: vet, tests, a real ubusd and uhttpd) (push) Blocked by required conditions
CI / Package files (push) Blocked by required conditions
Dawnline 0.20.0
- **Backend rewritten in Go.** One daemon, `dawnlined`, replaces the whole ucode backend: the rpcd plugin, the agent, the CLI and the updater. The package no longer depends on ucode.
- **Per-architecture builds.** The release carries the backend for arm64, ARMv7, mips, mipsle and amd64; the router fetches its own after the package is installed and checks its sha256. Dawnline does not run on other architectures (older ARM, 32-bit x86, RISC-V).
- **Faster on the router.** A firewall sync takes 0.06 s instead of 0.9 s; a full apply takes about 300 ms. The portal is served by the daemon itself, without a CGI process per request.
- **Live pages.** The admin and the portal follow a live stream from the router instead of re-fetching the whole state every 30 seconds.
- **Admin redesigned in the portal's style.** Three sections (Overview · Log · Settings); a server is a page of its own with People, Places and Settings; a device opens as a dialog over its person.
- **Changes wait for «Apply».** Every setting, and a person's access, is staged in one dock before it reaches the router.
- **Configuration rollout.** After a change that makes devices fetch their configuration again, the admin shows how many have fetched it and who has not. Every setting shows beforehand (⟳N) how many devices it will touch.
- **Waiting for a decision.** Overview and People put first the people who need one (access ended, limit reached, a device that does not connect), each with its one action a tap away.
- **Bulk actions on people.** «Select» (or a long press on a card) picks several people to extend by a month, pause or resume at once.
- **Undo.** An action done from a card or a request is undone from its toast.
- **Update dialogs redesigned.** An update is one view from its question to its result; Dawnline's own update runs in it like every other, and the page reloads itself once the new version answers. Other versions are a list, each with its notes and Install on a page of its own.
- **Site to site: networks behind a router.** A router connected as a device can have networks behind it (up to eight, /16 to /30): the server routes them and applies the router's access to them. A network that shares addresses with a tunnel, the router's own networks or another site is refused by name.
- **Routers from the portal.** People can add a router and name its networks on their own page; the admin accepts or declines with a reason, or lets them through without asking.
- **Check access and Health know the networks behind routers.**
- **Usage limits reworked.** A monthly limit no longer follows the calendar. A person's month starts with their first traffic and ends a month later; the next one starts only when they connect again, so time away uses up none of it. Before, everyone's limit reset on the 1st.
- Extra data lasts until that person's month ends; «Reset usage» starts the month anew.
- A limit is for a month or for the whole term; the admin and the portal show when it resets.
- Charts that said «this month» show the last 30 days.
- After the update nobody loses what was counted: the month that was running becomes the person's first and ends on the 1st.
- **Access by time of day removed.** The Hours limit is gone from the admin, the portal and the backend; saved hours are dropped.
- **Expiry simplified.** An end date always stops access, with a heads-up three days before; the per-person choice is gone.
- **No false disconnects from sleeping phones.** A phone or a tablet stays online through ten minutes of silence.
- **Mobile Safari fixes.** Pressing a control no longer sends the page to its top; tab tracks that do not fit scroll; switches animate after a press.
2026-11-13 09:43:50 +03:00

423 lines
12 KiB
Go

// Package github is GitHub for the Components tab, as dawnline.github: releases (with each file's sha256),
// one release by tag, and the changes between two tags, each kept ten minutes in the run directory under
// the name the ucode backend keeps its own (gh-<key>.json: whichever asked first, the other reads it, and
// the two share GitHub's hourly allowance).
//
// Trust: the API is only ever asked directly (TLS to api.github.com).
package github
import (
"encoding/json"
"fmt"
"io"
"net/http"
"regexp"
"strings"
"time"
"crypto/sha256"
"encoding/hex"
"github.com/tickcount/luci-app-dawnline/backend/files"
"os"
)
// Failure is why GitHub could not be asked, as a sentence to translate (Msg) and what goes into it (Args).
type Failure struct {
Msg string
Args []any
}
func (f Failure) Error() string { return fmt.Sprintf(f.Msg, f.Args...) }
// Client asks GitHub. Get fetches a URL (status 0: no answer); nil is the real one.
type Client struct {
RunDir string
Get func(url string) (status int, body []byte)
Now func() time.Time
}
func get(url string) (int, []byte) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return 0, nil
}
req.Header.Set("Accept", "application/vnd.github+json")
req.Header.Set("User-Agent", "Dawnline")
res, err := (&http.Client{Timeout: 12 * time.Second}).Do(req)
if err != nil {
return 0, nil
}
defer res.Body.Close()
body, err := io.ReadAll(io.LimitReader(res.Body, 8<<20))
if err != nil {
return 0, nil
}
return res.StatusCode, body
}
func (c *Client) now() int64 {
if c.Now != nil {
return c.Now().Unix()
}
return time.Now().Unix()
}
var rateLimit = regexp.MustCompile(`(?i)rate limit`)
// api is JSON from the API, or nil when GitHub did not answer or has no such thing; a Failure on a rate
// limit or another API error.
func (c *Client) api(url string) (any, error) {
fetch := c.Get
if fetch == nil {
fetch = get
}
status, body := fetch(url)
if status == 0 {
return nil, nil
}
var j any
if json.Unmarshal(body, &j) != nil {
return nil, nil
}
msg := ""
if m, ok := j.(map[string]any); ok {
msg, _ = m["message"].(string)
}
if status == 403 || status == 429 || rateLimit.MatchString(msg) {
return nil, Failure{Msg: "GitHub API rate limit reached: try again in an hour."}
}
if status == 404 {
return nil, nil
}
if status >= 400 {
if msg == "" {
msg = fmt.Sprintf("HTTP %d", status)
}
return nil, Failure{Msg: "GitHub: %s", Args: []any{msg}}
}
return j, nil
}
var unsafe = regexp.MustCompile(`[^A-Za-z0-9_.-]`)
// cached answers from the file while it is fresh, else asks; what could not be asked anew is the old answer.
func (c *Client) cached(key string, ttl int64, fn func() (any, error)) (any, error) {
path := c.RunDir + "/gh-" + unsafe.ReplaceAllString(key, "_") + ".json"
var kept struct {
At float64 `json:"at"`
Data any `json:"data"`
}
has := files.ReadJSON(path, &kept)
if has && float64(c.now())-kept.At < float64(ttl) {
return kept.Data, nil
}
data, err := fn()
if err != nil {
return nil, err
}
if data != nil {
files.WriteJSON(path, map[string]any{"at": c.now(), "data": data}, 0o644)
return data, nil
}
return kept.Data, nil
}
var digest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
// slim is what is kept of a release.
func slim(x map[string]any) map[string]any {
digests, ids := map[string]any{}, map[string]any{}
names := []any{}
list, _ := x["assets"].([]any)
for _, a := range list {
m, _ := a.(map[string]any)
name, _ := m["name"].(string)
names = append(names, m["name"])
if d, _ := m["digest"].(string); digest.MatchString(d) {
digests[name] = d[7:]
}
if id, ok := m["id"].(float64); ok && id > 0 && id == float64(int64(id)) {
ids[name] = id
}
}
or := func(v, dflt any) any {
if v == nil {
return dflt
}
return v
}
body, _ := x["body"].(string)
if len(body) > 12000 {
body = body[:12000]
}
pre, _ := x["prerelease"].(bool)
return map[string]any{
"tag": x["tag_name"], "name": or(x["name"], x["tag_name"]), "published": x["published_at"], "prerelease": pre, "url": x["html_url"],
"body": body, "assets": names, "digests": digests, "ids": ids,
}
}
// Releases is a repository's releases, newest first, five a page (drafts skipped).
func (c *Client) Releases(repo string, page int) ([]any, error) {
data, err := c.cached(fmt.Sprintf("rel-%s-%d", repo, page), 600, func() (any, error) {
j, err := c.api(fmt.Sprintf("https://api.github.com/repos/%s/releases?per_page=5&page=%d", repo, page))
list, ok := j.([]any)
if err != nil || !ok {
return nil, err
}
out := []any{}
for _, x := range list {
if m, ok := x.(map[string]any); ok && m["draft"] != true {
out = append(out, slim(m))
}
}
return out, nil
})
list, _ := data.([]any)
if list == nil {
list = []any{}
}
return list, err
}
// Release is one release by tag (its notes, its files and their sha256), or nil.
func (c *Client) Release(repo, tag string) (map[string]any, error) {
data, err := c.cached("tag-"+repo+"-"+tag, 600, func() (any, error) {
j, err := c.api("https://api.github.com/repos/" + repo + "/releases/tags/" + tag)
if m, ok := j.(map[string]any); ok && m["tag_name"] != nil && m["tag_name"] != "" {
return slim(m), err
}
return nil, err
})
m, _ := data.(map[string]any)
return m, err
}
var (
point = regexp.MustCompile(`^\s*[-*]\s+(.*)$`)
wrapped = regexp.MustCompile(`^\s+\S`)
version = regexp.MustCompile(`^(Dawnline )?v?[0-9]+\.[0-9]+(\.[0-9]+)?$`)
)
const space = " \t\r\n\v\f"
// Points is what changed in a commit message: a release commit lists its changes as "- " lines in the
// body, any other commit is its title. Wrapped points are joined back into one line.
func Points(msg string) []string {
lines := strings.Split(msg, "\n")
title := strings.Trim(lines[0], space)
out := []string{}
if strings.HasPrefix(title, "Merge ") {
return out
}
cur, open := "", false
for _, line := range lines[1:] {
if m := point.FindStringSubmatch(line); m != nil {
if open {
out = append(out, cur)
}
cur, open = strings.Trim(m[1], space), true
} else if open && wrapped.MatchString(line) {
cur += " " + strings.Trim(line, space)
} else if open {
out = append(out, cur)
open = false
}
}
if open {
out = append(out, cur)
}
if len(out) == 0 && title != "" && !version.MatchString(title) {
out = append(out, title)
}
return out
}
func message(commit any) string {
m, _ := commit.(map[string]any)
c, _ := m["commit"].(map[string]any)
s, _ := c["message"].(string)
return s
}
// Changes is what changed from one tag to another, newest first, twenty points at most: commit by commit,
// or the release's own commit when there is no base to compare with.
func (c *Client) Changes(repo, base, head string) ([]any, error) {
key := base
if base == "" {
key = "null"
}
data, err := c.cached("changes-"+repo+"-"+key+"-"+head, 600, func() (any, error) {
cut := func(points []any) any {
if len(points) > 20 {
points = points[:20]
}
return points
}
if base != "" {
j, err := c.api("https://api.github.com/repos/" + repo + "/compare/" + base + "..." + head)
if err != nil {
return nil, err
}
m, _ := j.(map[string]any)
if commits, ok := m["commits"].([]any); ok {
out := []any{}
for i := len(commits) - 1; i >= 0; i-- {
for _, p := range Points(message(commits[i])) {
out = append(out, p)
}
}
return cut(out), nil
}
}
j, err := c.api("https://api.github.com/repos/" + repo + "/commits/" + head)
if err != nil {
return nil, err
}
if m, _ := j.(map[string]any); m["commit"] != nil {
out := []any{}
for _, p := range Points(message(j)) {
out = append(out, p)
}
return cut(out), nil
}
return nil, nil
})
list, _ := data.([]any)
if list == nil {
list = []any{}
}
return list, err
}
// Mirrors are prefixes that fetch github.com files when GitHub itself is unreachable. A file may come
// through one only when its sha256 is known from the API, and is kept only if it matches.
var Mirrors = []string{"https://ghfast.top/", "https://gh-proxy.com/", "https://ghproxy.net/"}
// Latest is a repository's latest stable release, or nil when GitHub does not answer.
func (c *Client) Latest(repo string) map[string]any {
data, _ := c.cached("latest-"+repo, 300, func() (any, error) {
j, err := c.api("https://api.github.com/repos/" + repo + "/releases/latest")
if m, ok := j.(map[string]any); ok && m["tag_name"] != nil && m["tag_name"] != "" {
return slim(m), err
}
return nil, err
})
m, _ := data.(map[string]any)
return m
}
// AssetSHA256 is the sha256 GitHub publishes for a release's file, or "".
func (c *Client) AssetSHA256(repo, tag, name string) string {
r, _ := c.Release(repo, tag)
digests, _ := r["digests"].(map[string]any)
s, _ := digests[name].(string)
return s
}
// Assets is the names of a release's files (nil: no such release, or GitHub does not answer).
func (c *Client) Assets(repo, tag string) []string {
r, _ := c.Release(repo, tag)
if r == nil {
return nil
}
list, _ := r["assets"].([]any)
out := []string{}
for _, a := range list {
if s, is := a.(string); is {
out = append(out, s)
}
}
return out
}
// Fetch downloads a URL into a file; false when it did not come whole.
type Fetch func(url, dest string) bool
// HTTPFetch is Fetch over the network: five minutes at most, redirects followed.
func HTTPFetch(url, dest string) bool {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return false
}
req.Header.Set("User-Agent", "Dawnline")
res, err := (&http.Client{Timeout: 300 * time.Second}).Do(req)
if err != nil {
return false
}
defer res.Body.Close()
if res.StatusCode != 200 {
return false
}
f, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return false
}
n, err := io.Copy(f, res.Body)
if cerr := f.Close(); err != nil || cerr != nil || n == 0 {
os.Remove(dest)
return false
}
return true
}
// SHA256 is a file's sha256 in hex, or "".
func SHA256(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return ""
}
return hex.EncodeToString(h.Sum(nil))
}
var fromGitHub = regexp.MustCompile(`^https://(github\.com|raw\.githubusercontent\.com)/`)
// Download gets a file: GitHub itself first; a mirror only when the sha256 is known (from the API) and
// mirrors are allowed. A file that does not match the sha256 is thrown away and the next source is tried.
// It returns the URL that worked, or "". log is told of mirrors (words in t's language; nil: English).
func Download(fetch Fetch, url, dest, want string, mirrors bool, log func(string), t func(string) string) string {
if t == nil {
t = func(s string) string { return s }
}
say := func(format string, args ...any) {
if log != nil {
log(fmt.Sprintf(t(format), args...))
}
}
want = strings.ToLower(want)
type try struct {
url string
mirror bool
}
tries := []try{{url, false}}
if want != "" && mirrors && fromGitHub.MatchString(url) {
for _, m := range Mirrors {
tries = append(tries, try{m + url, true})
}
}
for i, x := range tries {
host := strings.SplitN(strings.TrimPrefix(x.url, "https://"), "/", 2)[0]
if x.mirror {
say("Trying mirror %s", host)
}
if fetch(x.url, dest) {
if st, err := os.Stat(dest); err == nil && st.Size() > 0 {
if want == "" || SHA256(dest) == want {
return x.url
}
say("Warning: %s sent a file that does not match GitHub's sha256, discarded", host)
}
}
os.Remove(dest)
if i == 0 && len(tries) > 1 {
say("GitHub did not answer, trying mirrors")
}
}
return ""
}