libertytoolsio/README.md

# Liberty Tools (libertytools.io) — Hugo static site

A Hugo site built on the [hugo-book](https://github.com/alex-shpak/hugo-book) theme,
with **Pagefind** client-side search and **lychee**-based dead-link checking. This
document is the single reference for understanding the architecture, building,
previewing, searching, and maintaining the site.

> If you only need to build and preview, jump to
> [Build & preview (§4)](#4-build--preview-make-commands). Everything you normally
> type is a `make` target.

---

## Table of contents

- [Architecture](#architecture)
- [1. Clone](#1-clone)
- [2. Prerequisites (install once)](#2-prerequisites-install-once)
- [3. Configuration](#3-configuration)
- [4. Build & preview (make commands)](#4-build--preview-make-commands)
- [5. Search (Pagefind)](#5-search-pagefind)
- [6. Dead links](#6-dead-links)
- [7. Troubleshooting](#7-troubleshooting)
- [8. Do not commit](#8-do-not-commit)
- [Repository layout](#repository-layout)

---

## Architecture

The site is a **fully static** output: no server-side rendering at request time, no
database, no runtime. Hugo renders the markdown to plain HTML files *at build time*;
a search index is generated from those files; and a trivial static file server
(delivering the built directory) is all that runs in front of a browser.

### The three moving parts

| Component    | Role                                                                                  |
|--------------|---------------------------------------------------------------------------------------|
| **Hugo**     | The static site generator. Reads `content/*.md` + the theme, emits `public/`.         |
| **hugo-book** | The theme (vendored under `themes/hugo-book/`). Layouts, SCSS styling, menu/TOC.      |
| **Pagefind**  | Client-side full-text search. Indexes the built `public/` into `public/pagefind/`.    |
| **lychee**    | Dead-link checker (used only by the `link-check`/`link-fix`/`external-check` targets).|

Only Hugo and Node (for Pagefind's `npx` wrapper) are required to build. `lychee` is
needed only if you run the dead-link targets.

### Content → build → serve, data flow

```
        ┌──────────────────────────────────────────────┐
        │                    content/                  │
        │        *.md pages + images (the source)      │
        └──────────────────────┬───────────────────────┘
                               │  make build   (hugo --gc)
                               ▼
        ┌──────────────────────────────────────────────┐
        │                   public/                    │
        │  rendered static HTML + fingerprinted CSS/JS │
        │  (hugo-book SCSS compiled by the extended    │
        │   build; each asset hash-named + SRI-checked)│
        └──────────────────────┬───────────────────────┘
                               │  make index   (pagefind --site public)
                               ▼
        ┌──────────────────────────────────────────────┐
        │              public/pagefind/                │
        │   Pagefind search index (WASM + JSON)        │
        │   — ships inside public/, so any static      │
        │     server can serve a working search        │
        └──────────────────────┬───────────────────────┘
                               │  make serve / make serve-lan
                               ▼
                     browser (any machine on the network)
```

Two consequences worth internalizing:

1. **`public/` is the single source of truth for what is served.** Both the search
   index and every preview server read from it. If the site "isn't showing your
   change," the build hasn't run — re-run `make index`.
2. **The Pagefind bundle lives inside `public/`.** That's why `make serve-lan` (a plain
   Python static server) serves working search with zero extra setup — there is no
   special search server required.

### The theme SCSS pipeline

`html-head.html` compiles the theme's stylesheet on every build:

```
book.scss  →  resources.ExecuteAsTemplate  →  css.Sass  →  resources.Minify  →  resources.Fingerprint
```

`css.Sass` is why the **Hugo *extended*** edition is mandatory (the standard build has
no SCSS compiler). `resources.Fingerprint` hashes the output into the filename
(`book.min.<sha>.css`) and embeds an SRI `integrity` attribute on the `<link>`; a
single byte change in the CSS produces a new filename, which is why old cached
stylesheets never collide with new ones.

### Where your customizations live

Two files hold all site-level customization — you rarely need to touch anything else:

- **`themes/hugo-book/layouts/_partials/docs/search.html`** — the entire search UI:
  the Pagefind `new PagefindUI({...})` config, the `#search`-scoped `<style>` block
  that fits the UI to the sidebar, and the keyboard-navigation script. (See §5.)
- **`themes/hugo-book/assets/_variables.scss`** — site-wide dimensions and font.
  This file is imported **after** the theme's `_defaults.scss`, so the values here
  *override* the theme defaults with plain (non-`!default`) assignments.

  | Variable                  | Current value | Effect                                              |
  |---------------------------|---------------|-----------------------------------------------------|
  | `:root { --font-size }`   | `17px`        | Site-wide base font (theme default is 16px). All `rem` sizes scale off this. |
  | `$container-max-width`    | `100rem`      | Max width of the centered content area (~1700px at 17px root). |
  | `$menu-width`             | `18.75rem`    | Width of the **left sidebar** — the category list *and* the search box *and* the search-results pane are all this one column. |
  | `$mobile-breakpoint`      | *(recomputed)*| **Must be recomputed whenever you change `$menu-width`/`$body-min-width`/`$toc-width`** — it is derived as `$menu-width + $body-min-width * 1.2 + $toc-width`. |

### Search index scoping (`data-pagefind-body`)

By default Pagefind indexes the *entire* `<body>` of each page — including the
sidebar, menu, and footer — which pollutes results. Instead, the article element in
each layout is tagged with `data-pagefind-body`:

```html
<article class="markdown book-article" data-pagefind-body>
```

Pagefind then indexes **only** that element and *skips any page that lacks the
attribute*. The attribute is present in `layouts/baseof.html` (default single + home),
`layouts/term.html` (list pages), and `layouts/posts/{single,list}.html`.
`404.html` is intentionally left untagged (it has its own standalone `<html>`).
**Do not remove these attributes** or the menu text will start appearing in search
results. After any change, confirm the indexed page count in the `make index` log —
a sharp drop means a page lost its tag (usually the home page).

### The `#search` CSS cascade trick

`pagefind-ui.css` is loaded in `<body>` (by the search partial) — **after** the theme's
head stylesheet. A CSS rule in the head at equal specificity therefore *loses* the
cascade. All search overrides live in a `#search`-scoped `<style>` block in
`search.html`; the `#search` id + class gives them higher specificity than
Pagefind's class-only rules regardless of load order. Keep this pattern when you tune
the search UI.

### Pagefind 1.5.2 quirks we work around (all in `search.html`)

These are non-obvious behaviors of the Default UI in Pagefind **1.5.2** (the version
`npx -y pagefind` resolves to). They cost real debugging time; they are documented
here so the next person doesn't rediscover them:

- **The results drawer is an in-flow child of the form.** Once results render,
  `.pagefind-ui__form` stretches to the full height of the results list (measured
  ~1800px for a dozen rows). The fixed-height sidebar (`.book-menu-content`,
  `position:fixed; overflow-y:auto`) simply scrolls it — that's *intended*. Consequence:
  any `%`-based height on the absolutely-positioned clear button would resolve against
  that stretched form, so the clear button uses **fixed** pixel dimensions keyed to the
  input.
- **There is no keyboard navigation, and stock Enter is a silent no-op.** The built
  `pagefind-ui.js` handles `Enter` with only `preventDefault()` — typing a query and
  pressing Enter navigates nowhere. A custom **capture-phase** `keydown` listener on
  `#search` implements ↑/↓ highlighting and Enter-to-open (see §5).
- **An empty thumbnail occupies the left of every row.** Even with zero images in the
  index, Pagefind renders an ~81px `.pagefind-ui__result-thumb` placeholder, which
  pushes result text ~90px right (a "right-justified" look). We hide it
  (`display:none`) because this index has no images.
- **Matched terms render in harsh UA-yellow.** Pagefind's CSS reset does
  `mark { all: revert }`, so `<mark>` falls back to the browser default (yellow). We
  restyle `#search .pagefind-ui__result mark` to a subdued slate.

### Dead-link automation

Two independent checks, deliberately split by scope and cost:

- **Internal** (`make link-check` / `make link-fix`): *offline*. `lychee` checks the
  source markdown against the built site (`--root-dir public`, `--scheme file`), so no
  network is involved. Confirmed-dead *internal page* links are auto-commented out by
  `scripts/fix_dead_links.py`; external/bare-domain/empty links are routed to a manual
  review file, never auto-edited.
- **External** (`make external-check`): *network*. `scripts/check_external_links.py`
  pings every `http(s)` link in `content/`. Report-only, always exits 0. A daily cron
  job runs it (see §6).

The removal mechanism for internal links is **commenting, not deleting**
(`[Old](/gone/)` → `<!-- [Old](/gone/) -->`). An HTML comment survives the Hugo build
verbatim, is invisible on the rendered page, is excluded from the search index, and is
no longer detected by `lychee` — which makes `make link-fix` **idempotent** (safe to
re-run; it never loops).

---

## 1. Clone

```
git clone https://dev.libertytools.io/adminuser/libertytoolsio.git
cd libertytoolsio
```

## 2. Prerequisites (install once)

All tools install into `~/.local/bin` (no sudo). Make sure `~/.local/bin` is on your
`PATH` (it usually is; if not: `export PATH="$HOME/.local/bin:$PATH"`).

- **Hugo extended (v0.167.0 or newer).** The theme compiles SCSS with `css.Sass`, so
  the *extended* build is required. From the repo root:
  ```
  cd "$(mktemp -d)"
  curl -sL -o h.tgz https://github.com/gohugoio/hugo/releases/download/v0.167.0/hugo_extended_0.167.0_linux-amd64.tar.gz
  echo "0163f5c3deddac1f494a1629ddc40c65d18de9d5794facd98f7f96ac2c7d8957  h.tgz" | sha256sum -c -
  tar xzf h.tgz
  install -m 0755 hugo "$HOME/.local/bin/hugo"
  hugo version        # must print ...+extended...
  ```
- **Node (v18+).** Pagefind runs through `npx -y pagefind`, which auto-installs the
  Pagefind CLI the first time (resolves to **1.5.x**; the UI quirks in
  [Architecture](#pagefind-152-quirks-we-work-around-all-in-searchhtml) are for 1.5.2).
  Verify: `node --version && npx --version`.
- **lychee (v0.24.2)** — link checker, needed only for the dead-link targets:
  ```
  cd "$(mktemp -d)"
  curl -sL -o ly.tgz https://github.com/lycheeverse/lychee/releases/download/lychee-v0.24.2/lychee-x86_64-unknown-linux-musl.tar.gz
  tar xzf ly.tgz
  install -m 0755 lychee-x86_64-unknown-linux-musl/lychee "$HOME/.local/bin/lychee"
  lychee --version    # lychee 0.24.2
  ```

## 3. Configuration

- **`config.toml`** — site-level settings: `theme = "hugo-book"`, `BookSearch = true`,
  `BookTheme = 'dark'`, `plausibleDomain`, and `markup.goldmark.renderer.unsafe = true`
  (required so raw HTML — including the commented-out dead links and inline `<style>`
  blocks — passes through the markdown renderer unharmed).
- **`themes/hugo-book/assets/_variables.scss`** — site dimensions + font (see the
  table in [Architecture](#where-your-customizations-live)). This is the file to edit
  to change the base font size, page width, or sidebar width.
- **`themes/hugo-book/layouts/_partials/docs/search.html`** — the search UI (see §5).

## 4. Build & preview (make commands)

Run from the project root. `public/`, `linkcheck/`, and `resources/_gen/` are
generated and gitignored.

| Target                | What it does |
|-----------------------|--------------|
| `make build`          | Build Hugo content into `public/` (`hugo --gc`). |
| `make index`          | `build` + build the Pagefind search index into `public/pagefind/`. **This is the normal "I edited content, re-make it" command.** |
| `make serve` / `make run` | `index` + serve the built site **with search** at `http://localhost:1414/` (local only). Uses Pagefind's own server, which binds 127.0.0.1. |
| `make serve-lan`      | Serve the built site on **all interfaces** so another machine on the network can open it: `http://<LAN-IP>:8090/`. Python stdlib only; build first with `make index`. |
| `make link-check`     | **Offline** scan of *internal* links in `content/` → `linkcheck/internal.json`. Fast, no network. |
| `make link-fix`       | Comment out confirmed dead *internal* links, then rebuild content + index. |
| `make external-check` | **Network** scan of all *external* links in `content/` → console report + `linkcheck/external_dead.txt`. Report-only; always exits 0. |
| `make clean`          | Remove `public/` and `linkcheck/`. |
| `make all`            | `build` + `index` (the default target). |

Notes on the targets:

- **`build` runs `hugo --gc`** — `--gc` garbage-collects orphaned resources from the
  asset cache.
- **`serve` / `serve-lan` do not rebuild.** They serve whatever is already in
  `public/`. Always run `make index` first (or use `make serve`, which chains `index`),
  or you'll be serving a stale build.
- **Why `serve` uses `pagefind --serve`, not `hugo server`:** `hugo server` renders in
  memory and may not write `public/`, leaving Pagefind with nothing to index.
  Serving the *built* `public/` through Pagefind's own static server is what makes the
  preview match production.
- **`serve-lan` is a plain static file server** (`scripts/serve_lan.py`, stdlib only)
  rooted at `public/` on `0.0.0.0:8090`. It exists specifically because `make serve`
  binds to 127.0.0.1 and is unreachable from other machines.

### Viewing changes locally

```
make serve
```
Open `http://localhost:1414/`. There is no live reload — re-run `make serve` after
editing content.

### Viewing from another machine on the same network

```
make index        # build first so public/pagefind/ exists
make serve-lan
```
`serve-lan` prints this machine's LAN IP at startup (also: `hostname -I | head -1`).
On the **other** machine, open `http://<that-printed-IP>:8090/`. The IP is dynamic
(DHCP); re-check it if it changes. Search works identically under this plain static
server as under `make serve` (the Pagefind bundle is part of `public/`, so no special
server is needed for cross-machine access).

## 5. Search (Pagefind)

The search box lives in the **left sidebar** on every page (injected via
`themes/hugo-book/layouts/_partials/docs/menu.html` → `docs/search.html`).

### Using it

- Press **`/`** anywhere on a page to jump to the search box (`focusOnSlash`).
- Type a query; results appear below the box, scoped to page content (see
  `data-pagefind-body` in [Architecture](#search-index-scoping-data-pagefind-body)).
- **Navigate with the keyboard:**
  - **`↓` / `↑`** move a highlight through the results, in visual order. Top-level
    hits *and* their sub-topics (Pagefind "sub-results") each get their own highlight —
    e.g. a query under "Hacking" can step through Hacking, then its Anonymity / OSINT /
    Security Courses sub-entries, one at a time. Wraps around at the ends.
  - **`Enter`** opens the highlighted result. If nothing is highlighted, it opens the
    top result. (Stock Pagefind does neither — this is the custom listener described in
    [Architecture](#pagefind-152-quirks-we-work-around-all-in-searchhtml).)
- The matched terms inside each result are shown in a subdued slate (not the browser's
  default yellow).

### Customizing the look

Everything is in the `#search`-scoped `<style>` block and the `new PagefindUI({...})`
call in `themes/hugo-book/layouts/_partials/docs/search.html`. **Always keep the
`#search` prefix** on overrides — see the [cascade trick](#the-search-css-cascade-trick).
Current values (verified against the built page):

| Element                              | Current value |
|--------------------------------------|---------------|
| Input font / height / padding        | `font-size: 0.85em; height: 2.4em; min-height: 2.4em; padding: 0 2.6em 0 2.4em;` |
| Magnifier icon (`.pagefind-ui__form:before`) | `top: 7px; left: 12px; width/height: 14px;` — fades to `opacity: 0` while the input has text |
| Clear button (`.pagefind-ui__search-clear`) | `top: 3px; right: 3px; height: 23px; font-size: 11px; padding: 0 6px;` (fixed px, *not* `%` — see form-stretch quirk) |
| Results layout (`.pagefind-ui__drawer`) | `flex-direction: column; gap: 0.4em;` (stack, don't wrap) |
| Shrink guards (`.results-area`, `.result-excerpt`) | `min-width: 0;` |
| Empty thumbnail (`.result-thumb`)     | `display: none;` (no images in this index) |
| Result title / link                   | `font-size: 1em; overflow-wrap: anywhere;` |
| Result excerpt                        | `font-size: 0.88em; overflow-wrap: anywhere; word-break: break-word;` |
| "N results for …" message             | `font-size: 0.8em;` |
| Keyboard-selected row (`.pf-kbd-selected`) | `background: rgba(100,116,139,0.5); outline: 2px solid #cbd5e1; outline-offset: -2px; color: #fff;` |
| Matched term (`mark`)                 | `background: rgba(100,116,139,0.45); color: #fff; font-weight: 600; border-radius: 2px;` |
| Excerpt length (words)                 | `excerptLength: 20` in the `new PagefindUI({...})` call |

The legacy Fuse.js search was **removed** (it double-loaded with Pagefind). A backup of
the old head partial is at `themes/hugo-book/layouts/_partials/docs/html-head.html.bkup`.

## 6. Dead links

### Internal links (fast, offline)

```
make link-check     # writes linkcheck/internal.json
make link-fix       # comments out confirmed dead internal page links, then rebuilds
```

How it works: `lychee` checks the source markdown against the built site
(`--root-dir public`). A link whose target page does not exist is a *dead internal
link* and `scripts/fix_dead_links.py` wraps it in an HTML comment:
`[Old](/gone/)` → `<!-- [Old](/gone/) -->`. A commented link disappears from both the
rendered page and the search index, and is **not** re-detected, so `make link-fix` is
idempotent and safe to run repeatedly.

**Safety rule:** only *internal page* links are auto-commented. External `https://`
failures, bare-domain links, and empty `href`s are written to
`linkcheck/manual_review.txt` and left for you to fix by hand (a bare domain may be a
typo to fix, not a link to delete).

### External links (network, daily)

```
make external-check    # pings every http/https link in content; report-only
```

It lists each dead external link with its `file:line` and writes
`linkcheck/external_dead.txt`. It does **not** edit anything. A scheduled job runs this
once a day (below).

### The daily external-link job

A cron job named **`libertytools-external-link-check`** runs
`scripts/check_external_links.py` **every day at 6:00am**. It is report-only and always
exits 0. Its output is saved and viewable with:

```
hermes cronjob list            # confirm it exists and is enabled
```
- To run it manually right now: `hermes cronjob run <job_id>` (from `hermes cronjob list`).
- To pause it: `hermes cronjob pause <job_id>`; to resume: `hermes cronjob resume <job_id>`.
- To change the time, edit the job's schedule in `config.yaml` (or ask the agent).

## 7. Troubleshooting

- **`hugo: command not found`** → you're using the non-extended build or it isn't on
  PATH. Re-run the Hugo install above and confirm `hugo version` says `+extended`.
- **Search shows no results** → the index wasn't built. Run `make index`, confirm the
  log prints `Indexed N pages` and `public/pagefind/pagefind-ui.js` exists, then reload.
  If the page count looks low, a layout lost its `data-pagefind-body` attribute.
- **`pagefind` / `lychee` not found** → re-run the relevant install in §2; both live in
  `~/.local/bin`.
- **`make serve` says port 1414 in use** → a previous server is still running. Stop it
  (Ctrl+C in its terminal) or find it: `ss -ltnp | grep 1414`.
- **Other machine can't reach `http://<LAN-IP>:8090/`** → confirm you're on the same
  network and using the IP `serve-lan` printed (`hostname -I | head -1` on the host).
  The host's firewall may need to allow inbound 8090.
- **Search is too big/small for the sidebar** → tune the `#search` block in
  `themes/hugo-book/layouts/_partials/docs/search.html` (see §5).
- **Search results look right-justified / have a big left gap** → the empty thumbnail
  isn't hidden; ensure `#search .pagefind-ui__result-thumb { display: none; }` is
  present (see the quirk in [Architecture](#pagefind-152-quirks-we-work-around-all-in-searchhtml)).
- **Up/down arrows don't highlight results** → the keyboard-nav script in
  `search.html` is missing or its `rowFor()` no longer matches the result DOM. Sub-results
  are nested inside the top-level `<li>` (see the quirk notes).

## 8. Do not commit

All changes from a maintenance pass are left in the working tree for review. Commit
**only after you have validated the result** (`make build` exits 0 and you've checked
the rendered output). The following are generated and gitignored — never commit them:

- `public/` — built site output
- `linkcheck/` — link-check reports
- `resources/_gen/` — Hugo asset-generation cache
- `.hermes/` — agent working files

## Repository layout

```
content/            # site pages (markdown + images)
themes/hugo-book/   # the theme (vendored). Do not edit themes/hugo-book.backup/.
  assets/_variables.scss                       # SITE overrides: font, page width, sidebar width, breakpoint
  layouts/baseof.html                          # root layout; <article data-pagefind-body>
  layouts/term.html                            # list/term pages (data-pagefind-body)
  layouts/posts/{single,list}.html             # blog post pages (data-pagefind-body)
  layouts/_partials/docs/search.html           # Pagefind UI + sidebar-fit CSS + keyboard nav
  layouts/_partials/docs/menu.html             # injects search.html into the sidebar
  layouts/_partials/docs/html-head.html        # (legacy fuse search removed; .bkup kept)
  layouts/_partials/docs/html-head.html.bkup   # backup of the pre-Pagefind head
scripts/
  fix_dead_links.py        # internal dead-link comment-out (idempotent)
  check_external_links.py  # external dead-link report (report-only, daily cron)
  serve_lan.py             # static LAN preview server (serves public/ on 0.0.0.0:8090)
Makefile              # build / index / serve / run / serve-lan / link-check / link-fix / external-check / clean
config.toml           # theme + BookSearch + BookTheme + goldmark.unsafe
```