# ASQVOX Production — Same-Box Cutover Runbook (fresh DB)

> **Decision (2026-07-04, user):** asqvox.com production runs on the **SAME VPS** (`65.254.80.37`)
> with a **FRESH empty DB**. It **replaces aiandhumn prod in-place** — reuses the `/opt/aiw/prod`
> stack, its loopback ports, and the `aiw-prod-pgdata` volume (wiped). The old aiandhumn data is
> **archived, then abandoned**. **DEV (`aiworkfllow.com`) is untouched.**
>
> This is the same-box variant. For the deep shared detail (env boot-blockers, image-fingerprint
> checks, Retell per-widget wiring, two-pass token specifics) this doc **cross-references
> `deploy/NEW-PROD-ENV-RUNBOOK.md`** rather than repeat it. Where the two disagree on *where it
> runs*, THIS doc wins (that runbook assumed a new dedicated server).

---

## ⚠️ Three hard gates — do not skip

1. **DISK (go/no-go).** The box is at **88%**. Building on a near-full disk has corrupted
   containerd shims **twice** (took down every stack, `handoff.md` 2026-06-13). **Phase 0 decides
   whether same-box is even viable.** If reclaim can't project `/` under ~75% with build headroom,
   **STOP and use a new server** (`NEW-PROD-ENV-RUNBOOK.md` §2) — do not force it.
2. **IRREVERSIBLE DATA LOSS.** Fresh-start abandons every aiandhumn tenant/widget/lead/conversation.
   The **Phase 1 archive is the only safety net** — verify the dump restores *before* wiping the volume.
3. **BUILD DISCIPLINE.** One service at a time, `df -h /` + prune between, **never in parallel with a
   dev build**. This is the shim-crash guard, non-negotiable on this box.

**aiandhumn.com brief outage:** during the asqvox build window, `aiandhumn.com` 301s to a
not-yet-live `asqvox.com`. Acceptable — it's a dying, already-noindexed domain. (Zero-gap variant:
keep aiandhumn marketing on a temporary port until asqvox is up — added complexity, skip unless you
care about a ~20-min blip on the old domain.)

---

## Phase 0 — Disk viability check (GO/NO-GO)

```bash
df -h /                                             # current headroom
docker system df                                    # RECLAIMABLE build cache + images (usually the big win)
sudo du -sh /var/lib/docker                         # docker's total footprint
sudo du -sh /home/* 2>/dev/null | sort -h | tail    # client sites (untouchable, ~39 GB last seen)
docker compose -f /opt/aiw/prod/docker-compose.yml images   # aiandhumn prod images to be freed
```

Reclaim you'll get: aiandhumn prod images + the `aiw-prod-pgdata` volume + build cache
(past incidents freed 5+ GB from build cache alone). **If the projected free space after Phase 3
can't clear ~75% with ~8-10 GB build headroom, do NOT proceed same-box.** Paste me the output and
I'll do the math with you.

---

## Phase 1 — Archive aiandhumn (mandatory; deletion is ahead)

DB name/user come from `/opt/aiw/prod/.env` `DATABASE_URL` (default `aiw_voice` / `aiw_voice_prod`).

```bash
cd /opt/aiw/prod
# DB dump (custom format)
docker compose exec -T postgres pg_dump -Fc -U aiw_voice aiw_voice_prod \
  > /root/aiandhumn-prod-$(date +%F).dump
# storage (uploaded files, KB raw/md)
tar czf /root/aiandhumn-storage-$(date +%F).tgz -C /opt/aiw/prod storage
# VERIFY the dump is readable, then copy BOTH off-box (rclone/scp to S3/other host)
pg_restore --list /root/aiandhumn-prod-*.dump | head
```

**Do not proceed until both archives are off-box AND the `pg_restore --list` succeeds.**

---

## Phase 2 — asqvox DNS + certs + nginx prep (aiandhumn still fully up — no disruption)

- **DNS:** 6 records on `asqvox.com` → `65.254.80.37`: `@`, `www`, `api`, `app`, `widget`, `cafe`.
  **Keep DNS-only / grey-cloud** if using Cloudflare — see the `trustProxy` note below.
- **cPanel/WHM:** create the `asqvox` account (or subdomains under an existing one); trigger **AutoSSL**
  for all 6 hosts. (One-time cPanel setup mirrors `deploy/README.md`.)
- **nginx user-includes (prepare, don't install yet):** copy `deploy/cpanel-nginx/aiandhumn/*`,
  then per file: rename hostnames `aiandhumn.com → asqvox.com`, **DROP the `X-Robots-Tag: noindex`
  blocks** (asqvox must be **indexable** — noindex was aiandhumn-only), keep the www→apex 301, HTTPS
  forcing, and `proxy_buffering off` (co-browse SSE). They proxy to the **same** loopback ports
  asqvox reuses: api `3020`, app `5183`, widget `5184`, marketing `5185`, cafe `5186`.

> **⚠️ `trustProxy` (deploy-critical, `0f3f623`).** The API only trusts `X-Forwarded-For` from
> loopback/private peers. Plain nginx → `127.0.0.1:302x` is fine. **Cloudflare orange-cloud is NOT**
> — `req.ip` becomes the CF edge IP → per-IP abuse caps, geo-IP, audit all break. Keep asqvox
> **DNS-only**, or add CF ranges to the trust list (code change) before enabling the proxy.

---

## Phase 3 — Tear down aiandhumn + reclaim disk

```bash
cd /opt/aiw/prod
docker compose down                        # stops ALL aiandhumn containers; KEEP the volume for now

# aiandhumn.com is now down → make the apex a pure-nginx 301 (final state) and drop the app surfaces:
#   - edit the aiandhumn APEX user-include  → replace the proxy body with:
#         location ~ ^/(?!\.well-known/) { return 301 https://asqvox.com$request_uri; }
#   - DELETE the app. / api. / cafe. / widget.aiandhumn.com user-includes
nginx -t && systemctl reload nginx

# reclaim disk — SAFE prunes (do NOT use `image prune -a` unless DEV is running; it would
# delete DEV's idle images and force a dev rebuild):
docker builder prune -f                    # build cache — usually the biggest reclaim
docker image prune -f                      # dangling only
docker compose down --rmi local            # remove THIS stack's images specifically (optional, deeper)

# Wipe the DB volume — ONLY after the Phase-1 archive is verified off-box (fresh start):
docker volume rm aiw-prod-pgdata

df -h /                                     # confirm well under ~75% before building
```

---

## Phase 4 — Retarget config to asqvox + build (reuses `/opt/aiw/prod`)

**Backend `/opt/aiw/prod/.env`** — keep all secrets; change only the URL/sender keys:
`APP_PUBLIC_BASE_URL=https://api.asqvox.com`, `APP_FRONTEND_URL=https://app.asqvox.com`,
`WIDGET_ALLOWED_ORIGINS=https://app.asqvox.com,https://asqvox.com`, `APP_BASE_URL=https://api.asqvox.com`,
`SMTP_FROM` → an asqvox.com sender (verify the domain at the SMTP provider).
Boot-blockers must be real (see `NEW-PROD-ENV-RUNBOOK.md` §2a): `WIDGET_CHALLENGE_SECRET`,
`COBROWSE_SUB_SECRET`, `PADDLE_API_BASE=https://api.paddle.com`, `AIW_CRON_DISABLED=false`,
`AIW_SRC=/opt/aiw/src`.

**Frontend `.env.production`** (URLs now, tokens are PLACEHOLDERS for pass 1 — real tokens in Phase 6):
`aiw-voice-app`, `aiw-voice-marketing`, `boojee-cafe-app` → asqvox `api./app./widget./` URLs
(exact keys: `NEW-PROD-ENV-RUNBOOK.md` §4a).

**Build one at a time, prune between:**
```bash
cd /opt/aiw/src && git checkout main && git pull origin main && git log -1 --oneline   # expect d3c8ba2 or later
cd /opt/aiw/prod
for s in migrate api widget app marketing cafe; do docker compose build $s && df -h /; done
```
Image-fingerprint checks (stale-image trap) → `NEW-PROD-ENV-RUNBOOK.md` §3 (journal = 30 idx, migrator has orphan-fix).

---

## Phase 5 — Bring up asqvox (fresh DB) + go live

```bash
cd /opt/aiw/prod
docker compose up -d postgres
docker compose up -d migrate && docker compose logs migrate   # 30 pending → [apply] 0000..0029 → complete; ZERO [orphan-fix]
docker compose up -d api
docker compose exec api node dist/scripts/db-status.js         # 30 entries / 30 applied / 0 drift / 0 pending
curl -fsS http://127.0.0.1:3020/health
docker compose up -d widget app marketing cafe                 # pass-1 (placeholder tokens)

# install the asqvox nginx user-includes now that the containers are up
cp <prepared asqvox includes> /etc/nginx/conf.d/users/asqvox/<host>/proxy.conf   # 5 hosts
nginx -t && systemctl reload nginx
```
`asqvox.*` now serves; `aiandhumn.com` already 301s here (Phase 3). `plans` seed rides in migrations
`0004`+`0011`, so pricing works immediately.

---

## Phase 6 — Recreate widgets + Retell + two-pass token re-bake

Fresh DB = everything below is empty and re-created by hand (`NEW-PROD-ENV-RUNBOOK.md` §1.5, §6):
- Sign up the first tenant at `https://app.asqvox.com/signup` (or seed a staff/ops account).
- **7-step wizard** → create widgets (at minimum: marketing orb, SPA demo, Boojee cafe) → note each `wt_…` token.
- **link-agent** each (`POST /api/admin/widgets/:id/link-agent`) → webhook auto-syncs to
  `https://api.asqvox.com/api/webhooks/retell`; paste each per-widget **cobrowse key** +
  **contact_sync_key** + **cafe command key** into the cloned Retell flow templates; **Publish**.
- Paste the **real tokens** into the 3 frontend `.env.production`, then re-bake:
```bash
docker compose build app marketing cafe && docker compose up -d app marketing cafe
```

---

## Phase 7 — Verify (asqvox smoke matrix)

Adapt `NEW-PROD-ENV-RUNBOOK.md` §5 to **asqvox** hostnames:
`api.asqvox.com/health` 200 · apex marketing 200 + orb · `www`→301 · `app.asqvox.com/login` 200 ·
`api.asqvox.com/api/auth/google` 302 · `widget.asqvox.com/aiw-voice-widget.js` 200 · cafe 200 ·
live voice call connects · `db-status` 30/30/0/0 · billing webhooks 503 (KYC) · co-browse nav ·
contact pill → Leads · cafe voice-add fills cart. **Plus:** `curl -sI https://aiandhumn.com/` → 301
to asqvox; then **Google Search Console → aiandhumn property → Change of Address → asqvox.com**.

---

## Phase 8 — Day-2 (finally set these up)

- **Daily `pg_dump -Fc` off-box** (cron + rclone/S3) — the old box never had this.
- **Uptime monitor** on `api.asqvox.com/health` + apex + app (Uptime Kuma or similar).
- **Disk alert at 80%** — the failure mode that forced this whole exercise.
- Keep the aiandhumn archive (Phase 1) in cold storage; do not delete it.

---

## What this does NOT change
- **DEV** (`aiworkfllow.com`) — untouched, keeps running on this box.
- **Client cPanel sites** — untouched.
- **Payload / migrations / Retell wiring specifics** — identical to `NEW-PROD-ENV-RUNBOOK.md`;
  only *where it runs* and the *old-box teardown* differ.