Runbook: PBS Secondary Backup Target (500GB SSD)¶

Goal: add a 2nd PBS datastore on a fresh 500GB SSD so the failing Apple HDD (8 pending sectors, trending up) is no longer our only backup target.

Why now: /dev/sdj Apple HDD has 8 pending sectors and rising (was 1 at start of week). Source data (VM 189 homeNas) is on the P410i SAS array, not at risk. But our ONLY backup chain currently lives on the failing drive. C4 off-site backup is deferred to July - no cloud safety net for ~6 weeks.

Plan: keep apple-tank running (any read still works fine, write quality is the failing part), add ssd-tank as 2nd datastore, configure PBS sync job to mirror apple-tank → ssd-tank after each daily backup.

Once ssd-tank is verified and a few days of sync are clean, can repoint PVE storage to ssd-tank as primary and demote apple-tank to sync-source until it truly dies.

Step 0 — Pre-flight (before plugging the SSD in)¶

Confirm 500GB SSD has no data you need: 1. Plug into laptop first (or any safe machine). 2. Check contents - if anything important, back up to homenas SMB share now. 3. Note the make + model + serial (for documentation).

Then disconnect from laptop and bring to the G7.

Step 1 — Connect to G7 + identify¶

Plug into a free USB 3.0 port on the G7 (use a USB 3.0 SATA enclosure if the SSD is bare). USB 3.0 ports on G7 are at the back panel.

From PVE host:

# wait 5s for kernel to enumerate
dmesg | tail -10
lsblk
ls -la /dev/disk/by-id/ | grep -i ssd-or-vendor-name

Expected: new /dev/sdX entry, ~500GB. Note the by-id path - it's stable across reboots (the /dev/sdX letter is not).

Step 2 — Wipe + create partition¶

CAUTION: triple-check the device path before wiping. /dev/sdj is the existing failing apple HDD. New SSD will be /dev/sdk or higher.

# replace sdX with the new SSD's device
DEV=/dev/sdX

# read-only sanity check first
lsblk $DEV
sfdisk -d $DEV 2>&1 || true

# nuke partition table + create fresh GPT
wipefs -a $DEV
sgdisk --zap-all $DEV
sgdisk --new=1:0:0 --typecode=1:8300 --change-name=1:pbs-ssd-tank $DEV

# verify
sfdisk -d $DEV

Step 3 — Format ext4 + label¶

mkfs.ext4 -L ssd-tank -m 0 ${DEV}1
# -L ssd-tank      gives it a friendly label
# -m 0             zero reserved blocks (this isn't a root fs, no need for
#                  emergency space)

# grab the new UUID
blkid ${DEV}1
# something like UUID="abc12345-..."

Note the UUID - we use it for PBS removable-datastore binding.

Step 4 — Pass through to PBS VM 200¶

The G7 is the hypervisor. PBS lives in VM 200. We need to attach the new SSD to VM 200 as a 2nd scsi disk.

# find the new SSD's by-id path
ls -la /dev/disk/by-id/ | grep $(basename $DEV)
# something like usb-Mass_Storage_Device_NEW_SERIAL-0:0 -> ../../sdX

# attach to VM 200 as scsi2 (scsi0=root, scsi1=apple-tank already)
qm set 200 -scsi2 /dev/disk/by-id/usb-Mass_Storage_Device_NEW_SERIAL-0:0,backup=0
# backup=0 because we don't back up the backup target itself

This is hot-attach via QGA (now working after Session 11 install). No VM restart needed. Verify from inside PBS:

ssh -i /root/.ssh/id_ed25519 [email protected] "lsblk && blkid"

Expected: new /dev/sdc (or whatever it gets) showing the ssd-tank ext4 partition.

Step 5 — Create PBS datastore¶

Inside PBS VM (ssh [email protected]):

# create as removable datastore (matches apple-tank pattern, allows physical
# unplug/replug without data loss)
proxmox-backup-manager datastore create ssd-tank ssd-tank \
  --backing-device <UUID-from-Step-3> \
  --reuse-datastore false

# verify
proxmox-backup-manager datastore list
proxmox-backup-manager datastore show ssd-tank

PBS auto-mounts on /mnt/datastore/ssd-tank and initializes the chunk store layout.

Step 6 — Register on PVE¶

From PVE host:

pvesm add pbs pbs-ssd-tank \
  --server 10.0.10.20 \
  --datastore ssd-tank \
  --username root@pam \
  --fingerprint <PBS-cert-fingerprint-same-as-apple-tank> \
  --password <PBS-root-password-from-vaultwarden>

Or do via PVE web UI: Datacenter → Storage → Add → Proxmox Backup Server.

Step 7 — PBS sync job (apple-tank → ssd-tank)¶

This is the meat. PBS supports sync between local datastores via proxmox-backup-manager sync-job.

Inside PBS VM:

proxmox-backup-manager sync-job create apple-to-ssd \
  --store ssd-tank \
  --remote-store apple-tank \
  --schedule "03:00" \
  --remove-vanished false
# schedule "03:00" runs at 03:00 daily, after the 02:00 PVE backup window
# remove-vanished=false keeps a copy on ssd-tank even if apple-tank prunes
# something away (gives extra retention safety)

Verify:

proxmox-backup-manager sync-job list
proxmox-backup-manager sync-job run apple-to-ssd
# the manual run pulls everything currently on apple-tank to ssd-tank
# duration: depends on size. Current apple-tank has ~10GB of dedup chunks,
# so first sync should complete in 5-15 minutes.

Watch progress:

journalctl -u proxmox-backup-proxy.service -f

Step 8 — Verify ssd-tank has clean chunks¶

proxmox-backup-manager garbage-collection start ssd-tank
proxmox-backup-manager garbage-collection list

# integrity check
proxmox-backup-manager verify ssd-tank --include-snapshots

Step 9 — Optional: repoint primary backup target to ssd-tank¶

If you want NEW backups to land on ssd-tank first (with sync going the other way to apple-tank for redundancy):

On PVE, edit the backup job:

# find the daily backup job ID
pvesh get /cluster/backup --output-format json | jq '.[].id'

# edit it
pvesh set /cluster/backup/<JOB-ID> --storage pbs-ssd-tank

Then flip the PBS sync job direction:

proxmox-backup-manager sync-job remove apple-to-ssd
proxmox-backup-manager sync-job create ssd-to-apple \
  --store apple-tank \
  --remote-store ssd-tank \
  --schedule "03:00" \
  --remove-vanished false

This way: - Fresh backups land on healthy SSD first. - Apple HDD gets sync-replicated copy (so it can fail without data loss). - When apple HDD does fail, just remove from PBS, no data lost.

Step 10 — Document + clean up¶

Update CLAUDE.md Section 6 with new datastore. Update homepage-services.yaml if desired. Add SSD make/model/serial to vault note "Hardware - 500GB SSD secondary PBS target".

If you decide Step 9 (repoint primary), update the daily backup job documentation in homelab-tracker.md Phase 3.1.

Rollback plan (if anything breaks)¶

The original apple-tank stays untouched throughout. If ssd-tank setup goes sideways: 1. proxmox-backup-manager datastore remove ssd-tank --keep-job-configs false 2. qm set 200 -delete scsi2 3. Unplug SSD.

Daily backups continue to apple-tank as before. No data lost.

Verification checklist post-setup¶

/var/log/brevo-quota.log shows no email storm from PBS sync alerts
proxmox-backup-manager datastore list shows both apple-tank + ssd-tank
proxmox-backup-manager sync-job list shows the sync job
First manual sync completed successfully
proxmox-backup-manager verify ssd-tank returns OK
PVE web UI Datacenter → Storage shows pbs-ssd-tank green
Tomorrow's 02:00 backup completes + 03:00 sync completes cleanly
SMART attributes on the 500GB SSD captured as baseline: smartctl -A /dev/<ssd-device> > /root/ssd-tank-baseline-$(date +%Y%m%d).txt