Operations — scripts / datasets / deploy

§1Environment

Activate env

module load conda/latest
eval "$(conda shell.bash hook)"
conda activate hang
# Don't use `source activate` — Unity's conda/latest is miniforge3 and only `conda activate` works.

§2SLURM policy

Useful commands

# check 2080Ti availability
sinfo -p gpu-preempt -o "%t %D %G" | grep 2080_ti

# running jobs
squeue -u $USER -o "%.18i %.9P %.20j %.8t %.10M %R"

# interactive debug session
srun --partition=gpu-preempt --gres=gpu:2080_ti:1 \
     --mem=16G --time=1:00:00 --pty bash

# inspect a finished job
sacct -j <JOBID> --format=JobID,State,ExitCode,Elapsed

# disk usage
du -sh /work/pi_nwycoff_umass_edu/hang/imagehide/*
df -h /work/pi_nwycoff_umass_edu/hang

§3Scripts filter by kind or search

Run the smoke test (first-time setup check)

cd /work/pi_nwycoff_umass_edu/hang/imagehide
sbatch scripts/smoke_test.sbatch
squeue -u $USER                              # check progress
tail -f logs/smoke-<JOBID>.out                # follow output

Logs land at logs/<name>-<JOBID>.{out,err}.

§4Datasets

All under /work/pi_nwycoff_umass_edu/hang/imagehide/data/. ~143K images, ~26 GB on disk after extraction cleanup (~84 GB including kept DPED archive).

Per-dataset notes

§5Manifests

data/train.txt and data/val.txt list absolute image paths, one per line, sorted. build_resolution_manifests.py filters per-resolution-class for the multi-res sampler.

Sidecar {train,val}_dims.tsv (path \t w \t h) is also written for arbitrary downstream filtering.

Multi-res dataloader

src/multi_res_dataset.py: MultiResDataset + MultiResBatchSampler + make_loader. Reads train_dims.tsv and materializes per-(H, W) eligibility lists. RES_POOL defines 18 (H, W) tuples; the sampler picks one per batch and derives batch size from a pixel budget (default 1.0 M). src/exp0_inn_train.py consumes via make_loader(args.manifest_dims, ...).

§6Site deploy

The public dashboard at yanghangai.github.io/imagehide is served by GitHub Pages from the gh-pages branch — not main. A commit to main won't update the live site.

Deploy steps

# 1. Commit + push source edits on main
git add index.html docs/html/ assets/ site/ docs/site_deploy.md
git commit -m "Site: <summary>"
git push origin main

# 2. Mirror to gh-pages via a worktree
git fetch origin gh-pages
git worktree add /tmp/imagehide-ghpages gh-pages
cd /tmp/imagehide-ghpages

# 2a. Dashboard + spec/design/plan/ops
cp /work/pi_nwycoff_umass_edu/hang/imagehide/index.html index.html
mkdir -p docs/html
cp /work/pi_nwycoff_umass_edu/hang/imagehide/docs/html/*.html docs/html/

# 2b. Feedback overlay assets
mkdir -p assets
cp /work/pi_nwycoff_umass_edu/hang/imagehide/assets/feedback.{css,js} assets/

# 2c. Paper site at /paper/ (only when site/ source changes)
mkdir -p paper/img
cp /work/pi_nwycoff_umass_edu/hang/imagehide/site/index.html paper/index.html
cp /work/pi_nwycoff_umass_edu/hang/imagehide/site/img/*.png paper/img/

# 2d. Rewrite repo-internal .md / src/ links → github.com/blob/main URLs
GH=https://github.com/yanghangAI/imagehide/blob/main
sed -i \
  -e "s|href=\"docs/inn_arch_notes.md\"|href=\"${GH}/docs/inn_arch_notes.md\"|g" \
  -e "s|href=\"docs/PLAN.md\"|href=\"${GH}/docs/PLAN.md\"|g" \
  -e "s|href=\"docs/scripts.md\"|href=\"${GH}/docs/scripts.md\"|g" \
  -e "s|href=\"index.md\"|href=\"${GH}/index.md\"|g" \
  -e "s|href=\"HANDOFF.md\"|href=\"${GH}/HANDOFF.md\"|g" \
  index.html
sed -i \
  -e "s|href=\"../../src/|href=\"${GH}/src/|g" \
  -e "s|href=\"../inn_arch_notes.md\"|href=\"${GH}/docs/inn_arch_notes.md\"|g" \
  docs/html/spec.html

# 3. Commit + push
git add -A
git commit -m "<summary>"
git push origin gh-pages
cd -
git worktree remove /tmp/imagehide-ghpages

Verify

# landing
curl -s https://yanghangai.github.io/imagehide/ | grep -oE 'A-tolerant|MWIP' | sort -u

# spec / design / plan / ops
for p in docs/html/{spec,design,plan,ops}.html; do
  printf "%-32s → HTTP %s\n" "/$p" "$(curl -s -o /dev/null -w '%{http_code}' https://yanghangai.github.io/imagehide/$p)"
done

Refresh archived paper-site eval samples

RUN=/work/pi_nwycoff_umass_edu/hang/autohide/runs/idea084/design002
cp "$RUN/eval_samples/cover_1024x1024.png"      site/img/cover.png
cp "$RUN/eval_samples/container_1024x1024.png"  site/img/container.png

python3 - <<'PY'
from PIL import Image
import numpy as np
cov = np.asarray(Image.open('site/img/cover.png').convert('RGB'), dtype=np.int16)
con = np.asarray(Image.open('site/img/container.png').convert('RGB'), dtype=np.int16)
amp = 20                                  # match site/index.html alt-text
res = np.clip(np.abs(con - cov) * amp, 0, 255).astype(np.uint8)
Image.fromarray(res).save('site/img/residual_x10.png', optimize=True)
PY