Mirror of the documentation in the DetFill-with-DHT repository (source file reproduce/README.md); links to files and release downloads point at GitHub, and the hintauc library is available from PyPI.

Reproducing the paper's results

This directory is the reproduction package for

Hint-AUC: Deterministic Region-based Hint Generation for Line Art Colorization Evaluation (IEEE TVCG 2026, DOI 10.1109/TVCG.2026.3738401). Main paper: Tables II–V and Fig. 9. Supplement: the hint-ratio-grid (α-grid) study, the segmentation-dependency study, the seed-sensitivity study, the additional training comparisons, the natural-image tables and the user study.

Quick index

replicability/run.sh and reproduce/examples/run_examples.sh fetch what they need from the releases (SHA-256 verified) and compare their results with the paper's numbers, the paper's archived images of the same examples and the authors' reference run; the full-scale and training scripts expect the data and checkpoints to be in place. Requirements and hardware: docs/setup.md. The Replicability-Stamp submission sheet is replicability/GRSI_SUBMISSION.txt.

Coverage: what each paper item needs

"Bit-exact" means the printed cell is recomputed from the released per-ratio metric files; "re-run" means the same protocol is executed again (the diffusion sampler is seeded but not bit-exact across GPU generations, so re-run metrics move in the last printed digit). A1 covers 324 printed cells, 322 exactly (see "Known deviations").

The four layers

It has four layers, from cheapest to most expensive. Everything runs from this repository plus the files attached to its GitHub releases (see checkpoints/README.md); nothing else is needed except the Danbooru2021 originals for layer D.

../replicability/run.sh (Fig. 9, one command, ≈ 3 min on a GPU) is the Graphics-Replicability-Stamp entry point and is independent of this directory.

Hardware behind the timings. Machine A: NVIDIA RTX A6000 (48 GB), 2 × AMD EPYC 9124 (32 threads), 377 GB RAM. Machine B: NVIDIA GeForce RTX 2080 Ti (11 GB), 2 × Intel Xeon Gold 6226R (32 threads), 187 GB RAM. Both Ubuntu 22.04, driver 535, CUDA 12.2. Fig. 9: 3 min on A, 4–5 min on B, 30 min on the CPUs of A (16 threads). Example suite: measured on B. Full-scale inference: about 35 min per (ratio, line-art source) cell of 3,000 images on A with the 96-channel scribble model, i.e. ≈ 14 GPU-hours per table row on A (about twice that on B).

A. Recompute the tables from the released metric files (no GPU)

conda env create -f replicability/environment.yml && conda activate detfill-grsi   # or any env with numpy, pandas, statsmodels
python reproduce/scripts/A1_tables_from_released_metrics.py   # Tables II/III (DetFill rows) + supplementary tables
python reproduce/scripts/A2_userstudy_glmm.py                 # Tables IV/V + the GLMM statistics of Sec. VII

A1_tables_from_released_metrics.py writes Markdown tables to reproduce/output/tables/ and a cell-by-cell comparison with the numbers printed in the paper (check_report.csv). Reference copies of both scripts' outputs are in expected/recomputed_tables/.

Result of the authors' run: A1 rebuilds 322 of the 324 compared cells to the printed precision; A2 reproduces all 30 percentages of Tables IV/V (the 6 pairwise and the 24 per-ratio values) exactly and the three GLMM statistics (z = 30.18; χ²(7) = 618.33, p = 2.75e-129; χ²(14) = 324.04, p = 1.12e-60). The two A1 exceptions are a rounding artefact and one provenance gap, both listed under "Known deviations" below; neither changes a conclusion.

What each table is rebuilt from (the rule is always the paper's: Hint-AUC = trapezoidal integral of the per-ratio metric over the hint ratio α ∈ {0, 1, 3, 5, 10, 25, 50, 100} %, computed per line-art source and reported as mean ± sample standard deviation over the three sources — XDoG, sketch simplification, SketchKeras):

expected/ also contains the analyses written for the review response that are not printed in the paper (userstudy_rank_stability/, segmenter_sensitivity/, alpha_grid/A-5/), with their scripts under paper_experiments/.

B. Example suite: eight experiments on example images (one GPU, no data preparation)

bash reproduce/examples/run_examples.sh                       # default "quick" set (≈ 1 h on one GPU)
EXAMPLES_MODE=smoke bash reproduce/examples/run_examples.sh   # 1 image, 2 ratios, scribble model only (≈ 5 min)
EXAMPLES_MODE=full  bash reproduce/examples/run_examples.sh   # all experiments, 9 images, 8 ratios (several GPU-hours)

The script creates (or reuses) the conda environment of replicability/run.sh, downloads the checkpoints and the 9-image example bundle from the releases (SHA-256 verified), runs each experiment with the same detfill/main.py entry point and the same hintauc evaluator as the full-scale experiments, and writes:

• reproduce/examples/output/grids/*.png — one labelled image grid per experiment (ground truth, line art, hints, outputs per hint ratio / per model),
• reproduce/examples/output/metrics/*.json — per-image and mean values of the seven metrics, and Hint-AUC where the full ratio grid was run,
• a printed comparison with reproduce/examples/expected/ (the authors' run of the same script) and, for the Table II protocol, the PSNR between each new output and the paper's archived output of the same image.

reproduce/examples/README.md documents each experiment, the images used and the authors' numbers.

C. The verbatim launchers

paper_experiments/README.md lists, per experiment, the shell and Python files that were actually run for the paper and the revision (with our cluster's paths and environment names). They are not runnable as-is elsewhere; the runnable equivalents are the scripts in scripts/ and examples/.

D. Full-scale re-run of a table row

D.1 Data

1. Test images. The test split is the 3,000 Danbooru2021 images listed in data/splits/test.txt (ids ending in 016); we do not redistribute them. The Danbooru2021 ids are Danbooru post ids, and the original Danbooru2021 mirror is offline, so fetch them from Danbooru by id: python reproduce/scripts/fetch_originals.py --id_file reproduce/data/splits/test.txt --out <DATA_ROOT>/segmentations/originals writes the 512 × 512 copies (<id>.image.png, bilinear resize of the file; bit-exact with the dataset's ground truth of training and metrics). Deleted posts and posts that need an account are reported. The stored hint maps (step 3) were computed from the files at their original resolution, so they must be used as shipped, not regenerated from the 512 × 512 copies.
2. Line art. Release v1.3 ships the exact line-art files used for every reported number (test_split_sketch_{sketchkeras,pysimp,XDoG}.tar.gz → <DATA_ROOT>/sketch/<extractor>/<id>.png). The wrappers that generated them are in scripts/sketch_tools/ (XDoG draws its σ/k jitter from an unseeded generator, so regenerated XDoG line art is not bit-identical).
3. Deterministic hints. Release v1.0 ships the stored maps (test_split_hint_maps_64.tar.gz → <DATA_ROOT>/hint_from_regions_64_rev/ and region64/* → <DATA_ROOT>/hint_from_regions_256/). They are the exact inputs of every reported number.
4. Checkpoints. checkpoints/README.md.

D.2 One table row = one command

# Table II, DetFill scribble row (default), 7 metrics, Hint-AUC over the paper grid
DATA_ROOT=/data/danbooru_test GPU=0 bash reproduce/scripts/run_hauc_pipeline.sh
# Table II, DetFill dot row
DATA_ROOT=/data/danbooru_test HINT=dot GPU=0 bash reproduce/scripts/run_hauc_pipeline.sh
# Table III (fixed random region order = ascending label order of the stored region map)
DATA_ROOT=/data/danbooru_test HINT_ORDER=label GPU=0 bash reproduce/scripts/run_hauc_pipeline.sh
# natural-image (ImageNet) rows: the *_real.yaml configs (64 channels, release v1.4 checkpoints), split data layout;
# the metrics use the sampler's ground_truth/ copies unless GT_DIR is given
DATA_ROOT=/data/imagenet_subset DOMAIN=real CKPT=checkpoints/detfill_scribble_imagenet_64ch_200ep.pth GPU=0 bash reproduce/scripts/run_hauc_pipeline.sh
# supplementary α-grid study: dense sweep, then integrate on the alternative grids
DATA_ROOT=/data/danbooru_test RATIOS="$(seq -f %.2f 0 0.02 1 | tr '\n' ' ') 0.01 0.03 0.05 0.075 0.15 0.25" TAG=dense \
    bash reproduce/scripts/run_hauc_pipeline.sh
python reproduce/paper_experiments/alpha_grid/B_auc_grid_sensitivity_7m.py \
    --summary reproduce/output/dense/metrics/per_ratio_summary.csv --out_dir reproduce/output/dense/auc
# smoke test of the pipeline: point DATA_ROOT at a copy that holds only a few test images (the loader colorizes every
# image it finds; TEST_BATCH=1 for image counts that the config's batch size does not divide)
DATA_ROOT=/data/danbooru_small TEST_BATCH=1 TYPES="2" GPU=0 bash reproduce/scripts/run_hauc_pipeline.sh

The inference loader processes the last incomplete batch, so any number of images works; TEST_BATCH only changes the batch composition (the sampler is seeded per batch, so a different batch size changes the outputs slightly). Every output directory carries a run_manifest.json (checkpoint hash, seed, hint type and order, ratio, sketch type, sampling steps, config hash, data root); a run whose conditions differ refuses to reuse the existing images. A resumed run consumes the sampler's random stream differently than an uninterrupted one, so its images are not bit-identical to a fresh run (the metrics move within the usual sampler spread). Before the inference starts, scripts/check_data_root.py verifies that every file the loader will read exists (ALLOW_MISSING=1 only reports). The evaluation step (eval_per_ratio.py) checks that all ratio directories hold the same images, reuses rows of an earlier run only when the prediction and ground-truth files are unchanged (SHA-256) and the metric settings match its run_manifest.json, and writes those hashes into per_image.csv.

run_hauc_pipeline.sh = detfill/main.py --sample_to_eval for every (ratio, line-art source) followed by scripts/eval_per_ratio.py, which uses the released evaluator hintauc.metrics.Evaluator (the paper's evaluation/eval_single_run.py: 256 × 256, MSE/PSNR/SSIM on [0, 1]; LPIPS-Alex; OpenCLIP ViT-B-32 laion2b; DINOv2-base; DreamSim) and writes per_ratio_summary.csv in the same format as the files in expected/, plus hauc.json. The perceptual metrics need pip install -r reproduce/requirements-metrics.txt.

Expected agreement (checked on 160 image pairs of the archived paper outputs): with requirements-metrics.txt the evaluator reproduces the archived per-image values to numerical precision for MSE, PSNR, LPIPS, DINO and DreamSim (max |Δ| ≤ 3e-6), SSIM to 5e-5 with the pinned torchmetrics==1.4.0 (torchmetrics 1.8 changed the SSIM implementation: per-image |Δ| up to 0.04, so the pin matters), and OpenCLIP within 8e-4 (the archived run scored OpenCLIP in half precision). Inference is seeded and the stored hint maps remove the hint-generation variance, but the diffusion sampler is not bit-exact across GPU generations or on CPU: re-generated outputs of the same image agree with the paper's archived outputs only approximately (about 25–30 dB PSNR in our checks, see reproduce/examples/). We have not re-run a complete table row on different hardware; expect small differences in the last printed digit.

D.3 Other studies at full scale

• Seed sensitivity (supp.) — paper_experiments/seed_sensitivity/gen_seeded_random_hints.py bakes a seeded uniformly random region order into 64 × 64 hint maps (10 seeds × 6 intermediate ratios; α = 0 % and 100 % are seed-independent); each (seed, α) set is then run through the pipeline above at RATIOS=1.00 with DATA_ROOT pointing at a data root whose hint_from_regions_64_rev/ holds that seed's maps, evaluated on the first 300 test ids with SketchKeras line art, and aggregated exactly as in A1 part 6. examples/ does this for three images.
• Segmentation dependency (supp.) — release v1.3 ships the DanbooRegion and SLIC hint maps of the test split, release v1.1 the two retrained models; run the pipeline with CKPT=<v1.1 file> and DATA_ROOT pointing at a copy of the data root whose hint maps were replaced by the segmenter's maps. Regeneration of the maps for other images: paper_experiments/segmenter_dependency/r2-2_scripts/ (D_danbooregion_seg.py, D_retrain_gen_hints.py, D_danboo_hints.py) and hint_generation/generate_hints.py --segmenter slic. Training of the two models: D_train_launch.sh / D_stage_train.sh (200 epochs, same protocol as scripts/B9_train_detfill.sh).
• Training DetFill from scratch — scripts/B9_train_detfill.sh (the paper's train.sh with the released config names; ~4–5 days on 10 GPUs).
• Diffusart-retrain (supp.) — paper_experiments/diffusart_retrain/code/ (our Diffusart re-implementation: training loop, deterministic-hint loader, inference and evaluation; README inside) with the two released EMA checkpoints of release v1.2.
• ColorizeDiffusion fine-tuning (Table II/III "200ep", supp. "7ep") — paper_experiments/coldiff_finetune/ (launchers for the official ColorizeDiffusion v1/v2 code; the fine-tuned weights are in release v1.5).

Known deviations and gaps (honest list)

• Hint regeneration is not bit-exact. The stored maps were generated from the original-resolution Danbooru2021 files, and FilFinder breaks medial-axis ties with an unseeded generator: regenerating the maps reproduces the region and dot counts but moves a few scribble pixels per image from run to run (E9 quantifies it). All reported numbers use the stored maps.
• Dot placement rule. Sec. IV-A describes the dot as the truncated mean of the longest-path pixels; the stored hint maps (training data and the released test maps) were produced with the medoid rule (dot_method="medoid", the library default; verified on the released maps). All reported numbers use the stored maps, so results are unaffected; dot_method="mean" reproduces the text's rule.
• Table II, DetFill scribble SSIM is printed as 0.724; the value is 0.72348 (printed as 0.7235 in the supplementary α-grid table). A rounding artefact in the main-table typesetting.
• Diffusart-retrain SSIM. The recovered per-ratio record of that run reproduces PSNR 19.457 and LPIPS 0.177 exactly but gives SSIM 0.7085 where the table prints 0.717 ± 0.066; the seven-metric evaluation the other cells were taken from was not recovered (expected/additional_training/diffusart_retrain_R3-2/NOTE.md).
• SSIM depends on the torchmetrics version. The published SSIM values were computed with torchmetrics 1.4.0; use the pinned version (requirements-metrics.txt, replicability/environment.yml) — 1.8.x gives per-image SSIM values that differ by up to 0.04 (all other metrics are unaffected).
• Line-art source labels. The DetFill data loader maps sketch_type 0 / 1 / 2 to sketch simplification / XDoG / SketchKeras; the archived per-ratio files name index 0 "XDoG" and 1 "pysimp" (a label swap in the old evaluation script). Every published value is a mean ± SD over the three sources, so no number depends on this; the per-source tables written by A1 use the loader's mapping.
• Dot rows. The DetFill dot rows date from the original submission and have no released per-ratio record. Re-running the released dot checkpoint on the released maps (300 test images, SketchKeras line art, run_hauc_pipeline.sh protocol) gives PSNR Hint-AUC 17.46 against 17.94 printed and LPIPS 0.24 against 0.18, i.e. the same model within the spread expected from a subset and a different sampling batch composition, not a bit-exact reproduction.
• Not covered by released per-ratio data: the Table II/III dot rows of DetFill, all baseline rows (PaintsTorch, Diffusart, ColorizeDiffusion v1/v2), the natural-image (ImageNet) tables and the legacy per-source tables of the supplement (earlier checkpoints). The dot row and the baselines can be regenerated with layer D and the respective official code (the ColorizeDiffusion fine-tuned weights are in release v1.5). The natural-image DetFill models behind the printed ImageNet tables are the 64-channel models of release v1.4 (our re-evaluation on 24 test images matches the printed Hint-AUC within 0.7 dB; the 96-channel models of v1.2 are a later generation), and the ImageNet test hint maps are in v1.2; without the archived per-image record those tables remain "same protocol", not bit-exact, reproductions.
• User study. The released trial table is anonymised and reproduces Tables IV/V and the GLMM exactly; the stimuli of three of the four methods are released (v1.3; line art, hints and colorizations — the original illustrations are fetched by id from the archive's index.csv), the ColorizeDiffusion-v2 stimuli were not preserved. The response-letter analyses in paper_experiments/userstudy_rank_stability/ read the raw per-participant CSVs through common/ab_loader.py, which are not released.

Layout

reproduce/
  scripts/            A1_tables_from_released_metrics.py  A2_userstudy_glmm.py  eval_per_ratio.py
                      run_hauc_pipeline.sh  B9_train_detfill.sh  sketch_tools/
  examples/           run_examples.sh  make_examples.py  compare_with_expected.py  expected/  README.md
  expected/           released per-ratio metric files and JSON summaries (inputs of A1), recomputed_tables/
  data/               userstudy/glmm_trials.csv, splits/{train,valid,test}.txt
  paper_experiments/  verbatim launchers and analysis scripts of the paper and the revision (README.md there)
  requirements-metrics.txt   perceptual-metric packages (pinned)
  output/             created by the scripts (git-ignored)