Call for Testing: Duplicity 4.0 Pre-Releases
The primary objective of automated backup systems is the prevention of data loss and the preservation of system recoverability. When backup software fails, the consequences—lost work, downtime, and unrecoverable information—impose severe costs across organizations and individuals.
To maximize the overall reliability, performance, and data security of the broader backup ecosystem, we invite system administrators, developers, and power users to participate in testing Duplicity 4.0 (duplicity4) pre-releases. By deploying and evaluating Duplicity 4.0 across diverse operating systems, storage backends, and workload profiles before the general production release, community testing delivers the greatest collective benefit: surfacing edge cases early, verifying throughput optimizations, and guaranteeing that restores remain completely deterministic and dependable.
Key Innovations and Changes in Duplicity 4.0
Duplicity 4.0 represents a major architectural modernization designed to maximize backup and restore throughput while minimizing resource consumption:
- Concurrent Backend Worker Pool: Multi-worker asynchronous volume compression, encryption, and upload pipeline (
src/backend.py) significantly reduces total backup duration. - Concurrent Restore Prefetching: Asynchronous volume prefetching for
restoreandverifyoperations drastically cuts recovery time. - Hardware-Accelerated Compression: Integration of
zlib-ngprovides accelerated gzip compression, maximizing CPU throughput during volume archiving. - Advanced File Handling:
- Native sparse file detection and efficient chunking via
--sparseand--no-sparse. - Preserved hardlink tracking and reconstruction via
--hardlinksand--no-hardlinks.
- Native sparse file detection and efficient chunking via
- New and Refined CLI Commands:
- Dedicated subcommand structure (
full,inc,restore,verify,compare,collection-status,list-current-files,cleanup,remove-older-than). - New
compareaction to inspect differences between backed-up archives and local files without performing a restore. - Standardized logging with named severity levels (
error,warning,notice,info,debug,critical) and explicit machine-readable error codes.
- Dedicated subcommand structure (
- Modernized Metadata & Storage Format:
- Manifests structured in JSON Lines format (Manifest v2) for improved extensibility.
- Machine-readable execution statistics (
jsonstat). - Bounded memory and disk utilization during deep signature verification.
- Modern Python Runtime: Full compatibility with
Python 3.10throughPython 3.15.
Where to Find Duplicity 4.0
Duplicity 4.0 source code, development tags, and release packages are accessible through official channels:
- GitLab Repository: https://gitlab.com/AlphaPrime/duplicity4.git
- GitLab Releases & Pre-Release Tarballs: https://gitlab.com/AlphaPrime/duplicity4/-/releases
- Issue Tracker: https://gitlab.com/AlphaPrime/duplicity4/-/issues
- Documentation: https://duplicity4.readthedocs.io/ and https://duplicity.us
- Mailing List: duplicity-talk@nongnu.org
Installing Pre-Releases via pip and pipx
Pre-compiled development releases and wheels are published on PyPI. Testing via pip or pipx allows for quick deployment in isolated environments without disturbing system-level backup utilities.
1. System Prerequisites
Duplicity links with native libraries for cryptography, delta compression, and accelerated archiving. Ensure the prerequisite development headers and runtime tools are present on your system:
Debian / Ubuntu:
sudo apt-get update
sudo apt-get install -y build-essential libgpgme-dev librsync-dev libssl-dev libffi-dev pkg-config python3 python3-dev python3-venv python3-pip python3-gpg zlib1g-dev
Fedora / RHEL / CentOS:
sudo dnf install -y gcc gpgme-devel librsync-devel openssl-devel libffi-devel pkgconfig python3-devel python3-gpg zlib-ng-compat-devel
macOS (via Homebrew):
brew install gpgme librsync openssl zlib-ng pkg-config
2. Isolated User Installation via pipx (Recommended)
Using pipx installs Duplicity inside an isolated virtual environment while making the duplicity CLI tool directly accessible:
# Install the latest pre-release version
pipx install --pip-args="--pre" duplicity
# Verify the installed version
duplicity --version
To upgrade an existing pipx test installation when newer pre-releases are uploaded:
pipx upgrade --pip-args="--pre" duplicity
To install with a custom command suffix so it runs side-by-side with production tools:
pipx install --pip-args="--pre" --suffix="-4.0" duplicity
duplicity-4.0 --version
3. Installation in a Python Virtual Environment (venv + pip)
For isolated testing within a dedicated shell directory:
# Create and activate a testing virtual environment
python3 -m venv duplicity4-test-env
source duplicity4-test-env/bin/activate
# Upgrade packaging tools
pip install --upgrade pip setuptools wheel
# Install the Duplicity 4.0 pre-release package with backend extras
pip install --pre duplicity
# Optionally install backend-specific client dependencies
pip install b2sdk boto3 azure-storage-blob pydrive2 paramiko
Running Directly from the Git Source Repository
For testers who want to test the latest commits directly:
# Clone the repository
git clone https://gitlab.com/AlphaPrime/duplicity4.git
cd duplicity4
# Build the C extension modules (librsync bridge)
make ext
# Execute directly from the source tree
PYTHONPATH=. src/__main__.py --version
High-Impact Testing Focus Areas
To maximize the value of your testing effort, we recommend focusing on areas where configuration variety is highest:
1. Full & Incremental Backup Workflows
Test full backups followed by successive incremental backup cycles to verify signature generation and delta chaining:
# Initial full backup to target backend
duplicity full /path/to/test_data target-backend://url/test-backup
# Incremental backup after modifying/adding files
duplicity inc /path/to/test_data target-backend://url/test-backup
2. Verification and Comparison Actions
Evaluate the accuracy and memory efficiency of archive validation:
# Compare archive contents directly with the filesystem
duplicity compare target-backend://url/test-backup /path/to/test_data
# Perform deep bit-level verification
duplicity verify target-backend://url/test-backup /path/to/test_data
3. Restore Integrity Testing
Perform test restores to alternate directories and verify file checksums, timestamps, and permissions:
# Restore complete backup set
duplicity restore target-backend://url/test-backup /path/to/restore_dir
# Restore a specific single file
duplicity restore --file-to-restore relative/file.txt target-backend://url/test-backup /path/to/restore_file.txt
4. Storage Backend Interoperability
Duplicity interacts with diverse remote protocols. Testing across varied storage destinations ensures high utility across user workflows:
- Local / POSIX storage (
file://) - SSH / SFTP (
pex+sftp://orparamiko+sftp://) - Amazon S3 & S3-compatible storage (
s3://ors3+http://) - Backblaze B2 (
b2://) - Google Cloud Storage / Google Drive (
gcs://,pydrive://) - Microsoft Azure Blob / OneDrive (
azure://,onedrive://) - WebDAV / rsync (
webdav://,rsync://)
5. Advanced File Attributes & Special Data Types
- Sparse Files: Back up directories containing large sparse files using
--sparseand verify that disk allocations are correctly preserved on restore. - Hard Links: Create multiple hard links to shared inodes, back up with
--hardlinks, and verify link preservation upon restore.
Risk Mitigation & Safety Recommendations
To avoid any disruption to critical data while evaluating pre-release builds:
- Use Dedicated Test Repositories: Direct all testing runs to separate test buckets, test directories, or separate backup targets. Do not overwrite primary production backup repositories with pre-release builds.
- Maintain Independent Secondary Backups: Ensure production datasets are protected by existing verified backup routines before conducting high-volume filesystem tests.
- Verify Restores in Sandbox Locations: Always restore to temporary validation folders (e.g.
/tmp/restore-test) rather than live system paths.
Feedback and Defect Reporting
Every bug discovered and fixed during pre-release testing protects the stability of all future users. When reporting defects, providing precise technical information minimizes turnaround time:
- Where to Report: Open an issue at https://gitlab.com/AlphaPrime/duplicity4/-/issues
- Essential Details to Include:
- Exact Duplicity version: output of
duplicity --version - Operating system distribution and architecture (
uname -a) - Python version (
python3 --version) - Backend protocol and library versions (e.g.,
boto3,b2sdk,paramiko) - Command executed (sanitizing private credentials/passwords)
- Detailed log output obtained with verbosity enabled (
-v notice,-v info, or-v debug) - Expected behavior versus observed outcome, including complete stack traces
- Exact Duplicity version: output of
Thank you for contributing your time and compute resources to ensure Duplicity 4.0 delivers the highest standard of data safety and performance.