ZapperEdge Direct
ZapperEdge Direct is a Python command-line tool for moving very large files and whole folders into Azure Blob Storage. It splits files into blocks, retries failures and picks up where it stopped.
- Python
An open-source command-line utility that uploads very large files to Azure Blob Storage in parallel blocks and resumes after any interruption.
- Python 3.8 or later
- Azure Blob Storage
- Block Blob staging and commit
- azure-storage-blob
- python-dotenv
The challenge
Moving a very large file into cloud storage is fragile. A single dropped connection, a laptop that goes to sleep or a restarted job can force a transfer that has run for hours to begin again from the first byte.
Teams preparing data for analytics and AI often face exactly this. Datasets are large, networks are uneven, and nobody wants to babysit an upload or guess how much of it actually arrived. A reliable, repeatable way to land data in storage is a basic need.
What it does
Chunked parallel upload
Splits each file into blocks of a configurable size and uploads several at once using the Azure Block Blob pattern, so large files move without loading everything into memory.
Resume after interruption
Re-running the same command skips blocks already staged and files already completed, so an interrupted upload continues from where it left off instead of restarting.
Retry with backoff
Transient network failures are retried per block with exponential backoff. The number of attempts and the base delay are set through configuration.
Folder uploads
Accepts a single file or a directory, walks folders recursively and keeps the subfolder structure as the blob path, with an optional virtual folder prefix.
Safe progress tracking
Progress is written to a log file in an atomic way and saved when the process is stopped by an interrupt, so a crash does not corrupt the record of what was uploaded.
Dry run and metadata
A dry-run option lists what would be uploaded without touching Azure. On commit, the tool stamps blob metadata and a content type on each file.
How it works
Configure the target
Set the Azure Storage connection string, the container and tuning values such as block size, concurrency and retry limits in an environment file or on the command line.
Point at files
Run the command against a file or a folder. Folders are walked recursively and files are processed in sorted order, one file at a time.
Stage blocks in parallel
Each file is cut into blocks and a pool of worker threads stages them in Azure, retrying failures. Progress is saved after every block.
Commit and summarize
When all blocks are staged, the block list is committed and the blob is finalized. A summary lists uploaded files and any that failed, which can be re-run.
Under the hood
- Language
- Python 3.8 or later
- Cloud storage
- Azure Blob Storage
- Block Blob staging and commit
- Libraries
- azure-storage-blob
- python-dotenv
- Interface
- Command-line tool
- Environment file configuration
- JSON progress logs
Inside the harness
Data transfer
It is the open-source companion to the Data transfer harness: a building block for landing large datasets in storage reliably before any governed AI use.
Reliability
Block-level retry with backoff, resumable runs and atomic progress records make long transfers recoverable, which is the error-handling behavior the harness expects.
Security
Access to the storage account should be governed outside the tool. Review how the connection string is stored and scoped before using it with regulated data.
Audit
Per-file progress logs and a final summary give a basic record of what was uploaded. Central audit trails would come from the surrounding harness.
Business outcome
Interrupted uploads no longer start over. Staged blocks are kept and skipped on the next run, so a dropped link costs only the work in flight rather than the whole transfer.
Predictable handling of large data. Configurable block size and concurrency let teams tune a transfer to a fast data center link or an unreliable network without changing tools.
Less manual supervision. Retries, resume and a clear summary of failed files mean long uploads can run unattended and only the failures need follow-up.
A transparent starting point. Teams can read the code, adapt it and run it in their own environment, instead of depending on a closed upload utility.
Where it is used
- Landing large analytics exports or model training datasets in Azure Blob Storage ahead of processing.
- Copying a folder tree of documents or archives into a container while keeping its structure.
- Uploading from sites with unreliable connections, where transfers must survive interruptions.
- Previewing a bulk upload with a dry run before committing to a long transfer.
Frequently asked questions
How do I install and use it?
It needs Python 3.8 or later and an Azure Storage connection string. Install the package or run it as a module, set values in an environment file, then run it against a file or folder.
What licence is it under?
The package metadata declares the MIT licence. The repository does not include a separate licence file, so confirm the terms with Zapper Edge before relying on them.
Is it production-ready?
The project describes itself as production-grade, but treat it as a starting point to review. Test it with your own data, network and security requirements before using it in production.
Does it work with other clouds?
No. The tool is built for Azure Blob Storage and uses the Block Blob pattern. Other storage services would need code changes.
Using this in a governed AI setup?
Talk to us about running it inside the harness, with policy, guardrails and audit around it.