pcopy - High-Performance Local File Transfer¶

pcopy [-r|-t] [-d] <source>... <destination>
```bash

### Options

- `-r`: Force rsync mode (delta transfer, best for updates)
- `-t`: Force tar mode (streaming, best for initial copies)
- `-d`: Delete source after successful transfer (move operation)
- `-h`: Show help

## Examples

### Basic File Operations

```bash
# Copy a directory with automatic mode selection
# pcopy detects if destination exists and chooses optimal method
pcopy ~/Documents/ /backup/

# Copy multiple individual files to a destination
# Works with any number of source files
pcopy file1.txt file2.txt script.sh /dest/

# Move a file (copy then delete original)
# Useful for large files where you want progress feedback
pcopy -d large-video.mp4 /media/external/
```bash

### Mode Selection Examples

```bash
# Force rsync mode for incremental updates
# Best when destination exists and you expect few changes
pcopy -r ~/project/ /backup/project/

# Force tar mode for fresh copies
# Best for new destinations or when you want streaming transfer
pcopy -t ~/iso-files/ /external/

# Let pcopy auto-detect the best method (recommended default)
# Checks destination state and chooses rsync or tar automatically
pcopy ~/data/ /backup/
```bash

### Large Data Transfers

```bash
# Copy virtual machine images
# Tar mode is efficient for large, uncompressed files
pcopy -t ~/VMs/ubuntu.vm /external/backup/

# Sync photo library with incremental updates
# Rsync mode preserves existing photos and only transfers new/changed ones
pcopy -r ~/Pictures/ /backup/photos/

# Archive old data with deletion
# Moves data to archive and removes from source
pcopy -d ~/old-data/ /archive/
```bash

### Cross-System Transfers

```bash
# Copy to external USB drive
# pcopy handles different filesystem types automatically
pcopy -r ~/data/ /media/usb-drive/backup/

# Transfer to network mount
# Works with NFS, SMB, or other network filesystems
pcopy ~/shared/ /mnt/network-share/

# Copy to slower storage (like tape backup)
# pcopy adapts to storage speed automatically
pcopy -t ~/archive/ /mnt/tape-drive/
```bash

### Batch Operations

```bash
# Copy multiple directories at once
# Each transfer uses optimal method for its destination
pcopy ~/docs/ ~/pics/ ~/music/ /external/backup/

# Use with find for selective copying
# Archive files older than 1 year
find ~/Documents -mtime +365 -type f | \
    xargs pcopy -d /archive/documents/

# Backup with date stamps
# Create timestamped backups
DATE=$(date +%Y%m%d)
pcopy -r ~/important/ "/backup/important-$DATE/"
```bash

### Performance Optimization

```bash
# Background transfer with monitoring
# Start copy in background and monitor progress
pcopy large-dataset/ /backup/ &
txmon  # Monitor transfer status

# Low priority transfer (won't slow down system)
# Use when you want transfer to not interfere with other work
nice -n 19 ionice -c 3 pcopy ~/data/ /slow-drive/

# High priority transfer (for urgent copies)
# When you need the transfer to complete quickly
pcopy -t urgent-file.dat /fast-ssd/
```bash

## How It Works

### Auto-Detection Logic

1. **Check destination**: Exists and has content?
2. **If exists**: Use rsync (delta transfer)
3. **If empty/new**: Use tar (streaming)
4. **Override**: `-r` or `-t` forces specific mode

### Transfer Modes

#### Rsync Mode (-r)

- **Best for**: Updates, partial changes, large directories
- **Method**: Delta synchronization
- **Benefits**: Only transfers changed blocks
- **Performance**: Fast for incremental updates

#### Tar Mode (-t)

- **Best for**: Initial copies, complete transfers
- **Method**: Compressed streaming
- **Benefits**: Single operation, compression
- **Performance**: Fast for new data

## Configuration

### Environment Variables

```bash
# Disable compression (faster, larger)
export PCOPY_NO_COMPRESS=1

# Custom buffer size
export PCOPY_BUFFER_SIZE=64M

# Disable progress bar
export PCOPY_QUIET=1
```bash

### Performance Tuning

```bash
# For slow networks
export RSYNC_OPTIONS="--bwlimit=1000"

# For fast local storage
export TAR_OPTIONS="--use-compress-program=pigz"
```bash

## Dependencies

### Required

- **rsync**: For delta transfers
- **tar**: For streaming transfers
- **pv**: For progress visualization

### Optional

- **pigz/pxz**: Parallel compression (faster than gzip)
- **nocache**: Prevents cache pollution during large transfers
- **ionice**: I/O priority control

## Troubleshooting

### No Progress Shown

**Symptom:** Transfer starts but no progress bar appears

**Cause:** `pv` not installed or not in PATH

**Fix:**

```bash
# Install pv
sudo apt install pv      # Ubuntu/Debian
sudo dnf install pv      # Fedora
brew install pv          # macOS
```bash

### Permission Denied

**Symptom:** "Permission denied" during transfer

**Cause:** Source/destination permission issues

**Fix:**

```bash
# Check permissions
ls -la /source/file
ls -ld /destination/

# Fix permissions
chmod +r /source/file
chmod +w /destination/
```bash

### Disk Space Insufficient

**Symptom:** "No space left on device"

**Cause:** Destination doesn't have enough space

**Fix:**

```bash
# Check space before transfer
df -h /destination/

# Free up space or use different destination
rm /destination/old-files
pcopy /source/ /different/destination/
```bash

### Slow Transfer Speed

**Symptom:** Transfer much slower than expected

**Cause:** Various performance issues

**Fix:**

```bash
# Check system load
uptime
iotop  # If available

# Use ionice for lower priority
ionice -c 3 pcopy /source/ /dest/

# Check for competing I/O
ps aux | grep -E "(rsync|tar|cp)"
```bash

## Performance

### Benchmarks

| Scenario                       | Rsync Mode | Tar Mode | Winner |
| ------------------------------ | ---------- | -------- | ------ |
| New 1GB file                   | 45s        | 32s      | Tar    |
| Update 1GB (1% changed)        | 2s         | 35s      | Rsync  |
| New directory tree             | 120s       | 95s      | Tar    |
| Sync directory (small changes) | 5s         | 125s     | Rsync  |

### Optimization Tips

- **Large initial copies**: Use `-t` (tar mode)
- **Frequent updates**: Use `-r` (rsync mode) or auto-detect
- **Network transfers**: Use compression options
- **Slow storage**: Reduce buffer sizes
- **Background**: Use `nice` and `ionice`

## Integration

### With Backup Scripts

```bash
#!/bin/bash
# Daily backup script
DATE=$(date +%Y%m%d)
pcopy -r ~/Documents/ "/backup/docs-$DATE/"
pcopy -d ~/Downloads/ /backup/downloads/
```bash

### With Archive Management

```bash
# Archive old photos
find ~/Pictures -name "*.jpg" -mtime +365 | \
    xargs pcopy -d -t /archive/photos/

# Sync to external drive
pcopy -r ~/data/ /media/external/backup/
```bash

### With Monitoring

```bash
# Monitor transfer progress
pcopy large-file.iso /dest/ &
txmon  # Monitor active transfers
```bash

## Technical Details

### Rsync Mode Implementation

```bash
rsync -avh --progress --partial --inplace \
      --exclude='.DS_Store' \
      "$source" "$dest" | pv -lep -s $(du -sb "$source" | cut -f1)
```bash

### Tar Mode Implementation

```bash
tar cf - "$source" | pv -s $(du -sb "$source" | cut -f1) | \
    tar xf - -C "$dest"
```bash

### Space Checking

- Pre-flight: Checks available space before transfer
- Formula: `source_size * 1.1` (10% safety margin)
- Fails early if insufficient space

### Error Handling

- **Atomic operations**: Uses temporary files/directories
- **Resume capability**: Partial transfers can resume
- **Cleanup**: Removes temporary files on failure
- **Exit codes**: 0=success, 1=error, 2=insufficient space

### Compression

- **Automatic**: Enabled for tar mode by default
- **Algorithms**: gzip (default), pigz (parallel), xz (high compression)
- **Detection**: Uses fastest available compressor

## Gotchas

### Insufficient Disk Space Detection Fails

**Problem:** Transfer appears to start but fails midway with "no space" error.

**Cause:** Space checking uses estimated size, but compressed transfers or
metadata can exceed estimates.

**Solution:** Always ensure at least 20% more space than source size:

```bash
# Check space manually
df -h "$destination"
du -sh "$source"
```bash

### Rsync vs Tar Mode Confusion

**Problem:** Unexpected transfer method selection leads to slow transfers.

**Cause:** pcopy auto-selects based on destination state, but this may not
match your expectations.

**Solution:** Force specific mode explicitly:

```bash
# Force rsync for incremental updates
pcopy -r source/ dest/

# Force tar for fresh copies
pcopy -t source/ dest/
```bash

### Progress Visualization Issues

**Problem:** Progress bar shows incorrect or no progress.

**Cause:** `pv` tool not installed or terminal doesn't support progress display.

**Solution:** Install `pv` or use simpler output:

```bash
# Install progress visualizer
sudo apt install pv  # or brew install pv

# Use without progress for scripting
pcopy --quiet source dest
```bash

### Large File Handle Limits

**Problem:** Transfer fails with "too many open files" error.

**Cause:** System ulimit too low for many small files.

**Solution:** Increase file handle limit temporarily:

```bash
# Check current limit
ulimit -n

# Increase for session
ulimit -n 4096

# Or run with increased limit
ulimit -n 4096 && pcopy source dest
```bash

### Cross-Filesystem Permission Issues

**Problem:** File permissions or ownership not preserved correctly.

**Cause:** Different filesystem types or mount options.

**Solution:** Check filesystem compatibility:

```bash
# Check filesystem types
df -T source dest

# Use tar mode for better attribute preservation
pcopy -t source dest
```bash

### Temporary Space Requirements

**Problem:** Transfer fails due to insufficient temp space in `/tmp`.

**Cause:** Tar mode needs temporary space for compression/decompression.

**Solution:** Ensure temp directory has adequate space:

```bash
# Check temp space
df -h /tmp

# Use alternative temp directory
export TMPDIR=/var/tmp && pcopy source dest
```bash

## See Also

- **[persync](persync.md)** - Resilient remote sync
- **[txmon](txmon.md)** - Transfer monitoring
- **[File Transfer Guide](../../README.md#file-transfer)** - Transfer strategy overview