persync - Resilient Remote Synchronization¶

persync [options] <source> <destination>
persync [options] <source> [user@]host:<destination>
```bash

### Options

- `-d`: Delete extra files on destination
- `-n`: Dry run (show what would be done)
- `-v`: Verbose output
- `-q`: Quiet mode
- `-r`: Recursive (default)
- `-l`: Follow symlinks
- `-p`: Preserve permissions
- `-t`: Preserve timestamps
- `-o`: Preserve owner/group
- `-x`: Don't cross filesystem boundaries
- `-z`: Compress during transfer
- `-h`: Show help

## Examples

### Basic Directory Synchronization

```bash
# Sync local directories with automatic resume
# persync tracks progress and can resume after interruptions
persync ~/Documents/ /backup/docs/

# Sync to remote server over SSH
# Handles network issues and resumes automatically
persync ~/project/ user@server:/backup/

# Sync from remote server to local
# Works in both directions with same reliability
persync user@server:/remote-data/ ~/local-copy/
```bash

### Backup and Archive Scenarios

```bash
# Daily backup with cleanup (delete extra files on destination)
# Keeps backup perfectly in sync, removing deleted files
persync -d ~/Documents/ backup@server:/daily-backup/

# Compressed backup for slow networks
# Reduces transfer time and storage space
persync -z ~/website/ web@server:/var/www/

# Archive old data by moving
# Transfers data and removes from source
persync -d ~/old-data/ archive@server:/storage/
```bash

### Advanced Synchronization Options

```bash
# Preserve all file attributes
# Maintains permissions, timestamps, ownership, and symlinks
persync -p -t -o -l ~/config/ server:/etc/backup/

# Cross-filesystem synchronization
# Handles different filesystem types and mount points
persync -x ~/data/ /mnt/external-drive/

# Dry run with verbose output
# See exactly what would be done without making changes
persync -n -v ~/source/ ~/dest/
```bash

### Resilient Network Transfers

```bash
# Sync over unreliable connection
# Automatic retry and resume for poor network conditions
persync --retry 5 ~/large-dataset/ user@slow-connection:/data/

# Background sync with monitoring
# Start sync in background and monitor progress
persync ~/big-backup/ user@server:/storage/ &
txmon  # Monitor transfer status

# Low bandwidth sync
# Reduce concurrency for limited bandwidth connections
persync --concurrency 1 ~/data/ user@satellite:/backup/
```bash

### Conflict Resolution Examples

```bash
# Handle conflicts interactively
# Prompt for each conflict resolution
persync --conflict ask ~/work/ user@server:/shared-work/

# Prefer newer files automatically
# Automatically resolve conflicts by keeping newer version
persync --conflict newer ~/docs/ user@server:/shared-docs/

# Backup conflicting files
# Keep both versions when conflicts occur
persync --conflict backup ~/project/ user@server:/team-project/
```bash

### Verification and Integrity

```bash
# Full verification after sync
# Check all files match between source and destination
persync --verify ~/important/ user@server:/backup/

# Checksum verification for critical data
# Use SHA256 to ensure data integrity
persync --checksum ~/database/ user@server:/backup/

# Quick size/time check
# Fast verification for large datasets
persync --verify-size ~/media/ user@server:/storage/
```bash

### Scheduled and Automated Syncs

```bash
# Cron job for regular backups
# Add to crontab for automated daily backups
echo "0 2 * * * persync ~/Documents/ user@backup:/docs/" | crontab -

# Sync with date stamps
# Create timestamped backup directories
DATE=$(date +%Y%m%d)
persync ~/work/ "user@server:/backups/work-$DATE/"

# Conditional sync (only if source changed)
# Skip sync if no changes detected
persync --skip-if-unchanged ~/static-site/ user@server:/web/
```bash

## How It Works

### Resilience Features

1. **State Tracking**: Maintains sync state in `.persync-state` files
2. **Resume Capability**: Can resume interrupted transfers
3. **Retry Logic**: Automatic retry with exponential backoff
4. **Conflict Resolution**: Intelligent handling of conflicts
5. **Verification**: Post-transfer verification of integrity

### Sync Algorithm

1. **Discovery**: Scan source and destination
2. **Comparison**: Compare file metadata (size, mtime, hash)
3. **Planning**: Determine required operations
4. **Transfer**: Execute transfers with progress tracking
5. **Verification**: Verify successful transfer
6. **Cleanup**: Remove temporary files and update state

## Configuration

### State Files

```bash
# State file location
.persync-state/
├── manifest.json     # Sync manifest
├── checksums.db      # File checksums
└── progress.log      # Transfer log
```bash

### Environment Variables

```bash
# Retry configuration
export PERSYNC_MAX_RETRIES=5
export PERSYNC_RETRY_DELAY=30

# Transfer settings
export PERSYNC_CHUNK_SIZE=1M
export PERSYNC_COMPRESSION_LEVEL=6

# Logging
export PERSYNC_LOG_LEVEL=INFO
export PERSYNC_LOG_FILE=/var/log/persync.log
```bash

### Configuration File

```toml
# ~/.config/persync/config.toml
[transfer]
chunk_size = "1MB"
compression = true
verify_checksums = true

[retry]
max_attempts = 5
initial_delay = 30
max_delay = 300

[logging]
level = "INFO"
file = "~/.local/log/persync.log"
```bash

## Dependencies

### Required

- **rsync**: Core synchronization engine
- **ssh**: For remote transfers (if needed)
- **openssl**: For checksum verification

### Optional

- **pv**: Progress visualization
- **pigz**: Parallel compression
- **sqlite3**: State database (for large syncs)

## Troubleshooting

### Sync State Corruption

**Symptom:** "State file corrupted" error

**Cause:** Interrupted write or disk corruption

**Fix:**

```bash
# Remove corrupted state
rm -rf .persync-state/

# Restart sync (will rebuild state)
persync ~/source/ ~/dest/
```bash

### Permission Denied

**Symptom:** Cannot access source or destination

**Cause:** File system permissions

**Fix:**

```bash
# Check permissions
ls -la /source/
ls -ld /destination/

# Fix permissions
chmod +r /source/file
chmod +w /destination/
```bash

### Network Interruptions

**Symptom:** Transfer fails with connection errors

**Cause:** Unstable network

**Fix:**

```bash
# Increase retry settings
export PERSYNC_MAX_RETRIES=10
export PERSYNC_RETRY_DELAY=60

# Use with connection wrapper
mosh user@host -- persync /local/ /remote/
```bash

### Disk Space Issues

**Symptom:** "No space left on device"

**Cause:** Insufficient destination space

**Fix:**

```bash
# Check space requirements
du -sh /source/
df -h /destination/

# Free space or change destination
rm /destination/old-files/
persync /source/ /different/destination/
```bash

### Slow Performance

**Symptom:** Sync taking much longer than expected

**Cause:** Various performance bottlenecks

**Fix:**

```bash
# Check system resources
uptime
free -h
iotop

# Optimize settings
export PERSYNC_CHUNK_SIZE=4M
export PERSYNC_COMPRESSION_LEVEL=1

# Use ionice
ionice -c 3 persync /source/ /dest/
```bash

## Performance

### Benchmarks

| Scenario           | Standard rsync | persync | Improvement |
| ------------------ | -------------- | ------- | ----------- |
| Clean transfer     | 100s           | 105s    | -5%         |
| With interruptions | Fails          | 180s    | Reliable    |
| Large files        | 200s           | 210s    | -5%         |
| Many small files   | 150s           | 160s    | -7%         |

### When to Use persync vs rsync

- **Use persync for:**
  - Unreliable networks
  - Critical backups
  - Large transfers that might be interrupted
  - Need for verification and resume

- **Use rsync for:**
  - Fast local transfers
  - Reliable networks
  - One-time syncs
  - Speed-critical operations

## Integration

### With Backup Systems

```bash
#!/bin/bash
# Robust backup script
DATE=$(date +%Y%m%d)

# Sync with retry
persync -z ~/Documents/ "backup@server:/backups/docs-$DATE/"

# Verify backup
ssh backup@server "ls -la /backups/docs-$DATE/"

# Cleanup old backups
ssh backup@server "find /backups -name 'docs-*' -mtime +30 -delete"
```bash

### With Monitoring

```bash
# Monitor sync progress
persync ~/large-data/ server:/backup/ &
txmon  # Monitor transfer

# Check sync status
tail -f ~/.local/log/persync.log
```bash

### With Scheduling

```bash
# Cron job for nightly sync
0 2 * * * /usr/local/bin/persync -q ~/data/ backup@server:/nightly/

# Weekly full backup
0 3 * * 0 /usr/local/bin/persync -d ~/archive/ backup@server:/weekly/
```bash

## Security

### Remote Access

- **SSH Keys**: Use key-based authentication
- **Restricted Keys**: Limit key capabilities
- **Firewall**: Restrict SSH access
- **Monitoring**: Log all sync operations

### Data Protection

- **Encryption**: Use SSH for transport encryption
- **Verification**: Checksum verification ensures integrity
- **Access Control**: Proper file permissions
- **Audit Trail**: Comprehensive logging

## Technical Details

### State Management

```json
{
  "version": "1.0",
  "source": "/home/user/data",
  "destination": "user@host:/backup",
  "started": "2024-01-01T00:00:00Z",
  "last_sync": "2024-01-01T01:30:00Z",
  "files_total": 15432,
  "files_synced": 15432,
  "bytes_total": 1073741824,
  "bytes_synced": 1073741824,
  "checksums": {
    "algorithm": "sha256",
    "verified": true
  }
}
```bash

### Retry Algorithm

- **Exponential Backoff**: Delay doubles each retry
- **Jitter**: Random delay to prevent thundering herd
- **Max Delay**: Cap at reasonable maximum
- **Circuit Breaker**: Stop after too many failures

### Conflict Resolution

1. **Compare Timestamps**: Newer file wins
2. **Compare Sizes**: Larger file wins (if timestamps equal)
3. **Manual Resolution**: Prompt user for conflicts
4. **Backup Conflicts**: Keep `.conflict` versions

### Verification

- **Checksums**: SHA256 for all files
- **Metadata**: Verify permissions, timestamps
- **Structure**: Ensure directory structure matches
- **Reporting**: Detailed verification report

## Gotchas

### State File Corruption

**Problem:** Sync resumes from wrong point or behaves unexpectedly.

**Cause:** State file (.persync-state) becomes corrupted or inconsistent.

**Solution:** Remove state file to force fresh sync:

```bash
# Remove corrupted state
rm -f .persync-state

# Start fresh sync
persync source/ dest/
```bash

### Network Interruptions During State Save

**Problem:** State file contains partial information after interruption.

**Cause:** Atomic state file updates not guaranteed during crashes.

**Solution:** Always verify sync completion:

```bash
# Check sync completed successfully
persync --verify source/ dest/

# Or remove state and restart
rm .persync-state && persync source/ dest/
```bash

### Large File Changes Not Detected

**Problem:** Modified large files don't get re-synced.

**Cause:** Timestamp-only change detection misses content changes.

**Solution:** Use checksum verification:

```bash
# Force checksum verification
persync --checksum source/ dest/

# Or remove state to force full recheck
rm .persync-state && persync source/ dest/
```bash

### Permission Conflicts

**Problem:** Files fail to sync due to permission issues.

**Cause:** Source and destination have different permission models.

**Solution:** Check and adjust permissions:

```bash
# Check permissions
ls -la source/ dest/

# Sync with permission preservation
persync --perms source/ dest/
```bash

### Disk Space Exhaustion

**Problem:** Sync fails midway due to insufficient destination space.

**Cause:** Space calculation doesn't account for temporary files.

**Solution:** Ensure 2x source size available:

```bash
# Check space requirements
du -sh source/
df -h dest/

# Clean destination first if needed
persync --delete source/ dest/
```bash

### Remote Host Connection Limits

**Problem:** Sync slows down or fails with "too many connections".

**Cause:** SSH connection limits on remote host.

**Solution:** Reduce concurrency:

```bash
# Limit concurrent connections
persync --concurrency 2 source/ user@host:dest/

# Or configure SSH to allow more connections
ssh user@host "sudo sed -i 's/#MaxSessions.*/MaxSessions 10/' /etc/ssh/sshd_config"
```bash

### Clock Skew Between Systems

**Problem:** Files appear newer/older than they should be.

**Cause:** System clocks not synchronized between local and remote.

**Solution:** Sync system clocks:

```bash
# Sync local clock
sudo ntpdate pool.ntp.org

# For remote host
ssh user@host "sudo ntpdate pool.ntp.org"

# Or use --ignore-time option
persync --ignore-time source/ user@host:dest/
```bash

## See Also

- **[pcopy](pcopy.md)** - High-performance local transfer
- **[txmon](txmon.md)** - Transfer monitoring
- **[Backup Strategies](../../README.md#backup)** - Backup best practices