Performance Optimization

Redis caching, operation timeouts, memory tuning, and performance best practices

Performance Optimization

Optimize Borg UI for large repositories, fast archive browsing, and efficient backups.

---

Redis Cache Configuration

Redis caching provides 600x faster archive browsing for large repositories (60-90 seconds → 100ms).

Quick Performance Comparison

Archive Size	Files	Without Cache	With Redis Cache
Small	1,000	~1 second	100ms
Medium	100,000	~10 seconds	100ms
Large	1,000,000	60-90 seconds	100ms
Very Large	10,000,000	10-15 minutes	1 second

How Caching Works

Architecture (from app/services/cache_service.py):

• Primary backend: Redis (persistent, distributed)
• Fallback backend: In-memory LRU cache (automatic if Redis unavailable)
• Compression: Automatic for archives 100KB (70-80% size reduction)
• Cache key format: archive:{repo_id}:{archive_name}

Cache flow:

1. First browse: Run borg list, serialize JSON, compress if 100KB, store in Redis
2. Subsequent browses: Retrieve from Redis, decompress, deserialize → instant results
3. Expiration: Entries auto-expire after TTL (default: 2 hours)
4. Eviction: LRU eviction when cache size limit reached

Local Redis Setup (Included)

The recommended installation includes Redis with no configuration needed.

docker-compose.yml (already configured):

services:
  redis:
    image: redis:7-alpine
    container_name: borg-redis
    restart: unless-stopped
    command: >
      redis-server
      --maxmemory 2gb
      --maxmemory-policy allkeys-lru
      --save ""
      --appendonly no
    volumes:
      - redis_data:/data

volumes:
  redis_data:

Configuration:

• maxmemory 2gb: Limit Redis memory usage
• allkeys-lru: Evict least-recently-used keys when memory full
• save "": Disable RDB snapshots (cache-only, not persistent)
• appendonly no: Disable AOF (faster, cache can be rebuilt)

Redis is already configured in the recommended installation. No setup needed unless you need more than 2GB cache.

External Redis for Large Repositories

Use cases:

• Repositories with 1M+ files
• Multiple large archives
• Need more than 2GB cache
• Shared cache across multiple Borg UI instances

Setup on dedicated server (NAS/workstation):

# On server with 32GB RAM
docker run -d \
  --name redis-cache \
  --restart unless-stopped \
  -p 6379:6379 \
  -m 24g \
  redis:7-alpine \
  redis-server \
  --maxmemory 20gb \
  --maxmemory-policy allkeys-lru \
  --save "" \
  --appendonly no

Configure in Borg UI:

1. Go to Settings → Cache
2. Enter Redis URL: redis://192.168.1.100:6379/0
3. Increase Max Cache Size to 20480 MB
4. Click Save Settings

Verify connection:

# Check Borg UI logs
docker logs borg-web-ui | grep -i redis
# Should show: "Archive cache initialized with external Redis backend"

# Test from UI
# Settings → Cache → Should show "Backend: Redis" and connection info

Cache Statistics

View in UI:

• Go to Settings → Cache
• Shows: backend type, hit rate, memory usage, entry count, connection info

Implementation (app/services/cache_service.py:270-310):

async def get_stats(self) -> Dict[str, Any]:
    info = client.info("memory")
    dbsize = client.dbsize()
    
    return {
        "backend": "redis",
        "available": True,
        "hits": self._hits,
        "misses": self._misses,
        "hit_rate": (hits / total_requests * 100),
        "size_bytes": info.get("used_memory", 0),
        "entry_count": dbsize
    }

Monitor via Redis CLI:

# Connect to Redis
docker exec -it borg-redis redis-cli

# Check memory usage
INFO memory

# Count cached archives
DBSIZE

# List all archive keys
KEYS archive:*

# Get specific archive size
MEMORY USAGE archive:1:my-backup-2024-02-28

---

Operation Timeouts

Configure timeouts for Borg operations to handle large repositories and slow networks.

Default Timeouts

From app/config.py:90-98 and migration 053_add_operation_timeouts.py:

Operation	Default	Use Case
mount_timeout	120s (2 min)	Archive browsing via FUSE mount
info_timeout	600s (10 min)	Repository info, cache building
list_timeout	600s (10 min)	Listing archives, archive contents
init_timeout	300s (5 min)	Creating new repositories
backup_timeout	3600s (1 hour)	Backup operations
source_size_timeout	3600s (1 hour)	Calculating source size with du

When to Increase Timeouts

Large repositories (100,000 files):

• info_timeout: Increase to 1800s (30 min) for repositories with very large cache
• list_timeout: Increase to 1800s if listing archives takes 10 minutes

Slow networks (remote SSH repositories):

• mount_timeout: Increase to 300s (5 min)
• info_timeout: Increase to 1800s (30 min)
• list_timeout: Increase to 1800s (30 min)

Very large source directories:

• source_size_timeout: Increase to 7200s (2 hours) for 100TB+ sources

Configure Timeouts

Via Web Interface (Recommended):

1. Go to Settings → System
2. Expand Operation Timeouts section
3. Adjust timeout values
4. Click Save Settings

Via Environment Variables:

# docker-compose.yml
environment:
  - BORG_MOUNT_TIMEOUT=300        # 5 minutes
  - BORG_INFO_TIMEOUT=1800        # 30 minutes
  - BORG_LIST_TIMEOUT=1800        # 30 minutes
  - BORG_INIT_TIMEOUT=600         # 10 minutes
  - BACKUP_TIMEOUT=7200           # 2 hours
  - SOURCE_SIZE_TIMEOUT=7200      # 2 hours

UI settings in database take precedence over environment variables (from app/services/backup_service.py:32-72).

Timeout resolution order:

1. Database settings (set via UI)
2. Environment variables
3. Config defaults

---

Memory Optimization

Redis Memory Management

LRU Eviction Policy:

Redis uses allkeys-lru to automatically evict least-recently-used entries when maxmemory limit is reached.

Memory calculation:

Archive files × 200 bytes = Approximate uncompressed size
Compression reduces by ~70-80%
Actual cache size ≈ (files × 200 bytes) × 0.25

Examples:
- 100,000 files  → ~20 MB → ~5 MB compressed
- 1,000,000 files → ~200 MB → ~50 MB compressed  
- 10,000,000 files → ~2 GB → ~500 MB compressed

Set maxmemory based on usage:

# For 10 archives with 1M files each
# 10 archives × 50 MB = 500 MB (minimum)
# Add 50% headroom = 750 MB
# Round up to 1 GB for safety

docker run -d redis:7-alpine redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru

In-Memory Cache Fallback

Configuration (app/services/cache_service.py:340-477):

• Default size: 2GB (configurable via CACHE_MAX_SIZE_MB)
• Automatic eviction: LRU (oldest entries removed when full)
• No persistence: Lost on restart

When fallback activates:

• Redis unavailable (connection refused, timeout)
• Redis health check fails 3+ times consecutively
• Redis URL set to disabled

Monitor fallback:

docker logs borg-web-ui | grep "switching to in-memory cache"

Browse Memory Limits

For very large archives (10M+ files):

Browsing archives with millions of files can consume significant memory during JSON parsing.

Monitor memory usage:

# Check Docker container memory
docker stats borg-web-ui

# Increase container memory limit if needed
docker run -m 4g ...  # 4GB limit

Mitigation strategies:

1. Use Redis cache (parses once, serves from cache)
2. Increase container memory limit
3. Browse subdirectories instead of root
4. Use Borg CLI for very large listings

---

Backup Performance

Parallel Backups

Maximum concurrent backups:

# app/config.py default
MAX_BACKUP_JOBS=5

Increase for powerful servers:

environment:
  - MAX_BACKUP_JOBS=10  # Allow 10 simultaneous backups

Too many concurrent backups can saturate I/O and network. Monitor system resources before increasing.

Source Size Calculation

Purpose: Calculate total source size before backup for accurate progress percentage and ETA.

Implementation (app/services/backup_service.py:433-567):

async def _calculate_source_size(self, source_paths: list[str], exclude_patterns: list[str] = None) -> int:
    """Calculate total size using du command"""
    # Local paths: du -s -B1 --exclude patterns
    # Remote SSH: ssh user@host "du -sb --exclude patterns /remote/path"

Performance impact:

• Local directories: Fast (1 second for small, ~10 seconds for 1TB)
• Remote SSH: Depends on network latency and remote disk speed
• Very large sources (100TB+): Can take 30+ minutes

Optimization:

• Runs in background (doesn’t block backup start)
• Uses configurable timeout (source_size_timeout)
• Skips if timeout exceeded (backup continues without ETA)

Disable size calculation:

# Set very low timeout to skip
environment:
  - SOURCE_SIZE_TIMEOUT=1  # 1 second (will timeout, backup proceeds)

Compression and Deduplication

Borg compression algorithms (set during repository creation):

Algorithm	Speed	Ratio	Use Case
none	Fastest	1:1	Pre-compressed data (videos, images)
lz4	Very fast	~2:1	General purpose, minimal CPU
zstd,3	Fast	~3:1	Recommended default
zstd,10	Moderate	~4:1	More compression, more CPU
zlib,6	Slower	~3.5:1	Legacy compatibility
lzma,6	Slowest	~5:1	Maximum compression

Recommendation:

# Create repository with fast compression
borg init --encryption=repokey-blake2 --compression=zstd,3 /path/to/repo

Custom flags in UI: Go to Repository → Settings → Advanced → Custom Borg Flags:

--compression zstd,3

---

Network Performance

SSH Multiplexing

Enable SSH connection sharing to speed up remote operations:

# Inside container: /home/borg/.ssh/config
Host *
    ControlMaster auto
    ControlPath /tmp/ssh-%r@%h:%p
    ControlPersist 10m
    Compression yes
    ServerAliveInterval 60
    ServerAliveCountMax 3

Apply configuration:

docker exec borg-web-ui tee /home/borg/.ssh/config << 'EOF'
Host *
    ControlMaster auto
    ControlPath /tmp/ssh-%r@%h:%p
    ControlPersist 10m
    Compression yes
EOF

docker exec borg-web-ui chmod 600 /home/borg/.ssh/config

Benefits:

• Reuses SSH connections (no re-authentication overhead)
• Faster for repositories with many small operations
• Reduces network latency

Bandwidth Limiting

Limit backup bandwidth for slow or metered connections:

# Repository → Settings → Advanced → Custom Borg Flags
--remote-ratelimit 5000  # 5 MB/s upload limit

Or via Borg environment:

environment:
  - BORG_REMOTE_RATELIMIT=5000  # 5 MB/s

---

Database Performance

SQLite Optimization

Borg UI uses SQLite for metadata storage.

Current settings:

• WAL mode: Enabled (concurrent reads during writes)
• Journal mode: WAL (Write-Ahead Logging)
• Synchronous: NORMAL (good balance of safety and speed)

Check database size:

docker exec borg-web-ui ls -lh /data/borg.db

Vacuum database (reclaim space):

docker exec borg-web-ui sqlite3 /data/borg.db "VACUUM;"

Analyze for query optimization:

docker exec borg-web-ui sqlite3 /data/borg.db "ANALYZE;"

Migration Performance

Database migrations run on startup (from app/database/migrations/__init__.py:12-55):

def run_migrations():
    """Run all pending migrations in order"""
    migration_files = sorted(migrations_dir.glob("[0-9][0-9][0-9]_*.py"))
    for migration_file in migration_files:
        migration_module.upgrade(connection)
        connection.commit()

Large databases may slow startup:

• 72 migrations execute sequentially
• Each migration checks for existing columns (idempotent)
• First startup: 5-10 seconds
• Subsequent startups: 1 second (migrations already applied)

---

Monitoring and Profiling

Performance Metrics

Cache hit rate:

# Settings → Cache
# Target: >80% hit rate for good performance
# Less than 50% hit rate = increase cache size or TTL

Backup speed:

# View in job logs
docker logs borg-web-ui | grep "Backup speed"

# Example output:
# Backup speed: 125.3 MB/s (original), 45.2 MB/s (compressed)

Repository statistics:

# Run from UI: Repository → Info
# Shows: total size, unique data, compression ratio, archive count

Log Analysis

Find slow operations:

# Find operations greater than 10 seconds
docker logs borg-web-ui | grep "duration" | awk '$NF > 10000'

# Find timeout errors
docker logs borg-web-ui | grep -i timeout

Enable debug logging:

environment:
  - LOG_LEVEL=DEBUG

Debug logging is verbose and may impact performance. Use only for troubleshooting.

---

Performance Tuning Checklist

Large Repository Optimization (1M+ files)

Cache:

• External Redis with 8-20GB memory
• Cache size: 8192-20480 MB
• Cache TTL: 1440 minutes (24 hours)
• Monitor hit rate 80%

Timeouts:

• info_timeout: 1800s (30 min)
• list_timeout: 1800s (30 min)
• mount_timeout: 300s (5 min)
• source_size_timeout: 7200s (2 hours)

Borg:

• Compression: zstd,3 or lz4
• Encryption: repokey-blake2 (faster than SHA256)
• Regular borg compact to reclaim space

Remote Repository Optimization (SSH)

SSH:

• SSH multiplexing enabled (ControlMaster auto)
• Compression enabled in SSH config
• SSH key authentication (no password delays)
• Dedicated backup user (no shell overhead)

Network:

• Bandwidth limiting if needed (--remote-ratelimit)
• Consider using local mount for very large datasets

Timeouts:

• Increase all timeouts by 2-3x for remote operations
• Monitor for timeout errors in logs

Container Resource Limits

Memory:

• Borg UI: 2-4GB for large archive browsing
• Redis: Match cache size + 500MB overhead

CPU:

• No limit for backup operations (compression benefits from all cores)
• Consider limiting for background jobs only

Example:

services:
  borg-ui:
    deploy:
      resources:
        limits:
          memory: 4G
        reservations:
          memory: 2G
  
  redis:
    deploy:
      resources:
        limits:
          memory: 20G
        reservations:
          memory: 10G

---

Related Documentation

Cache Configuration

Detailed Redis setup and troubleshooting

Troubleshooting

Performance issues and solutions

Maintenance

Database optimization and cleanup

Configuration

Environment variables and system settings