Async SmugMug to S3 Streamer

A high-performance asynchronous application for streaming photos directly from SmugMug to S3-compatible storage without requiring local disk space.

Features

• Direct Streaming: Downloads and uploads images in a single pass without local storage
• High Performance: Concurrent processing with configurable concurrency limits
• Smart Caching: SQLite-based tracking prevents duplicate transfers and enables crash recovery
• Metadata Preservation: Transfers SmugMug metadata (title, caption, keywords) to S3 object metadata
• Folder Structure: Maintains SmugMug album/folder hierarchy in S3
• Multiple Auth Methods: Supports OAuth 1.0a and API key authentication
• Rate Limiting: Built-in rate limiting to respect API quotas
• Batch Processing: Processes images in configurable batches for optimal performance

Requirements

• Python 3.13 or higher
• uv - Fast Python package manager
• SmugMug account with API credentials
• S3-compatible storage (AWS S3, B2, MinIO, etc.)

Installation

1. Clone the repository and navigate to the project directory:

cd smugmug-to-s3

2. Install dependencies using uv:

uv sync

Configuration

1. SmugMug OAuth Authentication (Recommended)

Generate OAuth tokens using the included console helper:

uv run python console.py

This will guide you through the OAuth flow and provide you with the necessary tokens.

2. Environment Variables

Create a .env file in the project directory with the following variables:

# SmugMug OAuth Credentials (Recommended)
SMUGMUG_OAUTH_CONSUMER_KEY=your_consumer_key
SMUGMUG_OAUTH_CONSUMER_SECRET=your_consumer_secret
SMUGMUG_OAUTH_TOKEN=your_oauth_token
SMUGMUG_OAUTH_TOKEN_SECRET=your_oauth_token_secret

# OR SmugMug API Key (Alternative)
SMUGMUG_API_KEY=your_api_key

# S3 Configuration (Required)
S3_ACCESS_KEY=your_s3_access_key
S3_SECRET_KEY=your_s3_secret_key
S3_BUCKET=your_bucket_name
S3_ENDPOINT_URL=https://your-s3-endpoint.com  # Optional, for S3-compatible services
S3_REGION=us-east-1  # Optional, defaults to us-east-1
S3_PREFIX=smugmug/  # Optional, defaults to smugmug/

# Performance Tuning (Optional)
MAX_CONCURRENT_DOWNLOADS=20  # Default: 20
BATCH_SIZE=50  # Default: 50
RATE_LIMIT_DELAY=0.01  # Default: 0.01 seconds

3. OAuth Consumer Key/Secret

To use the OAuth authentication flow, you need a consumer key and secret. You can obtain these by:

1. Visit SmugMug API
2. Register your application
3. Add the credentials to your .env file as SMUGMUG_OAUTH_CONSUMER_KEY and SMUGMUG_OAUTH_CONSUMER_SECRET

These credentials are required to run console.py for generating OAuth access tokens.

Usage

Basic Usage

Run the streamer to transfer all albums:

uv run python main.py

The application will:

1. Authenticate with SmugMug
2. Discover all albums in your account
3. Stream all images to your S3 bucket
4. Preserve folder structure and metadata

Testing Authentication

Test your SmugMug credentials without performing a full transfer:

uv run python -c "from main import *; config = load_config(); import asyncio; asyncio.run(AsyncSmugMugS3Streamer(config).get_user_info_async(asyncio.run(AsyncSmugMugS3Streamer(config).setup_smugmug_session())))"

Debugging

Run with interactive debugger:

uv run python -c "import ipdb; ipdb.set_trace(); exec(open('main.py').read())"

Performance Tuning

Adjust these environment variables for optimal performance:

• MAX_CONCURRENT_DOWNLOADS: Number of simultaneous downloads (default: 20)

  • Higher values = faster transfers but more memory usage
  • Recommended range: 10-50 depending on your bandwidth

• BATCH_SIZE: Number of images to process before waiting for completion (default: 50)

  • Larger batches = better throughput but more memory usage
  • Recommended range: 20-100

• RATE_LIMIT_DELAY: Delay between API calls in seconds (default: 0.01)

  • Increase if you encounter rate limiting errors
  • Decrease for faster transfers if your API quota allows

Example for maximum speed on a fast connection:

MAX_CONCURRENT_DOWNLOADS=50
BATCH_SIZE=100
RATE_LIMIT_DELAY=0.01

Crash Recovery

The application maintains a SQLite database (transfer_log.db) that tracks:

• Transferred files: Prevents duplicate uploads
• Album cache: Tracks which albums have been processed
• File metadata: Caches SmugMug file information

If the application crashes or is interrupted, simply restart it. The application will:

• Skip already-transferred files
• Resume from incomplete albums
• Continue where it left off

To force a fresh start, delete transfer_log.db (this will re-transfer all files).

Data Storage

The application creates the following local files:

• transfer_log.db: SQLite database for tracking transfers and caching
• .env: Your configuration and credentials (including OAuth consumer key/secret and tokens)

S3 Folder Structure

Images are organized in S3 as:

s3://your-bucket/
    smugmug/  (or your S3_PREFIX)
        Album Name 1/
            image1.jpg
            image2.jpg
        Folder/Album Name 2/
            image3.jpg
            image4.jpg

Metadata Preservation

The following SmugMug metadata is transferred to S3 object metadata:

• Title
• Caption
• Date
• Filename
• Format
• Keywords
• Watermarked status
• Hidden status
• Original MD5 hash
• Original file size

Access this metadata using the S3 API or AWS CLI:

aws s3api head-object --bucket your-bucket --key smugmug/Album/image.jpg

Troubleshooting

Authentication Errors

If you encounter authentication errors:

1. Verify your .env file has the correct credentials
2. Run uv run python console.py to regenerate OAuth tokens
3. Ensure your OAuth tokens haven't expired

Rate Limiting

If you see rate limiting errors (HTTP 429):

• Increase the RATE_LIMIT_DELAY value in your .env file
• Reduce MAX_CONCURRENT_DOWNLOADS to make fewer simultaneous requests

Out of Memory

If the application runs out of memory:

• Reduce MAX_CONCURRENT_DOWNLOADS
• Reduce BATCH_SIZE
• Process albums individually by modifying the code

Connection Timeouts

If downloads timeout frequently:

• Check your internet connection stability
• The application automatically retries failed transfers
• Restart the application to resume from where it stopped

Development

The codebase consists of:

• main.py: Core async streaming logic and SmugMug API integration
• common.py: OAuth authentication utilities
• console.py: Interactive OAuth token generation tool

Key classes:

• AsyncSmugMugS3Streamer: Main application class with async methods for SmugMug API interaction and S3 uploads

Typical performance:

• 20-100 images per minute (depending on network speed and image sizes)
• Minimal local disk usage (only SQLite database)
• Scales well with faster internet connections

License

See repository root for license information.