Template-Based Configuration System ¶

Overview ¶

The backup system uses a template substitution approach to pass configuration from the launching server to the EC2 backup instance. This solves the problem of .env files only existing on the launching server, not on the EC2 instance.

Architecture ¶

┌─────────────────────┐
│  Launching Server   │
│                     │
│  1. Read .env file  │
│  2. Read template   │
│  3. Substitute vars │
│  4. Launch EC2      │
└──────────┬──────────┘
           │
           │ user-data (cloud-config)
           │ with substituted values
           ▼
┌─────────────────────┐
│   EC2 Instance      │
│                     │
│  cloud-init runs    │
│  ├─ Sets env vars   │
│  ├─ Creates files   │
│  └─ Runs backup.py  │
│                     │
│  backup.py reads    │
│  environment vars   │
└─────────────────────┘

How It Works ¶

1. On the Launching Server ¶

The launch_backup_instance.py script:

1. Loads .env file - Reads configuration from local .env file
2. Reads template - Opens cloud-config.yaml.template
3. Substitutes placeholders - Replaces {{VARIABLE}} with actual values
4. Generates user-data - Creates final cloud-config with real values
5. Launches EC2 - Passes generated user-data to instance

Example substitution:

# Template
REMOTE_HOST={{REMOTE_HOST}}

# After substitution (with REMOTE_HOST=178.22.67.203 from .env)
REMOTE_HOST=178.22.67.203

2. On the EC2 Instance ¶

When the instance launches:

1. cloud-init runs - Processes the substituted cloud-config
2. Environment file created - /etc/environment.d/backup.conf with all variables
3. Backup script runs - efs_backup.py or mkw_backup.py reads environment variables
4. No .env needed - All configuration comes from environment set by cloud-init

File Structure ¶

Launch Scripts (run on local machine) ¶

• launch_backup_instance.py - Reads .env, substitutes template, launches EC2

Templates (local) ¶

• cloud-config.yaml.template - Contains {{PLACEHOLDER}} variables

Configuration (local) ¶

• .env - Your actual configuration (not in git)
• .env.example - Template showing all available variables

Backup Scripts (run on EC2) ¶

• efs_backup.py / mkw_backup.py - Use os.environ.get() to read config

Generated (on EC2 at runtime) ¶

• /etc/environment.d/backup.conf - Environment variables created by cloud-init

Configuration Flow ¶

.env (local)
    ↓
launch_backup_instance.py
    ↓
cloud-config.yaml.template → cloud-config (generated)
    ↓
EC2 user-data
    ↓
cloud-init
    ↓
/etc/environment.d/backup.conf
    ↓
backup script reads env vars

Benefits ¶

1. Single Source of Truth - Configuration lives in one place (.env)
2. No Remote Dependencies - No need to fetch files from S3 or remote servers
3. Secure - Sensitive data (SSH keys, tokens) passed securely via user-data
4. Flexible - Easy to override any value via .env
5. Testable - Can generate cloud-config without launching EC2

Configuration Variables ¶

EC2 Launch Configuration ¶

These control HOW the instance is launched:

• AWS_REGION
• IMAGE_ID
• INSTANCE_TYPE
• SECURITY_GROUP_ID
• KEY_NAME
• AVAILABILITY_ZONE
• IAM_INSTANCE_PROFILE

Backup Configuration ¶

These are passed TO the backup script on EC2:

Common (both EFS and MKW) ¶

• AWS_REGION
• SSM_SNAPSHOT_PARAMETER
• LOGFILE
• MOUNT_POINT
• RSNAPSHOT_CONF
• TELEGRAM_BIN
• TELEGRAM_TOKEN
• TELEGRAM_CHAT_ID
• HEALTHCHECK_URL
• S3_LOG_BUCKET
• DEVICE_NAME
• NVME_DEVICE
• NVME_PARTITION
• SIZE_RATIO_THRESHOLD
• RESIZE_MULTIPLIER
• MAX_RETRIES
• RETRY_DELAY

MKW-Specific ¶

• REMOTE_HOST
• REMOTE_PORT
• REMOTE_USER
• SSH_PRIVATE_KEY

EFS-Specific ¶

• EFS_FILESYSTEM_ID
• EFS_MOUNT_POINT

Usage ¶

Setup ¶

1. Copy example configuration:

   cd efs-backup-improved  # or mkw-backup-improved
   cp .env.example .env
   

2. Edit .env with your values:

   # Required: Update these
   EFS_FILESYSTEM_ID=fs-YOUR-ID
   TELEGRAM_TOKEN=your-bot-token
   TELEGRAM_CHAT_ID=your-chat-id
   
   # Optional: Override defaults if needed
   INSTANCE_TYPE=t3.large
   SIZE_RATIO_THRESHOLD=1.5
   

3. Launch backup:

   ./launch_backup_instance.py
   

Debugging ¶

To see the generated cloud-config without launching:

#!/usr/bin/env python3
from pathlib import Path
import os

# Load .env
env_file = Path('.env')
if env_file.exists():
    with open(env_file) as f:
        for line in f:
            line = line.strip()
            if line and not line.startswith('#') and '=' in line:
                key, value = line.split('=', 1)
                os.environ[key] = value

# Read template
with open('cloud-config.yaml.template') as f:
    template = f.read()

# Substitute
BACKUP_CONFIG = {key: os.environ.get(key, 'DEFAULT') for key in [...]}
for key, value in BACKUP_CONFIG.items():
    template = template.replace(f"{{{{{key}}}}}", str(value))

# Print result
print(template)

Template Syntax ¶

Templates use double curly braces for placeholders:

# Basic substitution
remote_host: {{REMOTE_HOST}}

# In write_files content
content: |
  AWS_REGION={{AWS_REGION}}
  LOGFILE={{LOGFILE}}

# In commands
- [mount, {{EFS_FILESYSTEM_ID}}.efs.{{AWS_REGION}}.amazonaws.com]

Security Considerations ¶

1. Never commit .env - Add to .gitignore
2. SSH Keys - Pass via environment, written to /root/.ssh/ with proper permissions
3. Tokens - Stored in environment, not in files
4. User-data - Transmitted securely by AWS, not logged

Troubleshooting ¶

Problem: Variables not substituted ¶

Check:

• Variable is in .env file
• Variable is in BACKUP_CONFIG dict in launch_backup_instance.py
• Template has correct {{VARIABLE}} syntax (double braces, exact name)

Problem: Backup script can't find configuration ¶

Check:

• /etc/environment.d/backup.conf exists on EC2 instance
• cloud-init completed successfully: tail /var/log/cloud-init-output.log
• Variable names match between template and backup script

Problem: Template not found ¶

Check:

• File is named cloud-config.yaml.template (not cloud-config.yaml)
• File is in same directory as launch_backup_instance.py

Migration from v2.2 ¶

If you have existing v2.2 scripts with hardcoded values:

1. Backup current cloud-config.yaml
2. Identify all hardcoded values (IPs, filesystem IDs, etc.)
3. Add them to .env
4. Use new template - Already has placeholders
5. Test launch - Verify substitution worked

Future Enhancements ¶

Possible improvements:

• Validate required variables before launch
• Support multiple environments (dev, staging, prod)
• Template includes for shared sections
• Variable interpolation (e.g., ${VAR1}/${VAR2})