Mnemonica Kubernetes Development Environment ¶

This repository contains Kubernetes configuration files for the Mnemonica development environment running on AWS EKS.

Table of Contents ¶

1. Architecture Overview
2. Core Services
3. Helm-Managed Services
4. Storage
5. Networking
6. Security
7. Monitoring & Observability
8. Deployment

Architecture Overview ¶

Cluster Information ¶

• Platform: AWS EKS
• Region: eu-west-1
• Cluster Name: dev
• Namespace: default (primary)
• Environment: testing/development

Application Architecture ¶

Mnemonica is a SaaS Media Asset Management (MAM) platform with the following architecture:

┌─────────────────────────────────────────────────────────────┐
│                        External Access                       │
│          (devel.mnemonica.com, devtools.mnemonica.com)      │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                    AWS Load Balancer                         │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────────┐
│                      Frontend (Nginx)                       │
│                   NodePort: 30001                           │
│                   Image: infomne/frontend                   │
└─────┬──────────────────────────────────────────────────────┘
      │
      ├──────────────┬──────────────┬──────────────┬─────────────┐
      ▼              ▼              ▼              ▼             ▼
┌─────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐  ┌────────────┐
│ Backend │   │ Centrifugo│  │  TUSD    │   │  Tools   │  │  Gatesrv   │
│ (uWSGI) │   │(WebSocket)│  │ (Upload) │   │  (Nginx) │  │  (Nginx)   │
└────┬────┘   └──────────┘   └─────┬────┘   └──────────┘  └────────────┘
     │                              │
     ├──────────────────────────────┴──────────────────┐
     ▼              ▼              ▼                    ▼
┌─────────┐   ┌──────────┐   ┌──────────────┐   ┌──────────────────┐
│  Celery │   │Celerybeat│   │Celery-Encoding│  │TUS Hook Listener │
│ Workers │   │(Scheduler)│  │   Workers     │   │                  │
└────┬────┘   └──────────┘   └──────┬───────┘   └──────────────────┘
     │                              │
     │         ┌────────────────────┴────────────────────┐
     │         ▼                                          ▼
     │   ┌──────────────────┐                  ┌──────────────────┐
     │   │ Encoding Pool    │                  │Traffic Log       │
     │   │    Master        │                  │   Listener       │
     │   └────────┬─────────┘                  └──────────────────┘
     │            │
     │            ▼
     │   ┌──────────────────┐
     │   │ Encoding Workers │
     │   │ (K8s Jobs)       │
     │   └──────────────────┘
     │
     ├──────────────┬──────────────┬──────────────┐
     ▼              ▼              ▼              ▼
┌─────────┐   ┌──────────┐   ┌─────────┐   ┌──────────┐
│  Redis  │   │ RabbitMQ │   │PostgreSQL│  │  Flower  │
│         │   │          │   │(External?)│  │(Monitor) │
└─────────┘   └──────────┘   └─────────┘   └──────────┘

Core Services ¶

Application Services ¶

Backend (backend-deployment-service.yaml) ¶

• Image: infomne/uwsgi-apps:v2.14.6-1-gab90065
• Purpose: Main Django/uWSGI application backend
• Port: 3031
• Replicas: 1
• Features:
  • Anti-affinity rules for high availability
  • Shared media storage via JuiceFS
  • Firebase Service Account integration
  • Centrifugo integration for real-time features
  • PodDisruptionBudget with minAvailable: 1

Frontend (frontend-deployment-service.yaml) ¶

• Image: infomne/frontend:v2.14.3-51-g580e75e
• Purpose: Nginx reverse proxy and static content serving
• Type: NodePort (30001)
• Replicas: 1
• Features:
  • Dynamic nginx configuration via ConfigMap template
  • Environment variable substitution at runtime
  • Shared media storage access
  • Signed links support

Celery Workers (celery-deployment.yaml) ¶

• Image: infomne/uwsgi-apps:v2.14.6-1-gab90065
• Purpose: Background task processing
• Replicas: 1
• Features:
  • Connected to RabbitMQ broker
  • Shared media storage access
  • Anti-affinity for distribution

Celery Beat (celerybeat-deployment.yaml) ¶

• Purpose: Scheduled task management
• Replicas: 1
• Features: Periodic task scheduler for Celery

Celery Encoding Workers (celery-encoding-deployment.yaml) ¶

• Purpose: Specialized workers for video/media encoding tasks
• Replicas: 1
• Encoding Configuration:
  • Format: MP4
  • Video Codec: h264 (pool: h264_nvenc with GPU acceleration)
  • Audio Codec: AAC
  • Video Bitrate: 1840000
  • Audio Bitrate: 160000
  • Height: 720p
  • Two-pass encoding enabled
  • Hardware acceleration: auto

Encoding Pool Master (encoding-pool-master-deployment.yaml) ¶

• Purpose: Manages dynamic encoding worker jobs in Kubernetes
• Replicas: 1
• Features:
  • Service Account with K8s API access (epmaster)
  • kubectl installed via init container
  • Spawns K8s Jobs for encoding workers
  • Max workers: 2
  • Worker start timeout: 180s

TUS Hook Listener (tus-hook-listener-deployment-service.yaml) ¶

• Purpose: Handles file upload completion webhooks from TUSD
• Port: 5634 (TCP)
• Replicas: 1

Traffic Log Listener (traffic-log-listener-deployment-service.yaml) ¶

• Purpose: Receives and processes syslog/traffic logs
• Port: 5140 (UDP)
• Replicas: 1

Flower (flower-deployment-service.yaml) ¶

• Purpose: Celery monitoring and management UI
• Replicas: 1

Utility Services ¶

n8n (n8n-deployment-service.yaml) ¶

• Image: docker.n8n.io/n8nio/n8n:latest
• Purpose: Workflow automation platform
• Type: NodePort (30004)
• Domain: n8n-dev.mnemonica.io
• Replicas: 1
• Storage: 10Gi EBS (gp2) at /home/node/.n8n
• Features:
  • Reloader integration for automatic restarts
  • SMTP integration via AWS SES
  • Security context with fsGroup for volume permissions
  • Strategy: Recreate (for SQLite database consistency)
• Related Files:
  • n8n-env-ConfigMap.yaml: Environment configuration
  • n8n-pv-pvc.yaml: Persistent volume claim
  • n8n-tgb.yaml: Target group binding

Infrastructure Services ¶

Redis (redis.yaml) ¶

• Image: redis:8.0-bookworm
• Purpose: Caching and session storage
• Service: redis-master (ClusterIP)
• Port: 6379
• Replicas: 1

RabbitMQ (Cluster Operator) ¶

• Purpose: Message broker for Celery tasks
• Operator: RabbitMQ Cluster Operator (rabbitmq.com/v1beta1)
• Image: rabbitmq:3.13-management
• Service: broker (ClusterIP), broker-nodes (Headless)
• Replicas: 3
• Storage: gp3 EBS (10Gi per pod, WaitForFirstConsumer)
• Virtual Hosts:
  • mnemonica.tasks (Celery tasks)
  • mnemonica.service (Service broker)
  • mnemonica.app (Application)
• Config: See /home/jeremy/Work/ClaudeCode/Mnemonica/rabbitmq/development/

Helm-Managed Services ¶

The following services are deployed via Helm charts:

Centrifugo (Real-time Messaging) ¶

• Chart: centrifugo-12.3.7
• App Version: 6.2.5
• Purpose: WebSocket server for real-time notifications and messaging
• Namespaces:
  • mfa: Multi-factor authentication channels
  • user: User-specific channels with history (10 messages, 30s TTL)
  • gate: Presence and join/leave tracking
  • signal: General signaling
• Features:
  • Admin panel enabled (insecure mode for dev)
  • CORS enabled for all origins

TUSD (File Upload Server) ¶

• Chart: tusd (v2.5.0)
• Purpose: Resumable file upload server (TUS protocol)
• Port: 1080
• Features:
  • Behind proxy mode
  • Upload directory: /var/www/media/mnemonica/storage/tus
  • Downloads disabled
  • Webhook integration with tus-hook-listener
  • Init container removes stale lock files
  • Runs as user 33 (www-data)

AWS Load Balancer Controller ¶

• Chart: aws-load-balancer-controller-1.17.1
• App Version: v2.17.1
• Namespace: kube-system
• Purpose: Manages AWS ALB/NLB for Kubernetes Ingress resources

Karpenter (Node Provisioning) ¶

• Purpose: Provisions EC2 nodes directly (~60s) for pending pods
• NodePools: general (ARM t4g), gpu (g4dn)
• Replaces: Cluster Autoscaler
• See AUTOSCALING.md

KEDA (Queue-Based Autoscaling) ¶

• Namespace: keda
• Purpose: Scales celery workers based on RabbitMQ queue depth
• See AUTOSCALING.md

JuiceFS CSI Driver ¶

• Chart: juicefs-csi-driver-0.31.0
• Namespace: kube-system
• Purpose: Provides shared filesystem storage via JuiceFS

Reloader ¶

• Chart: reloader-2.2.7
• App Version: v1.4.12
• Purpose: Automatically reloads deployments when ConfigMaps or Secrets change
• Annotation: reloader.stakater.com/search: "true" or reloader.stakater.com/match: "true"

Portainer ¶

• Chart: portainer-2.33.6
• App Version: ce-latest-ee-2.33.6
• Namespace: portainer
• Purpose: Kubernetes management UI

Amazon CloudWatch Observability ¶

• Chart: amazon-cloudwatch-observability-4.8.1
• Namespace: amazon-cloudwatch
• Purpose: CloudWatch integration for logs and metrics

Storage ¶

JuiceFS Persistent Volumes ¶

All persistent storage uses JuiceFS CSI driver with ReadWriteMany access mode.

dev-media PV/PVC (dev-media-pv-pvc.yaml) ¶

• Purpose: Shared media files storage
• Access Mode: ReadWriteMany
• Capacity: 10Pi (JuiceFS doesn't enforce capacity)
• Reclaim Policy: Retain
• Mount Options: cache-size=5120
• Secret: juicefs-secret
• Used By: backend, frontend, celery, encoding-pool-master, tusd, tus-hook-listener

dev-rabbitmq PV/PVC (rabbitmq-pv-pvc.yaml) ¶

• Purpose: RabbitMQ data persistence
• Access Mode: ReadWriteMany
• Capacity: 10Pi
• Reclaim Policy: Retain
• Mount Options: subdir=/mnesia/
• Secret: juicefs-rabbitmq

logs PV/PVC (logs-pv-pvc.yaml) ¶

• Purpose: Application logs storage
• Storage Backend: JuiceFS

Networking ¶

Services ¶

NodePort Services ¶

• frontend: Port 30001 (HTTP) - Primary application entry point
• n8n: Port 30004 (HTTP) - Workflow automation (n8n-dev.mnemonica.io)

ClusterIP Services (Internal) ¶

• backend: Port 3031 (TCP)
• redis-master: Port 6379 (TCP)
• tus-hook-listener: Port 5634 (TCP)
• traffic-log-listener: Port 5140 (UDP)

External Access & Load Balancing ¶

AWS Application Load Balancer (ALB) handles external traffic routing to the cluster.

DNS Configuration ¶

• Primary Domain: devel.mnemonica.com
• Tools Domain: devtools.mnemonica.com
• Alternative Domain: dev.mnemonica.com
• n8n Domain: n8n-dev.mnemonica.io
• Load Balancer: dev-external-lb.mnemonica.com
• CloudFront: d3qq7s3kaq01hx.cloudfront.net

AWS Target Groups ¶

The AWS Load Balancer Controller automatically creates and manages target groups based on the Target Group Binding (TGB) resources:

• frontend-tgb.yaml: Routes traffic to frontend service (NodePort 30001)
• gatesrv-tgb.yaml: Routes traffic to gatesrv service
• tools-tgb.yaml: Routes traffic to tools service
• n8n-tgb.yaml: Routes traffic to n8n service (NodePort 30004, host: n8n-dev.mnemonica.io)

The ALB is configured with:

• Health checks for each target group
• SSL/TLS termination (HTTPS)
• Path-based or host-based routing rules

For detailed ALB configuration, refer to the infrastructure documentation in MU-131025-152516.pdf.

Security ¶

Image Pull Secrets ¶

• regcred: Docker registry credentials for pulling images from private registry

Secrets (Referenced, not exposed) ¶

The following secrets are referenced in deployments (stored in linked secrets directory):

• uwsgi: Backend application secrets (DATABASE_URL for AWS RDS PostgreSQL, SECRET_KEY, AWS credentials, etc.)
• centrifugo: WebSocket server keys (tokenHmacSecretKey, apiKey)
• fsa: Firebase Service Account JSON for push notifications
• juicefs-secret: JuiceFS volume credentials for shared media storage
• juicefs-rabbitmq: JuiceFS credentials for RabbitMQ persistent volume

For detailed secrets configuration, see SECRETS.md

Service Accounts ¶

epmaster (encoding-pool-master) ¶

• Purpose: Allows encoding pool master to create/manage K8s Jobs
• Resources:
  • support/epmaster-service-account.yaml
  • support/epmaster-rolebinding.yaml

epw-check ¶

• Purpose: Encoding pool worker health checks
• Resources:
  • support/epw-check-service-account.yaml
  • support/epw-check-role.yaml
  • support/epw-check-role-binding.yaml

gatedeploy ¶

• Purpose: Gate deployment automation
• Resource: support/gatedeploy-sa.yaml

rollout-restart-cronjob ¶

• Purpose: Automated deployment restarts
• Resources:
  • support/rollout-restart-cronjob-sa.yaml
  • support/rollout-restart-role.yaml
  • support/rollout-restart-rolebinding.yaml

helm-version-monitor ¶

• Purpose: Monitor Helm chart versions
• Resources:
  • support/helm-version-monitor-sa.yaml
  • support/helm-version-monitor-role.yaml
  • support/helm-version-monitor-rolebinding.yaml
  • support/helm-version-monitor-cronjob.yaml

Pod Security ¶

Anti-Affinity Rules ¶

Services use preferredDuringSchedulingIgnoredDuringExecution podAntiAffinity to spread pods across nodes while allowing colocation when needed for autoscaling:

• backend
• frontend
• celery
• celery-encoding
• tus-hook-listener
• traffic-log-listener

Pod Disruption Budget (minAvail1-pdb.yaml) ¶

• Policy: Ensures at least 1 pod is available during voluntary disruptions
• Applies to: All pods with label pdb: minAvail1

Monitoring & Observability ¶

Amazon CloudWatch ¶

• Namespace: amazon-cloudwatch
• Purpose: Collects logs and metrics from the cluster
• Chart: amazon-cloudwatch-observability-4.8.1
• Features:
  • Container logs forwarding
  • Performance metrics collection
  • Integration with AWS CloudWatch dashboards

Flower (Celery Monitoring) ¶

• Purpose: Real-time Celery task monitoring
• Features:
  • Worker statistics and management
  • Task history and status
  • Broker monitoring
• Access: Via port-forward or internal service

Note: For detailed monitoring configuration and dashboard access, consult your AWS CloudWatch documentation.

Configuration Management ¶

ConfigMaps ¶

uwsgi-env (uwsgi-env-ConfigMap.yaml) ¶

Main application configuration including:

• Deployment environment: testing
• Site URL: https://devel.mnemonica.com
• Database and broker URLs
• Media/static/storage paths
• Encoding parameters
• Email configuration (mailpit)
• Redis cache URL
• Centrifugo host
• JWT configuration
• FCM configuration
• EKS cluster information
• Allowed hosts

frontend-env (frontend-env-ConfigMap.yaml) ¶

Frontend-specific environment variables

tools-env (tools-env-ConfigMap.yaml) ¶

Tools service configuration

Nginx Configuration Templates ¶

• frontend-nginx.conf.tpl-ConfigMap.yaml: Frontend Nginx template
• gatesrv-nginx.conf-ConfigMap.yaml: Gatesrv Nginx configuration
• tools-nginx.conf.tpl-ConfigMap.yaml: Tools Nginx template
• tools-index-file-ConfigMap.yaml: Tools landing page

encoding-pool-worker-job.tpl-ConfigMap.yaml ¶

Template for dynamically created encoding worker Jobs

fluent-bit-config-ConfigMap.yaml ¶

Log forwarding configuration

Deployment ¶

Prerequisites ¶

1. AWS EKS cluster (dev) in eu-west-1
2. kubectl configured with cluster access
3. Helm 3.x installed
4. Required secrets configured in secrets directory
5. JuiceFS volumes configured

Deployment Order ¶

1. Install Helm Charts (Infrastructure)

   # JuiceFS CSI Driver
   helm upgrade -i juicefs-csi-driver -n kube-system -f custom-values/juicefs-custom-values-dev.yaml
   
   # AWS Load Balancer Controller
   helm upgrade -i aws-load-balancer-controller -n kube-system
   
   # Karpenter (replaces Cluster Autoscaler, see docs/AUTOSCALING.md)
   helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
     --namespace kube-system -f karpenter/karpenter-values-dev.yaml
   kubectl apply -f karpenter/
   
   # Reloader
   helm upgrade -i reloader-1707990814
   

2. Deploy Storage

   kubectl apply -f dev-media-pv-pvc.yaml
   kubectl apply -f rabbitmq-pv-pvc.yaml
   kubectl apply -f logs-pv-pvc.yaml
   

3. Deploy Infrastructure Services

   # Redis
   kubectl apply -f redis.yaml
   
   # RabbitMQ (cluster operator — see rabbitmq/development/README.md)
   kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
   
   # Centrifugo
   helm upgrade -i centrifugo -f custom-values/centrifugo-custom-values-dev.yaml
   
   # TUSD
   helm upgrade -i tusd -f custom-values/tusd-custom-values-dev.yaml
   
   # Mailpit (dev email testing)
   kubectl apply -f mailpit.yaml
   

4. Deploy ConfigMaps

   kubectl apply -f uwsgi-env-ConfigMap.yaml
   kubectl apply -f frontend-env-ConfigMap.yaml
   kubectl apply -f frontend-nginx.conf.tpl-ConfigMap.yaml
   kubectl apply -f encoding-pool-worker-job.tpl-ConfigMap.yaml
   

5. Deploy Pod Disruption Budget

   kubectl apply -f minAvail1-pdb.yaml
   

6. Deploy Application Services

   kubectl apply -f backend-deployment-service.yaml
   kubectl apply -f frontend-deployment-service.yaml
   kubectl apply -f celery-deployment.yaml
   kubectl apply -f celerybeat-deployment.yaml
   kubectl apply -f celery-encoding-deployment.yaml
   kubectl apply -f encoding-pool-master-deployment.yaml
   kubectl apply -f tus-hook-listener-deployment-service.yaml
   kubectl apply -f traffic-log-listener-deployment-service.yaml
   kubectl apply -f flower-deployment-service.yaml
   

7. Deploy Autoscaling

   # HPA for HTTP workloads
   kubectl apply -f hpa/dev/
   
   # KEDA for queue-based scaling (see docs/AUTOSCALING.md for RabbitMQ secret setup)
   helm install keda kedacore/keda --namespace keda --create-namespace \
     -f keda/keda-values-dev.yaml
   kubectl apply -f keda/dev/
   

8. Deploy Target Group Bindings

   kubectl apply -f frontend-tgb.yaml
   kubectl apply -f gatesrv-tgb.yaml
   kubectl apply -f tools-tgb.yaml
   

9. Deploy Monitoring

   helm upgrade -i amazon-cloudwatch-observability -n amazon-cloudwatch \
     -f custom-values/amazon-cloudwatch-observability-values-dev.yaml
   
   helm upgrade -i portainer -n portainer -f custom-values/portainer-custom-values-dev.yaml
   

Verification ¶

# Check all pods are running
kubectl get pods

# Check services
kubectl get svc

# Check PVCs are bound
kubectl get pvc

# Check Helm releases
helm list -A

# Check logs
kubectl logs -l app=backend --tail=50

Directory Structure ¶

.
├── docs/                           # This documentation
├── custom-values/                  # Helm chart custom values
├── support/                        # Service accounts, roles, utilities
├── secrets/                        # Symlink to secrets repository
├── test/                          # Test configurations (ignored)
├── unused/                        # Deprecated configurations (ignored)
├── *-deployment-service.yaml      # Service deployments
├── *-ConfigMap.yaml               # Configuration maps
├── *-pv-pvc.yaml                  # Persistent volume claims
└── *-tgb.yaml                     # Target group bindings

Maintenance ¶

Updating Application Version ¶

1. Update image tag in deployment files
2. Apply changes: kubectl apply -f <deployment-file>.yaml
3. Reloader will automatically restart pods

Scaling ¶

HPA-managed workloads (backend, frontend) and KEDA-managed workloads (celery, celery-encoding) scale automatically. See AUTOSCALING.md.

For manual overrides:

kubectl scale deployment backend --replicas=2

Checking Logs ¶

kubectl logs -f deployment/backend
kubectl logs -f deployment/celery

Accessing Portainer ¶

Portainer provides a web UI for managing the cluster (exact access method TBD).

Important Notes ¶

Environment Characteristics ¶

• Environment Type: Development/Testing
• Namespace: All services run in default namespace
• Security: Some settings are relaxed for development (e.g., Centrifugo admin insecure mode)

External Dependencies ¶

• Database: PostgreSQL running on AWS RDS (external to cluster)
• Object Storage: S3 buckets via JuiceFS for persistent storage
• Load Balancer: AWS ALB for external traffic routing
• Monitoring: CloudWatch for observability

Technology Stack ¶

• Frontend: Nginx serving React/Vue.js application
• Backend: Django with uWSGI application server
• Task Queue: Celery with RabbitMQ broker
• Cache: Redis for session and application caching
• WebSocket: Centrifugo for real-time features
• File Upload: TUSD (TUS protocol) for resumable uploads
• Encoding: FFmpeg with NVIDIA GPU acceleration (h264_nvenc) when available
• Storage: JuiceFS CSI driver with S3 backend
• Push Notifications: Firebase Cloud Messaging (FCM)

Media Asset Management Features ¶

Based on the configuration, the platform supports:

• Video encoding and transcoding (multiple formats and bitrates)
• Resumable file uploads (TUS protocol)
• Media storage and retrieval with signed URLs
• Real-time notifications and collaboration
• Background task processing for encoding
• Dynamic worker scaling based on encoding workload

Support ¶

For issues or questions, contact:

• jeremy@mnemonica.com
• vincenzo@mnemonica.com
• info@mnemonica.com