Developer Guide

Required Tools

• Rust toolchain (version 1.86.0 or later)

  • Install Rust

• Git for version control

  • Install Git

• Kind (Kubernetes in Docker)

  • Install Kind

• Helm (Kubernetes package manager)

  • Install Helm

• Container Engine: Docker or Podman

  • Install Docker or Install Podman

• cargo-make - Rust task runner and build tool

  • Installation Guide

  • Documentation

• kubectl - Kubernetes command-line tool

  • Installation Guide

  • Documentation

Tools for debugging

• mirrord - Local development with Kubernetes environment

  • Installation Guide

  • Documentation

  • Required for: cargo make debug

Step 1: Clone the Repository

Step 2: Install kubectl (if needed)

Expected output:

• ✅ kubectl installation for your platform (Linux/macOS)

• ✅ Verification that kubectl is working

Step 3: Setup Kind Cluster with Traefik

What this does:

• 🏗️ Creates a Kind Kubernetes cluster

• 🚀 Installs Traefik ingress controller in traefik-system namespace

• 🔐 Generates wildcard TLS certificates for *.127.0.0.1.nip.io

• 🖥️ Sets up Traefik dashboard

• ⏳ Waits for all components to be ready

Expected output:

Step 4: Deploy Demo Applications

What this does:

• 🤖 Deploys whoami application in whoami namespace

• 🌐 Deploys httpbin application in httpbin namespace

• 🔐 Copies TLS certificates to application namespaces

• ⏳ Waits for deployments to be ready

• 🧪 Tests application endpoints

Expected output:

Step 5: Install CloudNativePG

What this does:

• 📦 Adds CloudNativePG Helm repository

• 🗄️ Installs CNPG operator in cnpg-system namespace

• ⏳ Waits for operator to be ready (120s timeout)

• 📊 Shows CNPG status and version

Expected output:

Step 6: Deploy RC-App

What this does:

• ✅ Validates CNPG is installed and ready

• 🚀 Deploys rc-app using Helm chart from k8s/rc-app

• ⏳ Waits for deployment to be available (120s timeout)

• 🧪 Performs comprehensive health checks with retries

• 📊 Shows deployment status and resource information

Expected output:

Test All Applications

Check Cluster Status

Monitor Deployments

Direct Database Connection (connect-postgres.sh)

For direct database access with an interactive psql session:

What this script does:

• ✅ Checks CNPG operator status

• 🔍 Finds PostgreSQL pod using label cnpg.io/podRole=instance

• 🔐 Extracts credentials from Kubernetes secret dev-rc-app-database-app

• 🚀 Establishes port forwarding to the PostgreSQL pod

• 💻 Launches interactive psql session

• 🧹 Automatically cleans up port forwarding on exit

Example output:

Persistent Port Forwarding (portforward-postgres.sh)

For maintaining a persistent database connection without launching psql:

What this script does:

• 🔄 Maintains persistent port forwarding connection

• 📊 Monitors connection health every 10 seconds

• 🔧 Automatically recovers from connection failures

• 📋 Displays connection URLs for external tools

• ⏳ Runs until manually stopped with Ctrl+C

Example output:

Pod Terminal Access (shell-postgres.sh)

For direct access to the PostgreSQL pod terminal with an interactive shell:

What this script does:

• ✅ Checks CNPG operator status

• 🔍 Finds PostgreSQL pod using label cnpg.io/podRole=instance

• 🖥️ Connects directly to the pod terminal with interactive bash

• 🔧 Provides access to all PostgreSQL tools within the pod

• 🧹 No port forwarding or secrets required

Example output:

Database Reset Script (database_reset.sql)

For resetting the database to a clean state during development and testing:

What this script does:

• 🗑️ Drops all application tables: definition, definitions, event, event_listener, event_sequence

• 🔧 Removes custom functions: event_store_begin_epoch(), event_store_current_epoch(), notify_event_listener()

• 📊 Drops sequences: event_sequence_event_id_seq

• 🧹 Completely cleans the database schema for fresh starts

Script contents:

When to use:

• 🧪 Before running integration tests

• 🔄 When switching between different development branches

• 🐛 When debugging database-related issues

• 🆕 When you need a completely fresh database state

Alternative execution methods:

Use Cases

Use connect-postgres.sh when:

• You need direct SQL access for debugging

• Running database migrations or admin tasks

• Exploring database schema and data

• One-time database operations

Use portforward-postgres.sh when:

• Connecting external database tools (pgAdmin, DBeaver, etc.)

• Running applications that need database access

• Long-running database connections

• Development with persistent database connectivity

Use shell-postgres.sh when:

• You need direct access to the PostgreSQL server environment

• Running database administration tasks (pg_dump, pg_restore)

• Debugging PostgreSQL server configuration

• Inspecting pod filesystem and logs

• Performing manual database operations within the pod

Use database_reset.sql when:

• Starting integration tests that require a clean database

• Switching between development branches with different schemas

• Debugging database-related issues by resetting to a known state

• Clearing test data after development sessions

• Preparing for fresh application deployments

Connection URLs

Both scripts provide connection URLs in multiple formats:

• Regular PostgreSQL URL: postgresql://username:password@localhost:5432/database

• JDBC URL: jdbc:postgresql://localhost:5432/database?user=username&password=password

• Individual connection details for manual configuration

These URLs can be used with:

• Database management tools (pgAdmin, DBeaver, TablePlus)

• Application configuration files

• Development environments

• CI/CD pipelines for database operations

Namespaces

• traefik-system - Traefik ingress controller and dashboard

• cnpg-system - CloudNativePG operator for PostgreSQL

• whoami - Demo application showing request details

• httpbin - HTTP testing and debugging service

• default - Main rc-app application

Applications

Health Endpoints

• RC-App Health: https://rc.127.0.0.1.nip.io/healthz

• RC-App Readiness: https://rc.127.0.0.1.nip.io/readyz

Common Issues

Recovery Commands

Building and Testing

Debugging with mirrord

For advanced debugging and development, you can use mirrord to run your local application while connecting to the Kubernetes cluster environment:

What this does:

• 🔍 Automatically discovers the rc-app pod in the cluster

• ✅ Validates the deployment has exactly one replica

• 🔗 Uses mirrord to mirror traffic from the Kubernetes pod to your local application

• 🐛 Runs the application locally with debug logging (RUST_LOG=rc_web=debug)

Prerequisites for debugging:

• mirrord must be installed: Installation Guide

• RC-app must be deployed: cargo make deploy-rc-app

• Deployment should have exactly 1 replica (default configuration)

Example debug session:

Benefits of mirrord debugging:

• Environment parity: Your local app runs with the same environment variables, secrets, and network access as the cluster

• Real traffic: Test with actual Kubernetes traffic patterns

• Database access: Connect to the same PostgreSQL database as the cluster

• Service discovery: Access other services in the cluster seamlessly

Troubleshooting debug issues:

Managing the Demo Environment