Docker Compose Introduction

What is Docker Compose?

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services, networks, and volumes. Then, with a single command, you create and start all the services from your configuration.

Think of Docker Compose as a conductor of an orchestra. While Docker manages individual containers (musicians), Docker Compose coordinates multiple containers (the entire orchestra) to work together harmoniously.

When to Use Docker Compose

Docker Compose is ideal for:

• Development environments: Set up a complete development environment with databases, message queues, etc.
• Automated testing: Create and teardown isolated testing environments
• Small deployments: Single-server deployments in production (for larger deployments, Kubernetes is preferred)
• CI/CD environments: Consistent testing and build environments

Real-world Example: Consider a web application with a Python Flask backend, a PostgreSQL database, and a Redis cache. Instead of manually starting and configuring each service, Docker Compose allows you to define all components, their relationships, and configuration in a single file, then bring everything up with one command.

Multi-container Applications

Modern web applications typically consist of multiple services:

• Web servers
• Application servers
• Databases
• Caching layers
• Message queues
• Worker processes

Before Docker Compose, developers might need to:

1. Remember specific Docker run commands with all parameters
2. Create shell scripts to start containers in the right order
3. Manually set up networks between containers
4. Track and manage volumes for persistence

Docker Compose solves these problems with a declarative approach. It's like the difference between giving step-by-step directions (imperative) versus providing a map with the destination marked (declarative).

Creating a docker-compose.yml File

The docker-compose.yml file is the heart of Docker Compose. This YAML file defines your application's services, networks, and volumes.

A simple example for a Python web application with a database:

version: '3'  # Docker Compose version

services:
  # Web application service
  web:
    build: ./web  # Path to Dockerfile
    ports:
      - "8000:8000"  # Host:Container port mapping
    volumes:
      - ./web:/app  # Mount code for development
    environment:
      - DATABASE_URL=postgresql://postgres:password@db:5432/mydatabase
    depends_on:
      - db  # Ensures database starts before web service
      
  # Database service
  db:
    image: postgres:13  # Using official PostgreSQL image
    volumes:
      - postgres_data:/var/lib/postgresql/data  # Named volume for persistence
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=mydatabase
      
volumes:
  postgres_data:  # Defines the named volume

Let's break down the key components:

Services

Services define the containers that make up your application. Each service can have:

• build: Path to a Dockerfile or build context
• image: Docker image to pull
• ports: Port mappings
• volumes: Volume mappings for persistent data or development
• environment: Environment variables
• depends_on: Service dependencies
• networks: Network connections
• command: Override the default command
• restart: Restart policy

Think of services as defining the "rooms" in your application house, each with its own purpose and connections to other rooms.

Volumes

Volumes provide persistent storage for your containers. They can be:

• Named volumes: Managed by Docker, persistent across container restarts
• Host paths: Directories on your host machine
• Anonymous volumes: Temporary volumes without a name

Volumes are like the storage spaces of your house - closets, cabinets, and storage rooms that keep things even when you renovate.

Networks

Networks allow containers to communicate with each other. By default, Docker Compose creates a network for your application, but you can define custom networks.

Networks are like the wiring and plumbing systems in a house, connecting different rooms and allowing them to share resources.

Environment Variables and Configuration

Proper configuration is critical for containerized applications. Docker Compose offers several ways to manage environment variables:

Inline Environment Variables

services:
  web:
    environment:
      - DEBUG=True
      - SECRET_KEY=dev_key

Using .env Files

For better security and organization, you can extract environment variables to a separate .env file:

# .env file
DEBUG=True
SECRET_KEY=dev_key

Then reference it in your compose file:

services:
  web:
    env_file:
      - .env

Variable Substitution

Docker Compose supports variable substitution using the ${VARIABLE} syntax:

services:
  web:
    image: myapp:${APP_VERSION:-latest}  # Uses APP_VERSION if defined, otherwise "latest"

Real-world application: This is particularly useful when you need different configurations for development, testing, and production environments. For example, you might have:

• .env.development with development settings
• .env.test with testing settings
• .env.production with production settings

And then run with: docker-compose --env-file .env.development up

Volume Mounts for Development

One of the most powerful features of Docker Compose for development is the ability to mount code directories from your host machine into containers. This enables:

• Live code editing: Edit code on your host machine, and see changes immediately
• No rebuild needed: Save time by avoiding container rebuilds for code changes
• Faster feedback loop: Rapid development iterations

services:
  web:
    build: ./web
    volumes:
      - ./web:/app  # Maps ./web on host to /app in container

Real-world example: When developing a Flask application, you can mount your source code directory into the container and enable Flask's debug mode. Then, whenever you save a Python file, Flask will automatically reload, instantly reflecting your changes without restarting the container.

version: '3'

services:
  web:
    build: ./web
    ports:
      - "5000:5000"
    volumes:
      - ./web:/app
    environment:
      - FLASK_ENV=development
      - FLASK_DEBUG=1
    command: flask run --host=0.0.0.0
    depends_on:
      - db
      
  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data
      # Mount database initialization scripts
      - ./database/init:/docker-entrypoint-initdb.d
    environment:
      - POSTGRES_PASSWORD=devpassword
      - POSTGRES_DB=myapp_dev
      
volumes:
  postgres_data:

Think of volume mounts like having a workshop with all your tools and materials (your code editor, linter, git) outside the container, but the assembly line (runtime environment) inside the container. You can use familiar tools while building in a consistent environment.

Setting up a Development Environment with Docker Compose

Let's walk through creating a development environment for a Python web application with Docker Compose:

Step 1: Project Structure

Create a directory structure like this:

my_project/
├── docker-compose.yml
├── .env
├── web/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py
│   └── ...
└── database/
    └── init/
        └── init.sql

Step 2: Dockerfile for Python Application

In web/Dockerfile:

FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Don't copy code - we'll use a volume mount instead
# COPY . .

EXPOSE 5000

CMD ["python", "app.py"]

Step 3: Sample Flask Application

In web/app.py:

from flask import Flask
import os
import psycopg2

app = Flask(__name__)

@app.route('/')
def hello():
    # Example of connecting to database
    conn = psycopg2.connect(
        host=os.environ.get('DB_HOST', 'db'),
        database=os.environ.get('DB_NAME', 'myapp_dev'),
        user=os.environ.get('DB_USER', 'postgres'),
        password=os.environ.get('DB_PASSWORD', 'devpassword')
    )
    cur = conn.cursor()
    cur.execute('SELECT 1')
    result = cur.fetchone()
    cur.close()
    conn.close()
    
    return f"Hello from Flask! Database connection test: {result[0]}"

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True)

Step 4: Database Initialization

In database/init/init.sql:

-- This script will be executed when the database container is created
CREATE TABLE IF NOT EXISTS example (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Add some sample data
INSERT INTO example (name) VALUES ('Sample Item 1'), ('Sample Item 2');

Step 5: docker-compose.yml

version: '3'

services:
  web:
    build: ./web
    ports:
      - "5000:5000"
    volumes:
      - ./web:/app
    environment:
      - DB_HOST=db
      - DB_NAME=myapp_dev
      - DB_USER=postgres
      - DB_PASSWORD=devpassword
    depends_on:
      - db
      
  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./database/init:/docker-entrypoint-initdb.d
    environment:
      - POSTGRES_PASSWORD=devpassword
      - POSTGRES_DB=myapp_dev
    ports:
      - "5432:5432"  # Expose for local DB tools
      
volumes:
  postgres_data:

Step 6: Environment Variables

In .env:

# Development environment settings
FLASK_ENV=development
COMPOSE_PROJECT_NAME=myapp_dev

# Database credentials - also defined in docker-compose.yml
DB_HOST=db
DB_NAME=myapp_dev
DB_USER=postgres
DB_PASSWORD=devpassword

Step 7: Running the Development Environment

# Start all services
docker-compose up

# Or in detached mode (background)
docker-compose up -d

# View logs
docker-compose logs -f

# Stop all services
docker-compose down

# Rebuild and start (after Dockerfile changes)
docker-compose up --build

With this setup, you can:

• Edit Python code in the web/ directory and see changes immediately
• Connect to the database using a local database tool (e.g., pgAdmin) via localhost:5432
• Easily add more services (Redis, Celery workers, etc.) as your application grows

Hot Reloading with Mounted Volumes

Hot reloading creates a seamless development experience. Here's how it works with Docker Compose:

1. Code is mounted from host to container via volumes
2. The application inside the container detects file changes
3. Application automatically reloads with new code

For different frameworks, you'll need different approaches:

Flask

environment:
  - FLASK_DEBUG=1  # Enables debug mode with auto-reload
command: flask run --host=0.0.0.0

Django

command: python manage.py runserver 0.0.0.0:8000

Node.js (with nodemon)

command: nodemon app.js

Docker Networking Basics

Docker Compose automatically creates a network for your application's services to communicate. Understanding networking is crucial for multi-container applications.

Key Networking Concepts

• Service discovery: Containers can reach each other using service names as hostnames
• Isolated networks: Each Compose project gets its own network, isolated from other projects
• Port mapping: Expose container ports to host with ports directive
• Custom networks: Create separate networks for different communication needs

Example: In our example above, the Flask application can connect to the database using db as the hostname, even though the actual IP address might change with each startup.

Custom Networks Example

version: '3'

networks:
  frontend:
    # Network for frontend components
  backend:
    # Network for backend services
  database:
    # Network specifically for database access

services:
  web:
    build: ./web
    networks:
      - frontend
      - backend
    
  api:
    build: ./api
    networks:
      - backend
      - database
    
  db:
    image: postgres:13
    networks:
      - database

This setup creates three networks and connects services only to the networks they need. The web service cannot directly access the db service, it must go through the api service.

Troubleshooting Common Docker Issues

When working with Docker Compose, you'll encounter common issues. Here's how to troubleshoot them:

1. Container Not Starting

Symptoms: Container exits immediately after starting

Debugging Steps:

• Check container logs: docker-compose logs [service_name]
• Try running with the interactive flag: docker-compose run --rm [service_name] sh
• Verify your CMD or command directive is correct

2. Services Can't Connect to Each Other

Symptoms: "Connection refused" or "Host not found" errors

Debugging Steps:

• Ensure service names are used as hostnames, not "localhost"
• Check if services are on the same network
• Verify the service you're trying to connect to has actually started
• Test connection from inside a container: docker-compose exec [service_name] ping [other_service]

3. Volume Mount Issues

Symptoms: Files not updating or permission errors

Debugging Steps:

• Verify path syntax (relative paths are relative to the docker-compose.yml file)
• Check file permissions on host and in container
• For Windows users, ensure file sharing is enabled

4. Port Conflicts

Symptoms: "Port is already allocated" errors

Debugging Steps:

• Check if another process is using the port: lsof -i :PORT (Unix) or netstat -ano | findstr PORT (Windows)
• Choose a different host port in your port mapping
• Stop other Docker containers that might be using the port

5. Changes Not Taking Effect

Symptoms: Configuration changes aren't reflected after restart

Debugging Steps:

• Force recreation of containers: docker-compose up -d --force-recreate
• Rebuild images: docker-compose build or docker-compose up --build
• Check if you're modifying the correct docker-compose.yml file

Afternoon Assignment: Create a Docker Compose Setup

Your assignment is to create a Docker Compose setup with a Python service and a simple database. The requirements are:

1. A Python Flask service with a simple endpoint to read/write data
2. A PostgreSQL database to store the data
3. Volume mounts for live code updates
4. Environment variables for configuration
5. A README explaining how to use your setup

File Structure:

assignment/
├── docker-compose.yml
├── .env
├── README.md
├── app/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app.py
└── database/
    └── init/
        └── init.sql

Hints:

• Use the examples from today's lesson as a starting point
• Ensure your Flask app properly handles database connections
• Create a simple API with endpoints for CRUD operations
• Test your setup thoroughly
• Include commands for starting, stopping, and rebuilding in your README

Extra Challenge: Add a third service, such as Redis for caching, and demonstrate communication between all three services.

Further Resources

• Docker Compose Documentation
• Docker Compose File Reference
• Docker Compose Samples
• Awesome Docker Compose
• Dockerizing Flask with Postgres, Gunicorn, and Nginx