Python Full Stack Web Developer Course

Week 3: Python Fundamentals (Part 2)

Friday Morning: Understanding Dependencies in Web Projects

The Web of Dependencies: Building on the Shoulders of Giants

Welcome to our exploration of dependencies in web projects! Today, we'll dive deep into one of the most critical aspects of modern web development—understanding, managing, and mastering the intricate web of dependencies that underpin virtually every professional web application.

As we transition from core Python concepts to web development, understanding dependencies becomes essential. Modern web applications aren't built in isolation; they're constructed by combining existing libraries, frameworks, and tools into cohesive systems that solve complex problems efficiently.

What Are Dependencies?

Analogy: Think of your web application as a gourmet meal. While you are the chef creating the final dish, you're not growing the vegetables, milling the flour, or crafting the cookware. Instead, you're relying on ingredients and tools created by others, combining them with your expertise to create something unique.

In web development terms:

Types of Dependencies in Web Projects

Real-world Example: Django, a popular web framework, directly depends on packages like asgiref, sqlparse, and others. When you install Django, pip automatically installs these transitive dependencies even though you never explicitly imported them in your code.

The Dependency Ecosystem in Python Web Development

Let's explore the typical categories of dependencies found in Python web projects:

Core Framework Dependencies

The foundation of most Python web applications:

Metaphor: These are like the structural framework of a building—load-bearing walls, foundation, and support beams that everything else attaches to.

Database Dependencies

Components that enable data storage and retrieval:

Code Example: In this Flask application, we depend on SQLAlchemy and a PostgreSQL driver:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://user:password@localhost/mydb'
db = SQLAlchemy(app)

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)

    def __repr__(self):
        return f'<User {self.username}>'

Authentication and Security Dependencies

Libraries that protect your application and users:

Analogy: These are the security systems of your application—the locks, alarms, authentication systems, and guards that protect your digital assets.

Frontend Integration Dependencies

Libraries that help bridge backend and frontend:

Real-world Usage: In a Flask project, Jinja2 templates connect Python data with HTML:

# Python route
@app.route('/users')
def list_users():
    users = User.query.all()
    return render_template('users.html', users=users)

# Jinja2 template (users.html)
<!DOCTYPE html>
<html>
<body>
    <h1>User List</h1>
    <ul>
    {% for user in users %}
        <li>{{ user.username }} ({{ user.email }})</li>
    {% endfor %}
    </ul>
</body>
</html>

Utility and Helper Dependencies

General-purpose tools that make development easier:

Metaphor: These are like the power tools of development—they don't define the structure but make specific tasks much more efficient.

Dependency Trees and Relationships

Dependencies rarely exist in isolation. They form complex trees or graphs of relationships.

Visualizing Dependency Trees

Consider this simplified dependency tree for a Flask web application:

Your Web Application
├── Flask 2.0.1
│   ├── Werkzeug 2.0.1
│   ├── Jinja2 3.0.1
│   │   └── MarkupSafe 2.0.1
│   ├── itsdangerous 2.0.1
│   └── click 8.0.1
├── Flask-SQLAlchemy 2.5.1
│   ├── Flask 2.0.1 (already installed)
│   └── SQLAlchemy 1.4.23
├── psycopg2-binary 2.9.1
├── Flask-Migrate 3.1.0
│   ├── Flask 2.0.1 (already installed)
│   ├── Flask-SQLAlchemy 2.5.1 (already installed)
│   └── Alembic 1.7.1
│       └── SQLAlchemy 1.4.23 (already installed)
└── Flask-Login 0.5.0
    └── Flask 2.0.1 (already installed)

Visualization Tools: You can generate actual dependency graphs with:

# Using pipdeptree
pip install pipdeptree
pipdeptree --graph-output png > dependencies.png

# Using pip-tools
pip install pip-tools
pip-compile --output-file=requirements.txt pyproject.toml

Understanding Version Constraints

Dependencies specify version ranges they're compatible with:

Constraint Meaning Example
== Exactly this version Flask==2.0.1
>= This version or newer SQLAlchemy>=1.4.0
<= This version or older Werkzeug<=2.0.3
~= Compatible release (same minor version) requests~=2.25.0 (allows 2.25.1 but not 2.26.0)
!= Not this version lxml!=4.6.0

Real-world Scenario: A common dependency conflict occurs between libraries requiring different versions of a shared dependency. For example, LibraryA might require requests>=2.20.0,<2.25.0 while LibraryB requires requests>=2.26.0. This creates an unsolvable constraint.

Dependency Hell: Common Problems and Solutions

"Dependency Hell" refers to the frustrating situations that arise when managing complex dependency relationships. Let's explore common problems and their solutions:

Version Conflicts

Problem: Two or more packages require incompatible versions of the same dependency.

Example:

# Package A's requirements
sqlalchemy>=1.4.0,<1.5.0

# Package B's requirements
sqlalchemy>=2.0.0

Solutions:

Dependency Bloat

Problem: Your project accumulates unnecessary dependencies, increasing size and complexity.

Analogy: This is like kitchen drawer syndrome—over time, you accumulate tools and gadgets you rarely use but that take up space and make it harder to find what you need.

Solutions:

Transitive Dependency Issues

Problem: Problems arising from dependencies of your dependencies, which you don't directly control.

Solutions:

Dependency Resolution Timeouts

Problem: In complex projects, pip may take a very long time to resolve dependencies or even time out.

Solutions:

Real-world Impact: Large companies like Instagram or Dropbox employ dedicated engineers just to manage Python dependencies and ensure smooth operation of their complex dependency trees.

Best Practices for Dependency Management in Web Projects

Let's explore proven strategies for managing dependencies effectively in web development:

Explicit is Better than Implicit

Always be explicit about your dependencies:

Good Example:

# requirements.txt

# Web framework
flask==2.0.1  # Pinned for stability in production

# Database
sqlalchemy>=1.4.23,<1.5.0  # Needs 1.4 features, but 1.5 has breaking changes
psycopg2-binary==2.9.1  # PostgreSQL driver

# Authentication
flask-login==0.5.0
pyjwt==2.1.0  # Required for JSON Web Token support

Versioning Strategy

Adopt a clear strategy for version constraints:

Analogy: Think of version pinning like cooking measurements. During experimentation (development), you might use "about a tablespoon" (ranges), but for a critical dinner party (production), you measure exactly "1.5 tablespoons" (pinned versions).

Separation of Concerns

Separate different types of dependencies:

# requirements.txt - Production dependencies
flask==2.0.1
sqlalchemy==1.4.23
requests==2.26.0

# requirements-dev.txt - Development dependencies
-r requirements.txt  # Include production dependencies
pytest==6.2.5
black==21.8b0
flake8==3.9.2

# requirements-deploy.txt - Deployment-specific dependencies
-r requirements.txt  # Include production dependencies
gunicorn==20.1.0
psycopg2-binary==2.9.1

Lock Files for Deterministic Builds

Ensure consistent environments with lock files:

# Generate a lock file with pip-tools
pip-compile --generate-hashes requirements.in -o requirements.txt

# Install with verification
pip install --require-hashes -r requirements.txt

Regular Maintenance

Treat dependencies as living parts of your codebase:

Tool Example:

# Check for outdated packages
pip list --outdated

# Check for security vulnerabilities
pip install safety
safety check -r requirements.txt

Testing Dependency Changes

Never update blindly:

Web-Specific Dependency Considerations

Web applications have unique dependency challenges:

Frontend and Backend Integration

Modern web projects often have both Python and JavaScript dependencies:

# Backend dependencies (Python)
flask==2.0.1
sqlalchemy==1.4.23

# Frontend dependencies (JavaScript - package.json)
{
  "dependencies": {
    "react": "^17.0.2",
    "axios": "^0.21.1",
    "bootstrap": "^5.1.0"
  }
}

Strategies for managing both:

Production vs. Development Environments

Web projects typically run in multiple environments:

Environment Dependency Approach Example
Development Include debugging tools, hot reloading flask==2.0.1, flask-debugtoolbar==0.11.0
Testing Include testing frameworks, fixtures pytest==6.2.5, pytest-flask==1.2.0
Staging Mirror production but with monitoring gunicorn==20.1.0, sentry-sdk==1.3.1
Production Minimal, performance-optimized gunicorn==20.1.0, uvloop==0.16.0

Containerization and Dependencies

Docker changes how dependencies are managed:

# Example Dockerfile for a Python web application
FROM python:3.9-slim

WORKDIR /app

# Copy dependency files first (for build caching)
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Run application
CMD ["gunicorn", "app:app"]

Docker best practices for dependencies:

Security Implications of Dependencies

Dependencies can introduce security vulnerabilities to your application:

Supply Chain Attacks

Scenario: Malicious code is inserted into a legitimate package you depend on.

Notable Examples:

Prevention Strategies:

Vulnerability Management

Dependencies will have security issues over time:

# Check for known vulnerabilities
pip install safety
safety check -r requirements.txt

# Sample output
+==============================================================================+
|                                                                              |
|                               /$$$$$$            /$$                         |
|                              /$$__  $$          | $$                         |
|           /$$$$$$$  /$$$$$$ | $$  \__//$$$$$$  | $$$$$$$  /$$   /$$         |
|          /$$_____/ |____  $$| $$$$   /$$__  $$ | $$__  $$| $$  | $$         |
|         |  $$$$$$   /$$$$$$$| $$_/  | $$$$$$$$| $$  \ $$| $$  | $$         |
|          \____  $$ /$$__  $$| $$    | $$_____/| $$  | $$| $$  | $$         |
|          /$$$$$$$/|  $$$$$$$| $$    |  $$$$$$$| $$  | $$|  $$$$$$$         |
|         |_______/  \_______/|__/     \_______/|__/  |__/ \____  $$         |
|                                                           /$$  | $$         |
|                                                          |  $$$$$$/         |
|  by pyup.io                                               \______/          |
|                                                                              |
+==============================================================================+
| REPORT                                                                       |
+==============================================================================+
| Found 1 vulnerability in 1 package                                           |
+==============================================================================+
| package       | installed | affected        | ID       | fixed in    |
+==============================================================================+
| requests      | 2.26.0    | <=2.26.0       | 38195    | 2.27.0      |
+==============================================================================+

Establish a Response Process:

  1. Regular vulnerability scanning (automated in CI)
  2. Severity assessment for each vulnerability
  3. Update plan based on severity
  4. Testing updates before deployment
  5. Emergency process for critical vulnerabilities

Principle of Least Dependency

Minimize your attack surface:

Example Decision Framework:

# Before adding a new dependency, ask:
1. Is this functionality essential?
2. Could we implement it ourselves with reasonable effort?
3. If we need a dependency:
   a. How well-maintained is it?
   b. How many downloads/users does it have?
   c. Does it have a history of security issues?
   d. How responsive are maintainers to security reports?
   e. How many transitive dependencies does it bring?

Dependency Strategies in Modern Web Frameworks

Let's examine how popular Python web frameworks approach dependencies:

Flask Dependency Philosophy

Flask follows a "microframework" approach:

Example Flask Project Structure:

# Minimal Flask dependencies
flask==2.0.1

# Optional extensions as needed
flask-sqlalchemy==2.5.1  # Only if you need ORM
flask-login==0.5.0       # Only if you need authentication
flask-migrate==3.1.0     # Only if you need migrations
flask-wtf==0.15.1        # Only if you need forms

Advantage: Fine-grained control, minimal bloat

Challenge: More decision-making, potential for compatibility issues between extensions

Django Dependency Approach

Django takes a "batteries-included" approach:

Example Django Dependency Set:

# Django itself includes many components
django==3.2.7

# Fewer external dependencies needed for basic functionality
# Still need specific adapters for databases
psycopg2-binary==2.9.1  # For PostgreSQL

# Additional packages typically for specialized needs
django-debug-toolbar==3.2.2
djangorestframework==3.12.4

Advantage: Integrated components, less decision fatigue

Challenge: Less flexibility, potential for unused components

FastAPI Dependency Approach

FastAPI takes a modern, modular approach:

Example FastAPI Dependencies:

# Core dependencies
fastapi==0.68.1
pydantic==1.8.2
starlette==0.14.2

# Server dependencies
uvicorn==0.15.0

# Optional features
python-multipart==0.0.5  # For form data
aiofiles==0.7.0          # For file handling
sqlalchemy==1.4.23       # For database support

Advantage: Modern design, clear optional components

Challenge: Still maturing ecosystem

Practical Example: Analyzing a Real Web Project

Let's analyze the dependency structure of a typical Flask web application to illustrate these concepts:

Sample Web Application Architecture

Flask Web App
├── Web Framework Layer
│   ├── Flask (Core framework)
│   ├── Flask-RESTful (API framework)
│   └── Flask-Cors (Cross-origin resource sharing)
├── Authentication Layer
│   ├── Flask-Login (Session management)
│   ├── Flask-JWT-Extended (JWT tokens)
│   └── passlib (Password hashing)
├── Database Layer
│   ├── Flask-SQLAlchemy (ORM)
│   ├── Flask-Migrate (Migrations)
│   └── psycopg2-binary (PostgreSQL driver)
├── Form Processing Layer
│   ├── Flask-WTF (Form handling)
│   └── email-validator (Email validation)
└── Utility Layer
    ├── Pillow (Image processing)
    ├── requests (HTTP client)
    ├── python-dotenv (Environment management)
    └── celery (Background tasks)

Dependency Analysis

Let's examine what this requirements.txt might look like:

# Web Framework Layer
flask==2.0.1
flask-restful==0.3.9
flask-cors==3.0.10

# Authentication Layer
flask-login==0.5.0
flask-jwt-extended==4.3.1
passlib==1.7.4

# Database Layer
flask-sqlalchemy==2.5.1
flask-migrate==3.1.0
psycopg2-binary==2.9.1

# Form Processing Layer
flask-wtf==0.15.1
email-validator==1.1.3

# Utility Layer
pillow==8.3.2
requests==2.26.0
python-dotenv==0.19.0
celery==5.1.2
redis==3.5.3  # For Celery broker

Analysis:

Dependency Insights

Potential Issues to Watch

Practical Exercise: Dependency Audit

Let's work through a practical exercise to analyze and improve a project's dependencies:

Scenario: Take Over a Legacy Web Application

You've inherited a Python web application with the following requirements.txt:

# Legacy requirements.txt
Flask>=1.0.0
Werkzeug>=0.14
Jinja2
SQLAlchemy
pymysql
requests
python-dateutil
pyjwt
pillow
redis
celery==4.0.0
beautifulsoup4
lxml
six
pytest

Problems with This Approach

Step 1: Identify Current Versions

First, create a virtual environment and install these packages:

python -m venv audit-env
source audit-env/bin/activate  # On Windows: audit-env\Scripts\activate
pip install -r requirements.txt
pip freeze > current-versions.txt

Step 2: Check for Vulnerabilities

pip install safety
safety check -r current-versions.txt

Step 3: Identify Direct vs. Transitive Dependencies

pip install pipdeptree
pipdeptree

Step 4: Reorganize Dependencies

Create separate files for different dependency types:

# requirements-prod.txt - Production dependencies
Flask==2.0.1
SQLAlchemy==1.4.23
pymysql==1.0.2
requests==2.26.0
python-dateutil==2.8.2
pyjwt==2.1.0
pillow==8.3.2
redis==3.5.3
celery==5.1.2
beautifulsoup4==4.10.0
lxml==4.6.3

# requirements-dev.txt - Development dependencies
-r requirements-prod.txt  # Include production dependencies
pytest==6.2.5
black==21.8b0
flake8==3.9.2
pytest-cov==2.12.1

Step 5: Document Dependency Purposes

Add comments to explain why each dependency exists:

# requirements-prod.txt - Production dependencies

# Web framework
Flask==2.0.1

# Database
SQLAlchemy==1.4.23  # ORM
pymysql==1.0.2      # MySQL driver

# External API communication
requests==2.26.0

# Utilities
python-dateutil==2.8.2  # Date manipulation
pyjwt==2.1.0            # Authentication tokens
pillow==8.3.2           # Image processing

# Background tasks
redis==3.5.3            # Message broker
celery==5.1.2           # Task queue

# Data processing
beautifulsoup4==4.10.0  # HTML parsing
lxml==4.6.3             # XML/HTML processing

Step 6: Implement Dependency Locking

Use pip-tools to create comprehensive lock files:

pip install pip-tools

# Convert to input files
cp requirements-prod.txt requirements-prod.in
cp requirements-dev.txt requirements-dev.in

# Generate locked files with hashes
pip-compile --generate-hashes requirements-prod.in -o requirements-prod.txt
pip-compile --generate-hashes requirements-dev.in -o requirements-dev.txt

Step 7: Create Dependency Update Process

Establish a regular update routine:

# Update script (update-deps.sh)
#!/bin/bash
echo "Updating dependencies..."

# Create virtual environment
python -m venv update-env
source update-env/bin/activate

# Install tools
pip install pip-tools safety

# Update production dependencies
pip-compile --upgrade requirements-prod.in -o requirements-prod.txt
safety check -r requirements-prod.txt

# Update development dependencies
pip-compile --upgrade requirements-dev.in -o requirements-dev.txt
safety check -r requirements-dev.txt

# Cleanup
deactivate
rm -rf update-env

echo "Dependencies updated and checked for vulnerabilities."

This exercise demonstrates how to take an unmaintained dependency set and transform it into a well-organized, secure, and maintainable system.

Future Trends in Dependency Management

The Python packaging ecosystem continues to evolve. Here are some important trends:

PEP 621 and pyproject.toml

Moving toward a standardized project configuration:

# Modern dependency specification in pyproject.toml
[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "web-app"
version = "1.0.0"
description = "A web application"
requires-python = ">=3.8"
dependencies = [
    "flask>=2.0.0",
    "sqlalchemy>=1.4.0",
    "requests>=2.25.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=6.0.0",
    "black>=21.5b0",
]

Dependency Auditing and Supply Chain Security

Increasing focus on security:

Container-Native Dependency Management

Integration with containerization:

Language-Agnostic Dependency Management

Tools that manage dependencies across languages:

Conclusion

Understanding dependencies is a critical skill for web developers. As we've explored:

Final Metaphor: Managing dependencies is like conducting an orchestra. Each dependency is an instrument that contributes to the whole, but they must be carefully coordinated to create harmony rather than cacophony. As the conductor, you don't need to play each instrument yourself, but you need to understand how they work together and ensure they're playing the right notes.

In the next sessions, we'll build on this foundation as we explore specific web frameworks and see these dependency concepts in action.

Additional Resources