Introduction to Python Project Structure
Welcome to our session on Python project structure best practices! As you progress from writing simple scripts to developing complex applications, the organization of your code becomes increasingly important. A well-structured project makes your code more maintainable, scalable, and collaborative.
Think of project structure as the architecture of a building. Just as architects carefully plan the layout of rooms, hallways, and utilities before construction begins, software developers should thoughtfully organize their code, assets, and configuration files. A well-designed building is not only functional and efficient but also easier to navigate and modify. Similarly, a well-structured Python project is easier to understand, debug, and extend.
In this session, we'll explore best practices for organizing Python projects of various sizes and complexities. We'll cover everything from simple script-based projects to complex full-stack web applications, with a focus on creating maintainable, scalable code that follows industry standards.
Why Project Structure Matters
Before diving into specific structures, let's understand why good project organization is crucial:
1. Maintainability
Well-organized code is easier to maintain. When files and modules have clear purposes and are logically grouped, developers can quickly locate and modify specific functionality without wading through unrelated code.
Real-World Scenario: Imagine a bug report comes in for your user authentication system. In a well-structured project, you can go directly to auth/login.py rather than searching through a monolithic 2,000-line app.py file.
2. Scalability
Good structure allows your project to grow gracefully. As you add features, the organization adapts without becoming unwieldy.
Real-World Scenario: When adding a new payment processor to your e-commerce application, you can simply add a new module in your existing payments package without reorganizing the entire codebase.
3. Collaboration
When multiple developers work on the same project, clear structure reduces merge conflicts and ensures everyone can find what they need.
Real-World Scenario: One developer can work on the authentication system while another focuses on the database models, with minimal interference since the code is properly separated.
4. Onboarding
New team members can become productive faster when they join a well-structured project. The organization serves as a map that guides them through the codebase.
Real-World Scenario: A new developer joins your team and can understand the project architecture within hours rather than weeks, allowing them to contribute meaningful code sooner.
5. Testing
Proper structure facilitates testing by making it easier to isolate components and write focused tests.
Real-World Scenario: Your data processing modules are separate from your web views, allowing you to write comprehensive unit tests for data handling logic without needing to simulate HTTP requests.
6. Deployment
Well-structured projects are easier to package, containerize, and deploy across different environments.
Real-World Scenario: Configuration is clearly separated from application code, making it simple to deploy the same codebase with different settings in development, staging, and production environments.
Real-World Analogy: Think of a poorly structured project like a messy workshop where tools are scattered everywhere. You might eventually find what you need, but it takes longer, and you're more likely to make mistakes. A well-structured project is like a workshop where every tool has its place—you can work more efficiently and with greater confidence.
Understanding Python Packaging
Before diving into specific project structures, let's review some fundamental concepts of Python packaging:
Modules vs. Packages
- Module: A single Python file containing code. For example,
utils.pyis a module. - Package: A directory containing modules and an
__init__.pyfile. For example, a directory namedutilswith an__init__.pyfile is a package.
The __init__.py File
This file marks a directory as a Python package. It can be empty or contain initialization code that runs when the package is imported.
# A simple __init__.py file that exposes specific functions
from .module1 import function1, function2
from .module2 import Class1
__all__ = ['function1', 'function2', 'Class1']
Absolute vs. Relative Imports
Python supports two types of import statements:
# Absolute import (preferred in most cases)
from mypackage.subpackage import module
# Relative import (useful within packages)
from .submodule import function # Import from sibling module
from ..parentpackage import module # Import from parent package
Namespace Packages (Python 3.3+)
Packages without __init__.py files, allowing parts of a package to be distributed across different directories or even different distributions.
Python Path and Import System
Understanding how Python finds modules when you import them is crucial:
- Python searches for modules in directories listed in
sys.path - The current directory is usually the first search location
- Other locations include installed packages and standard library paths
import sys
print(sys.path) # See where Python looks for imports
Simple Script Project Structure
For small, single-purpose scripts or utilities, a minimal structure is often sufficient:
simple_script/
├── script.py # Main script
├── utils.py # Helper functions
├── config.py # Configuration variables
├── requirements.txt # Dependencies
└── README.md # Documentation
This structure works well when:
- Your project has a single, focused purpose
- The total code is less than 1,000 lines
- You're the only developer or have a very small team
- The script doesn't need to be imported by other projects
Example: Data Processing Script
script.py - Main program flow:
#!/usr/bin/env python3
"""
Data processing script that reads CSV files,
performs transformations, and outputs results.
"""
import argparse
from utils import process_file, validate_data
from config import INPUT_DIRECTORY, OUTPUT_DIRECTORY
def main():
parser = argparse.ArgumentParser(description='Process CSV data files.')
parser.add_argument('--input', default=INPUT_DIRECTORY, help='Input directory')
parser.add_argument('--output', default=OUTPUT_DIRECTORY, help='Output directory')
args = parser.parse_args()
# Process files
files = get_input_files(args.input)
for file in files:
data = process_file(file)
if validate_data(data):
save_results(data, args.output)
else:
print(f"Validation failed for {file}")
def get_input_files(directory):
# Code to list files in directory
pass
def save_results(data, directory):
# Code to save processed data
pass
if __name__ == "__main__":
main()
utils.py - Helper functions:
"""Utility functions for data processing."""
import csv
import pandas as pd
def process_file(filename):
"""Process a single data file."""
# Read and process the file
df = pd.read_csv(filename)
# Perform transformations
return df
def validate_data(data):
"""Ensure data meets our requirements."""
# Validation logic
return True
config.py - Configuration:
"""Configuration settings for the data processor."""
import os
# Directories
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
INPUT_DIRECTORY = os.path.join(BASE_DIR, 'data/input')
OUTPUT_DIRECTORY = os.path.join(BASE_DIR, 'data/output')
# Processing options
MAX_ROWS = 10000
CHUNK_SIZE = 1000
Even in a simple project, this separation offers several benefits:
- The main script (
script.py) focuses on program flow - Utility functions are reusable and testable in isolation
- Configuration is centralized and easy to modify
Medium-Sized Application Structure
As projects grow, they benefit from more structured organization. Here's a typical layout for a medium-sized Python application:
medium_project/
├── mypackage/ # Main package
│ ├── __init__.py # Package initialization
│ ├── core.py # Core functionality
│ ├── helpers.py # Helper functions
│ └── subpackage/ # Sub-package for related functionality
│ ├── __init__.py
│ └── module.py
├── tests/ # Test directory
│ ├── __init__.py
│ ├── test_core.py
│ └── test_subpackage.py
├── docs/ # Documentation
│ └── index.md
├── scripts/ # Command-line scripts
│ └── run_analysis.py
├── setup.py # Installation script
├── requirements.txt # Dependencies
├── README.md # Project overview
└── LICENSE # License information
Key features of this structure:
- Code is organized into packages and sub-packages
- Tests are separated from application code
- Scripts directory contains executable entry points
- setup.py allows the package to be installed
Example: A Library for Data Visualization
mypackage/__init__.py - Package initialization:
"""Data visualization library for scientific data."""
from .core import plot_data, save_figure
from .helpers import load_data, preprocess_data
__version__ = '0.1.0'
__all__ = ['plot_data', 'save_figure', 'load_data', 'preprocess_data']
mypackage/core.py - Core functionality:
"""Core visualization functions."""
import matplotlib.pyplot as plt
import seaborn as sns
from .helpers import validate_data
def plot_data(data, plot_type='scatter', **kwargs):
"""
Create a visualization from the provided data.
Args:
data: Pandas DataFrame with the data to plot
plot_type: Type of plot to create (scatter, line, bar)
**kwargs: Additional arguments for the plot
Returns:
matplotlib.Figure: The created figure
"""
validate_data(data)
if plot_type == 'scatter':
fig, ax = plt.subplots()
ax.scatter(data['x'], data['y'], **kwargs)
elif plot_type == 'line':
fig, ax = plt.subplots()
ax.plot(data['x'], data['y'], **kwargs)
elif plot_type == 'bar':
fig, ax = plt.subplots()
ax.bar(data['x'], data['y'], **kwargs)
else:
raise ValueError(f"Unsupported plot type: {plot_type}")
return fig
def save_figure(fig, filename, dpi=300):
"""Save a figure to a file."""
fig.savefig(filename, dpi=dpi)
mypackage/helpers.py - Helper functions:
"""Helper functions for data handling."""
import pandas as pd
import numpy as np
def load_data(filename):
"""Load data from a file into a DataFrame."""
if filename.endswith('.csv'):
return pd.read_csv(filename)
elif filename.endswith('.xlsx'):
return pd.read_excel(filename)
elif filename.endswith('.json'):
return pd.read_json(filename)
else:
raise ValueError(f"Unsupported file format: {filename}")
def preprocess_data(data):
"""Clean and prepare data for visualization."""
# Remove missing values
data = data.dropna()
# Normalize numeric columns
for col in data.select_dtypes(include=[np.number]).columns:
data[col] = (data[col] - data[col].mean()) / data[col].std()
return data
def validate_data(data):
"""Ensure data meets requirements for visualization."""
required_columns = ['x', 'y']
if not all(col in data.columns for col in required_columns):
raise ValueError(f"Data must contain columns: {required_columns}")
return True
mypackage/subpackage/__init__.py - Subpackage initialization:
"""Advanced visualization components."""
from .module import create_dashboard
mypackage/subpackage/module.py - Additional functionality:
"""Dashboard creation module."""
import matplotlib.pyplot as plt
from ..core import plot_data
def create_dashboard(data_list, titles=None, figsize=(12, 8)):
"""
Create a dashboard with multiple plots.
Args:
data_list: List of DataFrames to plot
titles: List of titles for each plot
figsize: Size of the figure
Returns:
matplotlib.Figure: The dashboard figure
"""
n = len(data_list)
if titles is None:
titles = [f"Plot {i+1}" for i in range(n)]
fig, axes = plt.subplots(nrows=n, figsize=figsize)
if n == 1:
axes = [axes]
for i, (data, title) in enumerate(zip(data_list, titles)):
plot_data(data)
axes[i].set_title(title)
fig.tight_layout()
return fig
setup.py - Installation script:
from setuptools import setup, find_packages
setup(
name="mypackage",
version="0.1.0",
packages=find_packages(),
install_requires=[
"matplotlib>=3.4.0",
"seaborn>=0.11.0",
"pandas>=1.3.0",
"numpy>=1.20.0",
],
author="Your Name",
author_email="your.email@example.com",
description="A data visualization library for scientific data",
keywords="visualization, data, science",
url="https://github.com/yourusername/mypackage",
classifiers=[
"Development Status :: 3 - Alpha",
"Intended Audience :: Science/Research",
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
],
python_requires=">=3.7",
)
tests/test_core.py - Core functionality tests:
"""Tests for core visualization functions."""
import pytest
import pandas as pd
import matplotlib.pyplot as plt
from mypackage.core import plot_data, save_figure
@pytest.fixture
def sample_data():
"""Create sample data for testing."""
return pd.DataFrame({
'x': [1, 2, 3, 4, 5],
'y': [10, 15, 13, 17, 20]
})
def test_plot_data_scatter(sample_data):
"""Test scatter plot creation."""
fig = plot_data(sample_data, plot_type='scatter')
assert isinstance(fig, plt.Figure)
# Additional assertions...
def test_plot_data_invalid_type(sample_data):
"""Test error handling for invalid plot types."""
with pytest.raises(ValueError):
plot_data(sample_data, plot_type='invalid_type')
# More tests...
scripts/run_analysis.py - Command-line script:
#!/usr/bin/env python3
"""
Script to run data analysis and generate visualizations.
"""
import argparse
import pandas as pd
from mypackage import load_data, preprocess_data, plot_data, save_figure
def main():
parser = argparse.ArgumentParser(description='Generate data visualizations.')
parser.add_argument('input_file', help='Input data file')
parser.add_argument('output_file', help='Output image file')
parser.add_argument('--plot-type', default='scatter', choices=['scatter', 'line', 'bar'],
help='Type of plot to generate')
args = parser.parse_args()
# Load and process data
raw_data = load_data(args.input_file)
data = preprocess_data(raw_data)
# Create and save visualization
fig = plot_data(data, plot_type=args.plot_type)
save_figure(fig, args.output_file)
print(f"Visualization saved to {args.output_file}")
if __name__ == "__main__":
main()
This structure provides several advantages for medium-sized projects:
- Clear separation of concerns between modules
- Proper packaging allows the library to be installed and imported
- Tests are organized alongside the code they verify
- Command-line scripts provide user-friendly entry points
- Documentation is separated from code but lives in the repository
Large-Scale Application Structure
For large applications, especially web applications, a more sophisticated structure helps manage complexity. Here's a typical layout for a Flask web application:
flask_project/
├── app/ # Application package
│ ├── __init__.py # App initialization
│ ├── config.py # Configuration classes
│ ├── models/ # Database models
│ │ ├── __init__.py
│ │ ├── user.py
│ │ └── product.py
│ ├── routes/ # Route handlers
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ ├── main.py
│ │ └── api.py
│ ├── templates/ # Jinja2 templates
│ │ ├── base.html
│ │ ├── auth/
│ │ └── main/
│ ├── static/ # Static assets
│ │ ├── css/
│ │ ├── js/
│ │ └── img/
│ ├── services/ # Business logic
│ │ ├── __init__.py
│ │ ├── auth_service.py
│ │ └── email_service.py
│ └── utils/ # Utility functions
│ ├── __init__.py
│ └── helpers.py
├── migrations/ # Database migrations
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py
│ ├── test_models/
│ ├── test_routes/
│ └── test_services/
├── logs/ # Application logs
├── scripts/ # Utility scripts
│ ├── deploy.sh
│ └── seed_db.py
├── docs/ # Documentation
├── .env.example # Environment variables template
├── .flaskenv # Flask configurations
├── .gitignore # Git ignore rules
├── requirements/
│ ├── base.txt # Shared dependencies
│ ├── dev.txt # Development dependencies
│ └── prod.txt # Production dependencies
├── setup.py # Package installation
├── wsgi.py # WSGI entry point
├── manage.py # Command-line interface
├── README.md # Project documentation
├── LICENSE # License information
└── docker-compose.yml # Docker configuration
Key features of this structure:
- Highly modular organization by functionality
- Separation of business logic, models, and routes
- Templates and static assets for web interface
- Support for database migrations
- Comprehensive test organization
- DevOps configurations (Docker, deployment scripts)
- Environment-specific requirements
Let's look at some example files from this structure:
app/__init__.py - Application factory:
"""Flask application factory module."""
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_login import LoginManager
# Initialize extensions
db = SQLAlchemy()
migrate = Migrate()
login_manager = LoginManager()
def create_app(config_name='development'):
"""Create and configure the Flask application."""
app = Flask(__name__)
# Load configuration
from app.config import config
app.config.from_object(config[config_name])
# Initialize extensions
db.init_app(app)
migrate.init_app(app, db)
login_manager.init_app(app)
# Set up login manager
from app.models.user import User
login_manager.login_view = 'auth.login'
@login_manager.user_loader
def load_user(user_id):
return User.query.get(int(user_id))
# Register blueprints
from app.routes.main import main_bp
from app.routes.auth import auth_bp
from app.routes.api import api_bp
app.register_blueprint(main_bp)
app.register_blueprint(auth_bp, url_prefix='/auth')
app.register_blueprint(api_bp, url_prefix='/api')
# Register error handlers
from app.routes import errors
errors.register_handlers(app)
return app
app/config.py - Configuration classes:
"""Application configuration module."""
import os
from dotenv import load_dotenv
# Load environment variables from .env file
load_dotenv()
class Config:
"""Base configuration class."""
SECRET_KEY = os.environ.get('SECRET_KEY', 'dev-key-please-change')
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL', 'sqlite:///app.db')
SQLALCHEMY_TRACK_MODIFICATIONS = False
MAIL_SERVER = os.environ.get('MAIL_SERVER', 'smtp.example.com')
MAIL_PORT = int(os.environ.get('MAIL_PORT', 587))
MAIL_USE_TLS = os.environ.get('MAIL_USE_TLS', 'true').lower() == 'true'
MAIL_USERNAME = os.environ.get('MAIL_USERNAME')
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')
MAIL_DEFAULT_SENDER = os.environ.get('MAIL_DEFAULT_SENDER', 'noreply@example.com')
class DevelopmentConfig(Config):
"""Development configuration."""
DEBUG = True
SQLALCHEMY_ECHO = True
class TestingConfig(Config):
"""Testing configuration."""
TESTING = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
WTF_CSRF_ENABLED = False
class ProductionConfig(Config):
"""Production configuration."""
DEBUG = False
# Add production-specific settings here
# Configuration dictionary
config = {
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig,
'default': DevelopmentConfig
}
app/models/user.py - User model:
"""User model module."""
from datetime import datetime
from flask_login import UserMixin
from werkzeug.security import generate_password_hash, check_password_hash
from app import db
class User(db.Model, UserMixin):
"""User model for authentication and profile information."""
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), unique=True, index=True)
email = db.Column(db.String(120), unique=True, index=True)
password_hash = db.Column(db.String(128))
is_active = db.Column(db.Boolean, default=True)
created_at = db.Column(db.DateTime, default=datetime.utcnow)
# Relationships
# posts = db.relationship('Post', backref='author', lazy='dynamic')
def __repr__(self):
return f''
def set_password(self, password):
"""Set the user's password hash."""
self.password_hash = generate_password_hash(password)
def check_password(self, password):
"""Check if the provided password matches the hash."""
return check_password_hash(self.password_hash, password)
app/routes/auth.py - Authentication routes:
"""Authentication routes."""
from flask import Blueprint, render_template, redirect, url_for, flash, request
from flask_login import login_user, logout_user, current_user, login_required
from app.models.user import User
from app.services.auth_service import register_user
from app.forms.auth import LoginForm, RegistrationForm
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login', methods=['GET', 'POST'])
def login():
"""User login view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = LoginForm()
if form.validate_on_submit():
user = User.query.filter_by(username=form.username.data).first()
if user is None or not user.check_password(form.password.data):
flash('Invalid username or password')
return redirect(url_for('auth.login'))
login_user(user, remember=form.remember_me.data)
next_page = request.args.get('next')
if not next_page or not next_page.startswith('/'):
next_page = url_for('main.index')
return redirect(next_page)
return render_template('auth/login.html', title='Sign In', form=form)
@auth_bp.route('/logout')
def logout():
"""User logout view."""
logout_user()
return redirect(url_for('main.index'))
@auth_bp.route('/register', methods=['GET', 'POST'])
def register():
"""User registration view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = RegistrationForm()
if form.validate_on_submit():
user = register_user(
username=form.username.data,
email=form.email.data,
password=form.password.data
)
flash('Congratulations, you are now a registered user!')
return redirect(url_for('auth.login'))
return render_template('auth/register.html', title='Register', form=form)
app/services/auth_service.py - Authentication service:
"""Authentication service module."""
from app import db
from app.models.user import User
from app.services.email_service import send_welcome_email
def register_user(username, email, password):
"""
Register a new user.
Args:
username: User's username
email: User's email address
password: User's password
Returns:
User: The created user object
"""
user = User(username=username, email=email)
user.set_password(password)
db.session.add(user)
db.session.commit()
# Send welcome email asynchronously
send_welcome_email(user)
return user
def verify_user_email(user_id, token):
"""
Verify a user's email address.
Args:
user_id: ID of the user
token: Email verification token
Returns:
bool: True if verification succeeded, False otherwise
"""
user = User.query.get(user_id)
if not user:
return False
# Verify token logic...
user.email_verified = True
db.session.commit()
return True
wsgi.py - WSGI entry point:
"""WSGI entry point for the application."""
from app import create_app
app = create_app('production')
if __name__ == '__main__':
app.run()
manage.py - Command-line interface:
#!/usr/bin/env python
"""Management script for the application."""
import os
import click
from flask.cli import FlaskGroup
from app import create_app, db
from app.models.user import User
app = create_app(os.getenv('FLASK_ENV', 'development'))
cli = FlaskGroup(app)
@cli.command('create_admin')
@click.argument('username')
@click.argument('email')
@click.password_option()
def create_admin(username, email, password):
"""Create an admin user."""
user = User(username=username, email=email, is_admin=True)
user.set_password(password)
db.session.add(user)
db.session.commit()
click.echo(f'Admin user {username} created.')
@cli.command('reset_db')
@click.confirmation_option(prompt='Are you sure you want to reset the database?')
def reset_db():
"""Reset the database."""
db.drop_all()
db.create_all()
click.echo('Database has been reset.')
if __name__ == '__main__':
cli()
The large-scale structure provides these advantages:
- Highly modular with clear separation of concerns
- Routes, models, services, and utilities are distinct
- The application factory pattern enables different configurations
- Multiple entry points for different use cases (WSGI, CLI)
- Blueprints organize routes by feature
- Services layer isolates business logic from routes
Advanced Structure: Domain-Driven Design
For very large and complex applications, especially those with multiple developers or teams, Domain-Driven Design (DDD) offers a powerful organizing principle. In DDD, the codebase is organized around business domains rather than technical layers:
ddd_project/
├── src/
│ ├── users/ # User domain
│ │ ├── __init__.py
│ │ ├── models.py # User domain models
│ │ ├── repositories.py # User data access
│ │ ├── services.py # User business logic
│ │ └── routes.py # User API endpoints
│ ├── products/ # Product domain
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── repositories.py
│ │ ├── services.py
│ │ └── routes.py
│ ├── orders/ # Order domain
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── repositories.py
│ │ ├── services.py
│ │ └── routes.py
│ └── shared/ # Shared components
│ ├── __init__.py
│ ├── database.py # Database connection
│ ├── security.py # Security utilities
│ └── utils.py # Common utilities
├── app/ # Application assembly
│ ├── __init__.py
│ ├── config.py # Configuration
│ ├── api.py # API initialization
│ └── cli.py # CLI commands
├── tests/ # Tests by domain
│ ├── users/
│ ├── products/
│ └── orders/
└── infrastructure/ # Infrastructure concerns
├── database/ # Database scripts
├── logging/ # Logging configuration
└── messaging/ # Message queue setup
In this structure:
- Each domain module contains all the code related to that domain, regardless of type (models, services, etc.)
- The shared module contains code used across domains
- The app module wires everything together
- The infrastructure module handles technical concerns
DDD shines in complex applications where:
- The business domain is complex with many rules and processes
- Multiple teams work on different parts of the application
- Different parts of the system evolve at different rates
- Long-term maintainability is crucial
This structure facilitates:
- Better alignment with business domains
- Clearer boundaries between system components
- Easier parallel development by multiple teams
- More focused testing of domain logic
Choosing the Right Structure
Selecting the appropriate project structure depends on several factors:
Project Size and Complexity
| Size | Characteristics | Recommended Structure |
|---|---|---|
| Small |
|
Simple script structure |
| Medium |
|
Package-based structure |
| Large |
|
Application structure or DDD |
Project Type
| Type | Recommended Structure |
|---|---|
| Command-line tool | Simple or medium structure with scripts/ directory |
| Library/package | Medium structure with clear API |
| Web application | Large application structure |
| Enterprise application | Domain-driven design |
Team Considerations
- Team Size: Larger teams need more structure to coordinate effectively
- Team Experience: Less experienced teams may benefit from more prescriptive structures
- Team Organization: If teams are organized by feature, DDD may align better
Evolution Strategy
It's important to remember that project structure can evolve as your application grows:
- Start with a simple structure suitable for your current needs
- As complexity increases, refactor toward a more structured organization
- Follow the principle of "emergent design" rather than over-engineering initially
Tip: For new projects, choose a structure that's slightly more sophisticated than what you think you need. This provides room to grow without requiring immediate restructuring.
Project Structure Best Practices
Regardless of the specific structure you choose, these best practices will help keep your project organized and maintainable:
1. Follow the Single Responsibility Principle
Each module or package should have a single, well-defined purpose. If a module is doing too many things, it's time to split it.
Good: Separate auth_service.py, email_service.py, and payment_service.py
Bad: A single services.py with all service functions mixed together
2. Keep Related Files Together
Files that change together should be located near each other in the directory structure.
Good: Putting user model, repository, service, and routes in a users/ package
Bad: Spreading user-related code across many different directories based solely on type
3. Create Clear API Boundaries
Make it obvious what parts of your code are public interfaces vs. internal implementation details.
Good: Using __init__.py to expose only the public interface, using underscore prefixes for internal functions
Bad: No clear distinction between public and private components
4. Separate Configuration from Code
Configuration settings should be separated from application logic.
Good: Using environment variables, config files, or a dedicated config.py module
Bad: Hardcoding configuration values throughout the codebase
5. Keep the Root Directory Clean
The project root should contain only high-level files and directories.
Good: Moving implementation details into subdirectories, keeping only essential files at the root
Bad: Dozens of Python files in the root directory
6. Use Common Conventions
Follow established naming and organization conventions to make your project more approachable.
Good: Using standard names like tests/, docs/, and README.md
Bad: Inventing unique naming schemes that deviate from community standards
7. Include Essential Project Files
Every project should include certain standard files.
Essential files:
README.md- Project overview, installation instructions, basic usage examplesLICENSE- The project's license termsrequirements.txtor equivalent - Project dependencies.gitignore- Files for Git to ignore- Setup or build script (
setup.py,pyproject.toml, etc.)
8. Document Your Structure
Make sure new team members can understand your project organization.
Good: Including a "Project Structure" section in your README or documentation
Bad: Assuming others will intuitively understand your organization
9. Test Directory Structure Should Mirror Source
Organize tests to reflect the structure of the code they test.
Good: If you have app/models/user.py, create tests/test_models/test_user.py
Bad: Test organization that doesn't correspond to the source structure
10. Use Consistent Import Style
Establish and follow consistent rules for imports within your project.
Good: Using absolute imports from the project root for clarity
Bad: Mixing absolute and relative imports inconsistently
Common Structure Patterns
Several structural patterns appear across many Python projects. Understanding these can help you recognize and apply common organizational strategies:
1. Application Factory Pattern
Creating the application object via a factory function rather than globally. This pattern is especially common in Flask applications.
# app/__init__.py
def create_app(config_name='development'):
app = Flask(__name__)
# Configure the app
# Register extensions, blueprints, etc.
return app
Benefits:
- Enables different configurations for different environments
- Simplifies testing by allowing test-specific configurations
- Avoids circular import issues
2. Blueprint Pattern
Organizing routes into modular components, often by feature area. Common in Flask and similar frameworks.
# app/routes/auth.py
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login')
def login():
# Login logic
# app/__init__.py
from app.routes.auth import auth_bp
app.register_blueprint(auth_bp, url_prefix='/auth')
Benefits:
- Modular organization of routes
- Separate URL prefixes for different areas
- Enables parallel development
3. Repository Pattern
Abstracting data access behind repository interfaces.
# app/repositories/user_repository.py
class UserRepository:
def get_by_id(self, user_id):
# Query the database
return User.query.get(user_id)
def create(self, user_data):
# Create a new user
user = User(**user_data)
db.session.add(user)
db.session.commit()
return user
Benefits:
- Abstracts data access details from business logic
- Makes testing easier through mocking
- Allows changing the data storage mechanism without affecting other code
4. Service Layer Pattern
Encapsulating business logic in service modules separate from routes and models.
# app/services/auth_service.py
def authenticate_user(username, password):
user = User.query.filter_by(username=username).first()
if user and user.check_password(password):
return user
return None
def register_user(username, email, password):
user = User(username=username, email=email)
user.set_password(password)
db.session.add(user)
db.session.commit()
send_welcome_email(user) # Side effect managed by service
return user
Benefits:
- Separates business logic from HTTP handling
- Makes business logic more testable
- Can be reused across different entry points (web, CLI, API)
5. Command Pattern
Encapsulating operations as command objects. Common in CLI-heavy applications and task queues.
# app/commands/user_commands.py
class CreateUserCommand:
def __init__(self, username, email, password):
self.username = username
self.email = email
self.password = password
def execute(self):
user = User(username=self.username, email=self.email)
user.set_password(self.password)
db.session.add(user)
db.session.commit()
return user
Benefits:
- Encapsulates operations in a clean way
- Enables features like command queuing, logging, and undo
- Simplifies complex operation sequences
6. Settings Module Pattern
Centralizing configuration in a dedicated module, often with environment-specific subclasses.
# app/config.py
class Config:
DEBUG = False
SECRET_KEY = os.environ.get('SECRET_KEY', 'dev-key')
class DevelopmentConfig(Config):
DEBUG = True
class ProductionConfig(Config):
# Production-specific settings
Benefits:
- Centralizes configuration
- Supports different environments
- Makes configuration explicit and documented
Tools for Project Structure Management
Several tools can help you create and maintain good project structures:
Project Templates
- cookiecutter: Generate projects from templates
pip install cookiecutter cookiecutter https://github.com/audreyfeldroy/cookiecutter-pypackage - Framework CLI tools: Most frameworks have tools to generate projects
django-admin startproject myproject flask create-app --name myapp
Code Quality Tools
- pylint: Check code quality and enforces structure conventions
pip install pylint pylint mypackage - flake8: Similar to pylint but more focused on style
pip install flake8 flake8 mypackage - black: Automatic code formatting
pip install black black mypackage - isort: Organize imports consistently
pip install isort isort mypackage
Documentation Tools
- Sphinx: Generate documentation from docstrings
pip install sphinx sphinx-quickstart - mkdocs: Simpler Markdown-based documentation
pip install mkdocs mkdocs new myproject
Project Structure Analysis
- pyan3: Analyze and visualize dependencies
pip install pyan3 pyan3 mypackage/*.py --dot > deps.dot dot -Tpng deps.dot -o deps.png - pipdeptree: Visualize package dependencies
pip install pipdeptree pipdeptree
Real-World Examples
Examining popular open-source projects can provide valuable insights into effective project structures. Here are some notable examples:
Flask (Web Framework)
Flask uses a simple, package-based structure that's easy to understand:
flask/
├── docs/ # Documentation
├── examples/ # Example applications
├── src/
│ └── flask/ # Main package
│ ├── __init__.py # Public API
│ ├── app.py # Application object
│ ├── blueprints.py # Blueprint support
│ ├── cli.py # Command-line interface
│ ├── config.py # Configuration handling
│ ├── globals.py # Global objects
│ ├── sessions.py # Session handling
│ ├── templating.py # Template support
│ └── ... # Other modules
├── tests/ # Test suite
├── setup.py # Package setup
└── pyproject.toml # Project metadata
Key Insights:
- Clear separation of public API in __init__.py
- Modules organized by functionality
- Tests separate from source code
- Examples directory for learning
Django (Web Framework)
Django uses a more complex structure reflecting its larger scope:
django/
├── django/ # Main package
│ ├── __init__.py
│ ├── apps/ # Application registry
│ ├── conf/ # Settings and configuration
│ ├── contrib/ # Bundled apps (admin, auth, etc.)
│ ├── core/ # Core functionality
│ ├── db/ # Database layer
│ ├── http/ # HTTP handling
│ ├── template/ # Template system
│ ├── urls/ # URL routing
│ ├── utils/ # Utilities
│ └── views/ # View functions
├── docs/ # Documentation
├── tests/ # Test suite
├── scripts/ # Utility scripts
└── setup.py # Package setup
Key Insights:
- Highly modular with clear boundaries
- Separation of core framework from bundled apps
- Utils package for shared functionality
- Each area of functionality has its own package
Requests (HTTP Library)
Requests uses a simpler structure appropriate for its focused purpose:
requests/
├── requests/ # Main package
│ ├── __init__.py # Public API
│ ├── api.py # API functionality
│ ├── models.py # Data models
│ ├── sessions.py # Session handling
│ ├── structures.py # Data structures
│ └── utils.py # Utilities
├── docs/ # Documentation
├── tests/ # Test suite
└── setup.py # Package setup
Key Insights:
- Simple flat structure for a focused library
- Clear separation of concerns between modules
- Public API exposed in __init__.py
- Comprehensive test suite
SQLAlchemy (ORM Library)
SQLAlchemy uses a more complex structure suitable for its comprehensive nature:
sqlalchemy/
├── lib/
│ └── sqlalchemy/ # Main package
│ ├── __init__.py
│ ├── engine/ # Database engine
│ ├── ext/ # Extensions
│ ├── orm/ # Object-relational mapping
│ ├── sql/ # SQL expression language
│ └── ... # Other modules
├── doc/ # Documentation
├── test/ # Test suite
├── examples/ # Example applications
└── setup.py # Package setup
Key Insights:
- Hierarchical structure for a complex library
- Clear separation between core components (engine, ORM, etc.)
- Extensions in a separate package
- Examples provided for learning
Exercise: Refactoring a Project Structure
Let's practice by refactoring a poorly structured project into a well-organized one:
Original Structure (Poorly Organized)
messy_project/
├── app.py # Contains everything: models, routes, business logic
├── utils.py # Miscellaneous utilities
├── templates/ # HTML templates
├── static/ # Static assets
├── test.py # All tests in one file
└── requirements.txt # Dependencies
The app.py file is over 2,000 lines long and contains:
- Database models
- Route handlers
- Business logic
- Authentication code
- Utility functions
- Configuration settings
Task: Refactor this into a well-structured Flask application
Steps:
- Create a proper package structure
- Separate models, views, and business logic
- Organize templates and static files
- Move configuration to a dedicated module
- Structure tests properly
- Add necessary project files
Target Structure:
organized_project/
├── app/ # Application package
│ ├── __init__.py # Application factory
│ ├── config.py # Configuration
│ ├── models/ # Database models
│ │ ├── __init__.py
│ │ ├── user.py
│ │ └── item.py
│ ├── routes/ # Route handlers
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ └── main.py
│ ├── services/ # Business logic
│ │ ├── __init__.py
│ │ └── item_service.py
│ ├── templates/ # HTML templates
│ │ ├── base.html
│ │ ├── auth/
│ │ └── main/
│ ├── static/ # Static assets
│ │ ├── css/
│ │ ├── js/
│ │ └── img/
│ └── utils/ # Utilities
│ ├── __init__.py
│ └── helpers.py
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py # Test fixtures
│ ├── test_models/
│ ├── test_routes/
│ └── test_services/
├── .env.example # Environment variables template
├── .gitignore # Git ignore rules
├── README.md # Project documentation
├── requirements.txt # Dependencies
└── wsgi.py # WSGI entry point
Implementation Steps
- Create the directory structure
# Create directories mkdir -p organized_project/app/{models,routes,services,templates/{auth,main},static/{css,js,img},utils} mkdir -p organized_project/tests/{test_models,test_routes,test_services} touch organized_project/app/{__init__.py,config.py} touch organized_project/app/{models,routes,services,utils}/__init__.py touch organized_project/tests/{__init__.py,conftest.py} touch organized_project/{.env.example,.gitignore,README.md,requirements.txt,wsgi.py} - Extract models from app.py
# Extract user model # From app.py: # class User(db.Model): # ... # Create app/models/user.py: """User model module.""" from app import db from flask_login import UserMixin from werkzeug.security import generate_password_hash, check_password_hash class User(db.Model, UserMixin): id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String(64), unique=True, index=True) # ... # Extract item model # Create app/models/item.py """Item model module.""" from app import db from datetime import datetime class Item(db.Model): id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(100), nullable=False) # ... - Extract routes from app.py
# Extract auth routes # Create app/routes/auth.py """Authentication routes.""" from flask import Blueprint, render_template, redirect, url_for, flash, request from flask_login import login_user, logout_user, current_user, login_required from app.models.user import User from app.services.auth_service import register_user from app.forms.auth import LoginForm, RegistrationForm auth_bp = Blueprint('auth', __name__) @auth_bp.route('/login', methods=['GET', 'POST']) def login(): # Login logic... # Extract main routes # Create app/routes/main.py """Main application routes.""" from flask import Blueprint, render_template, request from flask_login import login_required from app.models.item import Item from app.services.item_service import get_items, create_item main_bp = Blueprint('main', __name__) @main_bp.route('/') def index(): # Homepage logic... - Extract business logic to services
# Create app/services/item_service.py """Item service module for business import matplotlib.pyplot as plt import seaborn as sns from .helpers import validate_data from .exceptions import PlotTypeError def plot_data(data, plot_type='scatter', title=None, xlabel=None, ylabel=None, **kwargs): """ Create a visualization from the provided data. Args: data: Pandas DataFrame with the data to plot plot_type: Type of plot to create (scatter, line, bar, histogram) title: Title for the plot xlabel: Label for x-axis ylabel: Label for y-axis **kwargs: Additional arguments for the plot Returns: matplotlib.Figure: The created figure Raises: PlotTypeError: If the plot type is not supported ValueError: If data doesn't contain required columns """ # Validate data has required columns validate_data(data) # Set up the figure and axes fig, ax = plt.subplots(figsize=kwargs.pop('figsize', (10, 6))) # Create the plot based on type if plot_type == 'scatter': ax.scatter(data['x'], data['y'], **kwargs) elif plot_type == 'line': ax.plot(data['x'], data['y'], **kwargs) elif plot_type == 'bar': ax.bar(data['x'], data['y'], **kwargs) elif plot_type == 'histogram': ax.hist(data['x'], bins=kwargs.pop('bins', 10), **kwargs) else: raise PlotTypeError(f"Unsupported plot type: {plot_type}") # Add labels and title if provided if xlabel: ax.set_xlabel(xlabel) if ylabel: ax.set_ylabel(ylabel) if title: ax.set_title(title) # Apply styling sns.set_style("whitegrid") fig.tight_layout() return fig def save_figure(fig, filename, dpi=300, transparent=False): """ Save a figure to a file. Args: fig: Matplotlib figure to save filename: Output filename dpi: Resolution in dots per inch transparent: Whether to use a transparent background """ fig.savefig(filename, dpi=dpi, transparent=transparent) return filenamedatavis/helpers.py - Helper functions:
"""Helper functions for data handling and preprocessing.""" import pandas as pd import numpy as np import os from .exceptions import FileFormatError, DataValidationError def load_data(filename): """ Load data from a file into a DataFrame. Args: filename: Path to the data file Returns: pandas.DataFrame: Loaded data Raises: FileFormatError: If the file format is not supported FileNotFoundError: If the file doesn't exist """ if not os.path.exists(filename): raise FileNotFoundError(f"File not found: {filename}") # Determine file type from extension if filename.endswith('.csv'): return pd.read_csv(filename) elif filename.endswith('.xlsx') or filename.endswith('.xls'): return pd.read_excel(filename) elif filename.endswith('.json'): return pd.read_json(filename) elif filename.endswith('.pickle') or filename.endswith('.pkl'): return pd.read_pickle(filename) else: raise FileFormatError(f"Unsupported file format: {filename}") def preprocess_data(data, normalize=True, handle_missing=True): """ Clean and prepare data for visualization. Args: data: pandas DataFrame to process normalize: Whether to normalize numeric columns handle_missing: Whether to handle missing values Returns: pandas.DataFrame: Processed data """ # Work on a copy to avoid modifying the original df = data.copy() # Handle missing values if requested if handle_missing: # Drop rows with any missing values df = df.dropna() # Normalize numeric columns if requested if normalize: numeric_cols = df.select_dtypes(include=[np.number]).columns for col in numeric_cols: # Skip columns with zero standard deviation if df[col].std() > 0: df[col] = (df[col] - df[col].mean()) / df[col].std() return df def validate_data(data): """ Ensure data meets requirements for visualization. Args: data: pandas DataFrame to validate Raises: DataValidationError: If data doesn't meet requirements """ required_columns = ['x', 'y'] # Check if data is a DataFrame if not isinstance(data, pd.DataFrame): raise DataValidationError("Data must be a pandas DataFrame") # Check if required columns exist if not all(col in data.columns for col in required_columns): raise DataValidationError(f"Data must contain columns: {required_columns}") # Check if data has any rows if len(data) == 0: raise DataValidationError("Data contains no rows") return Truedatavis/exceptions.py - Custom exceptions:
"""Custom exceptions for the DataVis package.""" class DataVisError(Exception): """Base exception for all DataVis errors.""" pass class PlotTypeError(DataVisError): """Raised when an unsupported plot type is requested.""" pass class FileFormatError(DataVisError): """Raised when an unsupported file format is encountered.""" pass class DataValidationError(DataVisError): """Raised when data fails validation checks.""" passdatavis/advanced/__init__.py - Advanced subpackage initialization:
"""Advanced visualization components for interactive and animated plots.""" from .interactive import create_interactive_plot from .animations import create_animation __all__ = ['create_interactive_plot', 'create_animation']datavis/advanced/interactive.py - Interactive visualization:
"""Interactive visualization module.""" import matplotlib.pyplot as plt from matplotlib.widgets import Slider, Button from ..core import plot_data def create_interactive_plot(data, x_column='x', y_column='y', **kwargs): """ Create an interactive plot with sliders to control display parameters. Args: data: DataFrame with the data to plot x_column: Column to use for x-axis y_column: Column to use for y-axis **kwargs: Additional arguments for the plot Returns: matplotlib.Figure: The interactive figure """ # Create the initial plot fig, ax = plt.subplots(figsize=kwargs.pop('figsize', (10, 8))) plt.subplots_adjust(bottom=0.25) # Make room for sliders # Initial plotting scatter = ax.scatter( data[x_column], data[y_column], alpha=0.5, s=50 # Size of points ) # Add sliders ax_alpha = plt.axes([0.25, 0.15, 0.65, 0.03]) alpha_slider = Slider(ax_alpha, 'Alpha', 0.1, 1.0, valinit=0.5) ax_size = plt.axes([0.25, 0.1, 0.65, 0.03]) size_slider = Slider(ax_size, 'Size', 10, 200, valinit=50) # Update function for sliders def update(val): scatter.set_alpha(alpha_slider.val) scatter.set_sizes([size_slider.val]) fig.canvas.draw_idle() # Connect the sliders to the update function alpha_slider.on_changed(update) size_slider.on_changed(update) # Add a reset button ax_reset = plt.axes([0.8, 0.025, 0.1, 0.04]) reset_button = Button(ax_reset, 'Reset') def reset(event): alpha_slider.reset() size_slider.reset() reset_button.on_clicked(reset) # Set labels and title ax.set_xlabel(x_column) ax.set_ylabel(y_column) ax.set_title(kwargs.pop('title', 'Interactive Plot')) return figsetup.py - Installation script:
from setuptools import setup, find_packages # Read long description from README with open("README.md", "r", encoding="utf-8") as fh: long_description = fh.read() setup( name="datavis", version="0.2.1", packages=find_packages(), install_requires=[ "matplotlib>=3.4.0", "seaborn>=0.11.0", "pandas>=1.3.0", "numpy>=1.20.0", ], author="Your Name", author_email="your.email@example.com", description="A data visualization library for scientific data", long_description=long_description, long_description_content_type="text/markdown", keywords="visualization, data, science, matplotlib", url="https://github.com/yourusername/datavis", classifiers=[ "Development Status :: 3 - Alpha", "Intended Audience :: Science/Research", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "License :: OSI Approved :: MIT License", "Topic :: Scientific/Engineering :: Visualization", ], python_requires=">=3.7", )tests/test_core.py - Core functionality tests:
"""Tests for core visualization functions.""" import pytest import pandas as pd import matplotlib.pyplot as plt from datavis.core import plot_data, save_figure from datavis.exceptions import PlotTypeError import os import tempfile @pytest.fixture def sample_data(): """Create sample data for testing.""" return pd.DataFrame({ 'x': [1, 2, 3, 4, 5], 'y': [10, 15, 13, 17, 20] }) def test_plot_data_scatter(sample_data): """Test scatter plot creation.""" fig = plot_data(sample_data, plot_type='scatter') assert isinstance(fig, plt.Figure) # Check that figure has expected components assert len(fig.axes) == 1 ax = fig.axes[0] assert len(ax.collections) == 1 # One scatter plot # Cleanup plt.close(fig) def test_plot_data_line(sample_data): """Test line plot creation.""" fig = plot_data(sample_data, plot_type='line') assert isinstance(fig, plt.Figure) # Check that figure has expected components assert len(fig.axes) == 1 ax = fig.axes[0] assert len(ax.lines) == 1 # One line # Cleanup plt.close(fig) def test_plot_data_with_labels(sample_data): """Test plot creation with labels and title.""" title = "Test Plot" xlabel = "X Values" ylabel = "Y Values" fig = plot_data(sample_data, title=title, xlabel=xlabel, ylabel=ylabel) ax = fig.axes[0] assert ax.get_title() == title assert ax.get_xlabel() == xlabel assert ax.get_ylabel() == ylabel # Cleanup plt.close(fig) def test_plot_data_invalid_type(sample_data): """Test error handling for invalid plot types.""" with pytest.raises(PlotTypeError): plot_data(sample_data, plot_type='invalid_type') def test_save_figure(sample_data): """Test saving a figure to a file.""" fig = plot_data(sample_data) # Create a temporary file with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp: filename = tmp.name try: # Save the figure saved_path = save_figure(fig, filename) # Check that the file exists and has content assert os.path.exists(saved_path) assert os.path.getsize(saved_path) > 0 finally: # Cleanup plt.close(fig) if os.path.exists(filename): os.unlink(filename)scripts/generate_report.py - Command-line script:
#!/usr/bin/env python3 """ Script to generate a data visualization report. """ import argparse import pandas as pd import os import sys from pathlib import Path # Add parent directory to path for importing datavis package sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) from datavis import load_data, preprocess_data, plot_data, save_figure from datavis.advanced import create_interactive_plot def main(): """Main entry point for the script.""" parser = argparse.ArgumentParser(description='Generate a data visualization report.') parser.add_argument('input_file', help='Input data file (CSV, Excel, JSON)') parser.add_argument('output_dir', help='Output directory for report files') parser.add_argument('--x-col', default='x', help='Column to use for x-axis') parser.add_argument('--y-col', default='y', help='Column to use for y-axis') parser.add_argument('--title', default='Data Visualization', help='Plot title') parser.add_argument('--types', default='scatter,line,bar', help='Comma-separated list of plot types to generate') args = parser.parse_args() # Ensure output directory exists os.makedirs(args.output_dir, exist_ok=True) try: # Load and process data print(f"Loading data from {args.input_file}...") raw_data = load_data(args.input_file) # Rename columns if needed if args.x_col != 'x' or args.y_col != 'y': raw_data = raw_data.rename(columns={args.x_col: 'x', args.y_col: 'y'}) print(f"Processing data ({len(raw_data)} rows)...") data = preprocess_data(raw_data) print(f"Data processed. {len(data)} rows after preprocessing.") # Generate plots plot_types = args.types.split(',') for plot_type in plot_types: plot_type = plot_type.strip() print(f"Generating {plot_type} plot...") fig = plot_data( data, plot_type=plot_type, title=f"{args.title} - {plot_type.capitalize()}", xlabel=args.x_col, ylabel=args.y_col ) # Save the figure output_file = os.path.join(args.output_dir, f"{plot_type}_plot.png") save_figure(fig, output_file) print(f"Saved to {output_file}") print("Report generation complete!") except Exception as e: print(f"Error generating report: {str(e)}") return 1 return 0 if __name__ == "__main__": sys.exit(main())This medium-sized structure provides several advantages:
- Clear separation of concerns between core functionality, helpers, and advanced features
- Well-defined API with proper error handling
- Comprehensive testing of functionality
- Proper packaging for distribution
- Command-line interface for common operations
- Structured documentation and examples
When to use this structure:
- For libraries that will be shared across projects
- When developing tools with a clear API
- For projects that will grow over time but maintain a focused purpose
- When testing is important for maintaining reliability
When to graduate from this structure: As your project continues to grow in complexity, you might notice:
- The package becoming too large to understand as a whole
- Different components evolving at different rates
- Need for more formalized boundaries between components
- Multiple teams working on different aspects of the project
At this point, consider moving to a large-scale application structure or domain-driven design approach.
Large-Scale Application Structure
For large applications, especially web applications or complex systems, a more sophisticated structure helps manage the inherent complexity. This approach is like organizing a large factory with multiple production lines, specialized departments, and clear workflows. Here's a typical layout for a Flask web application:
flask_ecommerce/
├── app/ # Application package
│ ├── __init__.py # App initialization
│ ├── config.py # Configuration classes
│ ├── models/ # Database models
│ │ ├── __init__.py
│ │ ├── user.py
│ │ ├── product.py
│ │ └── order.py
│ ├── routes/ # Route handlers
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ ├── main.py
│ │ ├── products.py
│ │ └── api.py
│ ├── templates/ # Jinja2 templates
│ │ ├── base.html
│ │ ├── index.html
│ │ ├── auth/
│ │ └── products/
│ ├── static/ # Static assets
│ │ ├── css/
│ │ ├── js/
│ │ └── img/
│ ├── services/ # Business logic
│ │ ├── __init__.py
│ │ ├── auth_service.py
│ │ ├── product_service.py
│ │ └── email_service.py
│ └── utils/ # Utility functions
│ ├── __init__.py
│ ├── validators.py
│ └── formatters.py
├── migrations/ # Database migrations
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py
│ ├── test_models/
│ ├── test_routes/
│ └── test_services/
├── logs/ # Application logs
├── scripts/ # Utility scripts
│ ├── deploy.sh
│ └── seed_db.py
├── docs/ # Documentation
│ ├── index.md
│ ├── api.md
│ └── deployment.md
├── .env.example # Environment variables template
├── .flaskenv # Flask configurations
├── .gitignore # Git ignore rules
├── requirements/
│ ├── base.txt # Shared dependencies
│ ├── dev.txt # Development dependencies
│ └── prod.txt # Production dependencies
├── setup.py # Package installation
├── wsgi.py # WSGI entry point
├── manage.py # Command-line interface
├── README.md # Project documentation
├── LICENSE # License information
└── docker-compose.yml # Docker configuration
Key features of this structure:
- Highly modular organization by functionality
- Clear separation between presentation, business logic, and data access
- Support for multiple environments (development, testing, production)
- Advanced configuration management
- Comprehensive testing structure
- DevOps integration
- Database migration support
This structure works well for:
- Web applications with multiple features
- Projects with 10,000+ lines of code
- Teams with 5+ developers
- Applications that will be deployed to multiple environments
- Projects with complex business logic
Example Files in a Large Application
Let's explore some key files from this large-scale structure:
app/__init__.py - Application factory:
"""Flask application factory module."""
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_login import LoginManager
from flask_mail import Mail
import logging
from logging.handlers import RotatingFileHandler
import os
# Initialize extensions
db = SQLAlchemy()
migrate = Migrate()
login_manager = LoginManager()
mail = Mail()
def create_app(config_name='development'):
"""Create and configure the Flask application."""
app = Flask(__name__)
# Load configuration
from app.config import config
app.config.from_object(config[config_name])
# Initialize extensions
db.init_app(app)
migrate.init_app(app, db)
login_manager.init_app(app)
mail.init_app(app)
# Set up login manager
from app.models.user import User
login_manager.login_view = 'auth.login'
login_manager.login_message = 'Please log in to access this page.'
@login_manager.user_loader
def load_user(user_id):
return User.query.get(int(user_id))
# Register blueprints
from app.routes.main import main_bp
from app.routes.auth import auth_bp
from app.routes.products import products_bp
from app.routes.api import api_bp
app.register_blueprint(main_bp)
app.register_blueprint(auth_bp, url_prefix='/auth')
app.register_blueprint(products_bp, url_prefix='/products')
app.register_blueprint(api_bp, url_prefix='/api/v1')
# Set up logging
if not app.debug and not app.testing:
if not os.path.exists('logs'):
os.mkdir('logs')
file_handler = RotatingFileHandler(
'logs/flask_ecommerce.log',
maxBytes=10240,
backupCount=10
)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'
))
file_handler.setLevel(logging.INFO)
app.logger.addHandler(file_handler)
app.logger.setLevel(logging.INFO)
app.logger.info('Flask E-commerce startup')
# Register error handlers
from app.routes import errors
errors.register_handlers(app)
return app
app/config.py - Configuration classes:
"""Application configuration module."""
import os
from dotenv import load_dotenv
# Load environment variables from .env file
basedir = os.path.abspath(os.path.dirname(__file__))
load_dotenv(os.path.join(os.path.dirname(basedir), '.env'))
class Config:
"""Base configuration class."""
SECRET_KEY = os.environ.get('SECRET_KEY', 'dev-key-please-change')
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL', 'sqlite:///' + os.path.join(basedir, 'app.db'))
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Mail settings
MAIL_SERVER = os.environ.get('MAIL_SERVER', 'smtp.example.com')
MAIL_PORT = int(os.environ.get('MAIL_PORT', 587))
MAIL_USE_TLS = os.environ.get('MAIL_USE_TLS', 'true').lower() == 'true'
MAIL_USERNAME = os.environ.get('MAIL_USERNAME')
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')
MAIL_DEFAULT_SENDER = os.environ.get('MAIL_DEFAULT_SENDER', 'noreply@example.com')
# Application settings
PRODUCTS_PER_PAGE = 12
ADMIN_EMAIL = os.environ.get('ADMIN_EMAIL', 'admin@example.com')
UPLOAD_FOLDER = os.path.join(basedir, 'static', 'uploads')
ALLOWED_EXTENSIONS = {'png', 'jpg', 'jpeg', 'gif'}
class DevelopmentConfig(Config):
"""Development configuration."""
DEBUG = True
SQLALCHEMY_ECHO = True
TEMPLATES_AUTO_RELOAD = True
class TestingConfig(Config):
"""Testing configuration."""
TESTING = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
WTF_CSRF_ENABLED = False
MAIL_SUPPRESS_SEND = True
class ProductionConfig(Config):
"""Production configuration."""
DEBUG = False
@classmethod
def init_app(cls, app):
# Log to stderr
import logging
from logging import StreamHandler
file_handler = StreamHandler()
file_handler.setLevel(logging.INFO)
app.logger.addHandler(file_handler)
# Production-specific middleware, etc.
from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app)
# Configuration dictionary
config = {
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig,
'default': DevelopmentConfig
}
app/models/user.py - User model:
"""User model module."""
from datetime import datetime
from flask import current_app
from flask_login import UserMixin
from werkzeug.security import generate_password_hash, check_password_hash
import jwt
from time import time
from app import db
class User(db.Model, UserMixin):
"""User model for authentication and profile information."""
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), unique=True, index=True)
email = db.Column(db.String(120), unique=True, index=True)
password_hash = db.Column(db.String(128))
first_name = db.Column(db.String(64))
last_name = db.Column(db.String(64))
is_active = db.Column(db.Boolean, default=True)
is_admin = db.Column(db.Boolean, default=False)
created_at = db.Column(db.DateTime, default=datetime.utcnow)
last_login = db.Column(db.DateTime, nullable=True)
# Relationships
orders = db.relationship('Order', backref='customer', lazy='dynamic')
def __repr__(self):
return f''
def set_password(self, password):
"""Set the user's password hash."""
self.password_hash = generate_password_hash(password)
def check_password(self, password):
"""Check if the provided password matches the hash."""
return check_password_hash(self.password_hash, password)
def get_reset_password_token(self, expires_in=600):
"""Generate a token for password reset."""
return jwt.encode(
{'reset_password': self.id, 'exp': time() + expires_in},
current_app.config['SECRET_KEY'],
algorithm='HS256'
)
@staticmethod
def verify_reset_password_token(token):
"""Verify a password reset token."""
try:
id = jwt.decode(
token,
current_app.config['SECRET_KEY'],
algorithms=['HS256']
)['reset_password']
except:
return None
return User.query.get(id)
def to_dict(self):
"""Convert user to dictionary for API responses."""
return {
'id': self.id,
'username': self.username,
'email': self.email,
'first_name': self.first_name,
'last_name': self.last_name,
'is_admin': self.is_admin,
'created_at': self.created_at.isoformat() + 'Z',
'last_login': self.last_login.isoformat() + 'Z' if self.last_login else None
}
app/services/auth_service.py - Authentication service:
"""Authentication service module."""
from datetime import datetime
from flask import current_app
from app import db
from app.models.user import User
from app.services.email_service import send_email
import jwt
from time import time
def register_user(username, email, password, first_name=None, last_name=None):
"""
Register a new user.
Args:
username: User's username
email: User's email address
password: User's password
first_name: User's first name (optional)
last_name: User's last name (optional)
Returns:
User: The created user object
Raises:
ValueError: If username or email already exists
"""
# Check if username or email already exists
if User.query.filter_by(username=username).first():
raise ValueError(f"Username '{username}' is already taken")
if User.query.filter_by(email=email).first():
raise ValueError(f"Email '{email}' is already registered")
# Create the user
user = User(
username=username,
email=email,
first_name=first_name,
last_name=last_name
)
user.set_password(password)
# Make first user an admin
if User.query.count() == 0:
user.is_admin = True
# Save to database
db.session.add(user)
db.session.commit()
# Send welcome email
send_welcome_email(user)
return user
def login_user_service(user):
"""
Update user login information.
Args:
user: User object
"""
user.last_login = datetime.utcnow()
db.session.commit()
def send_welcome_email(user):
"""
Send a welcome email to a new user.
Args:
user: User object
"""
subject = "Welcome to Flask E-commerce!"
text_body = f"""
Hi {user.first_name or user.username},
Thank you for registering with Flask E-commerce!
Your account has been created successfully. You can now log in at:
{current_app.config['SITE_URL']}/auth/login
Best regards,
The Flask E-commerce Team
"""
html_body = f"""
Hi {user.first_name or user.username},
Thank you for registering with Flask E-commerce!
Your account has been created successfully. You can now
log in.
Best regards,
The Flask E-commerce Team
"""
send_email(user.email, subject, text_body, html_body)
def generate_password_reset_token(email):
"""
Generate a password reset token for the given email.
Args:
email: User's email address
Returns:
tuple: (user, token) or (None, None) if user not found
"""
user = User.query.filter_by(email=email).first()
if not user:
return None, None
token = user.get_reset_password_token()
return user, token
def reset_password(token, new_password):
"""
Reset a user's password using a token.
Args:
token: Password reset token
new_password: New password
Returns:
bool: True if successful, False otherwise
"""
user = User.verify_reset_password_token(token)
if not user:
return False
user.set_password(new_password)
db.session.commit()
return True
app/routes/auth.py - Authentication routes:
"""Authentication routes."""
from flask import Blueprint, render_template, redirect, url_for, flash, request, current_app
from flask_login import login_user, logout_user, current_user, login_required
from app.models.user import User
from app.services.auth_service import (
register_user, login_user_service, generate_password_reset_token,
reset_password as service_reset_password
)
from app.services.email_service import send_email
from app.forms.auth import LoginForm, RegistrationForm, ResetPasswordRequestForm, ResetPasswordForm
import jwt
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login', methods=['GET', 'POST'])
def login():
"""User login view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = LoginForm()
if form.validate_on_submit():
user = User.query.filter_by(username=form.username.data).first()
if user is None or not user.check_password(form.password.data):
flash('Invalid username or password')
current_app.logger.info(f"Failed login attempt for username: {form.username.data}")
return redirect(url_for('auth.login'))
login_user(user, remember=form.remember_me.data)
login_user_service(user)
current_app.logger.info(f"User logged in: {user.username}")
next_page = request.args.get('next')
if not next_page or not next_page.startswith('/'):
next_page = url_for('main.index')
return redirect(next_page)
return render_template('auth/login.html', title='Sign In', form=form)
@auth_bp.route('/logout')
def logout():
"""User logout view."""
logout_user()
flash('You have been logged out')
return redirect(url_for('main.index'))
@auth_bp.route('/register', methods=['GET', 'POST'])
def register():
"""User registration view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = RegistrationForm()
if form.validate_on_submit():
try:
user = register_user(
username=form.username.data,
email=form.email.data,
password=form.password.data,
first_name=form.first_name.data,
last_name=form.last_name.data
)
current_app.logger.info(f"New user registered: {user.username}")
flash('Congratulations, you are now a registered user! Please check your email for confirmation.')
return redirect(url_for('auth.login'))
except ValueError as e:
flash(str(e))
return redirect(url_for('auth.register'))
return render_template('auth/register.html', title='Register', form=form)
@auth_bp.route('/reset_password_request', methods=['GET', 'POST'])
def reset_password_request():
"""Request password reset view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = ResetPasswordRequestForm()
if form.validate_on_submit():
user, token = generate_password_reset_token(form.email.data)
if user:
send_password_reset_email(user, token)
flash('Check your email for instructions to reset your password')
else:
flash('Check your email for instructions to reset your password')
return redirect(url_for('auth.login'))
return render_template('auth/reset_password_request.html', title='Reset Password', form=form)
@auth_bp.route('/reset_password/', methods=['GET', 'POST'])
def reset_password(token):
"""Reset password view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = ResetPasswordForm()
if form.validate_on_submit():
if service_reset_password(token, form.password.data):
flash('Your password has been reset')
return redirect(url_for('auth.login'))
else:
flash('Invalid or expired token')
return redirect(url_for('auth.reset_password_request'))
return render_template('auth/reset_password.html', title='Reset Password', form=form)
def send_password_reset_email(user, token):
"""Send password reset email."""
reset_url = url_for('auth.reset_password', token=token, _external=True)
subject = "Reset Your Password"
text_body = f"""
Dear {user.first_name or user.username},
To reset your password, please visit the following link:
{reset_url}
If you did not request a password reset, please ignore this email.
Sincerely,
The Flask E-commerce Team
"""
html_body = f"""
Dear {user.first_name or user.username},
To reset your password, please click here.
If you did not request a password reset, please ignore this email.
Sincerely,
The Flask E-commerce Team
"""
send_email(user.email, subject, text_body, html_body)
wsgi.py - WSGI entry point:
"""WSGI entry point for the application."""
from app import create_app
app = create_app('production')
if __name__ == '__main__':
app.run()
manage.py - Command-line interface:
#!/usr/bin/env python
"""Management script for the application."""
import os
import click
from flask.cli import FlaskGroup
from app import create_app, db
from app.models.user import User
from app.models.product import Product, Category
from app.models.order import Order, OrderItem
import random
from datetime import datetime, timedelta
from faker import Faker
app = create_app(os.getenv('FLASK_ENV', 'development'))
cli = FlaskGroup(app)
fake = Faker()
@cli.command('create_admin')
@click.argument('username')
@click.argument('email')
@click.password_option()
def create_admin(username, email, password):
"""Create an admin user."""
user = User(username=username, email=email, is_admin=True)
user.set_password(password)
db.session.add(user)
db.session.commit()
click.echo(f'Admin user {username} created.')
@cli.command('reset_db')
@click.confirmation_option(prompt='Are you sure you want to reset the database?')
def reset_db():
"""Reset the database."""
db.drop_all()
db.create_all()
click.echo('Database has been reset.')
@cli.command('seed_db')
@click.option('--users', default=10, help='Number of users to create')
@click.option('--products', default=50, help='Number of products to create')
@click.option('--categories', default=5, help='Number of categories to create')
@click.option('--orders', default=20, help='Number of orders to create')
def seed_db(users, products, categories, orders):
"""Seed the database with sample data."""
# Create users
click.echo('Creating users...')
for i in range(users):
user = User(
username=fake.user_name(),
email=fake.email(),
first_name=fake.first_name(),
last_name=fake.last_name(),
is_active=True,
created_at=fake.date_time_this_year()
)
user.set_password('password')
db.session.add(user)
# Create admin user
admin = User(
username='admin',
email='admin@example.com',
first_name='Admin',
last_name='User',
is_active=True,
is_admin=True,
created_at=fake.date_time_this_year()
)
admin.set_password('adminpass')
db.session.add(admin)
db.session.commit()
# Create categories
click.echo('Creating categories...')
category_names = ['Electronics', 'Clothing', 'Books', 'Home & Kitchen', 'Toys', 'Sports', 'Beauty', 'Automotive']
created_categories = []
for i in range(min(categories, len(category_names))):
category = Category(
name=category_names[i],
description=fake.paragraph()
)
db.session.add(category)
created_categories.append(category)
db.session.commit()
# Create products
click.echo('Creating products...')
for i in range(products):
product = Product(
name=fake.catch_phrase(),
description=fake.paragraphs(nb=3),
price=round(random.uniform(10, 500), 2),
stock=random.randint(0, 100),
category=random.choice(created_categories),
created_at=fake.date_time_this_year()
)
db.session.add(product)
db.session.commit()
# Create orders
click.echo('Creating orders...')
users_list = User.query.all()
products_list = Product.query.all()
for i in range(orders):
user = random.choice(users_list)
order_date = fake.date_time_this_year()
order = Order(
customer=user,
created_at=order_date,
status=random.choice(['pending', 'paid', 'shipped', 'delivered', 'cancelled']),
shipping_address=fake.address(),
total_amount=0
)
db.session.add(order)
db.session.flush() # To get the order ID
# Add 1-5 items to the order
total_amount = 0
for j in range(random.randint(1, 5)):
product = random.choice(products_list)
quantity = random.randint(1, 3)
item_price = product.price
item_total = quantity * item_price
order_item = OrderItem(
order=order,
product=product,
quantity=quantity,
unit_price=item_price
)
db.session.add(order_item)
total_amount += item_total
order.total_amount = total_amount
db.session.commit()
click.echo('Database has been seeded!')
if __name__ == '__main__':
cli()
The large-scale structure provides significant advantages for complex applications:
- Highly modular with clear separation of concerns
- Each component (models, routes, services) has a specific responsibility
- The application factory pattern enables testing and different deployment configurations
- Business logic is isolated in service modules, making it reusable and testable
- Multiple entry points (web interface, API, command-line) share the same core code
- Advanced error handling and logging capabilities
- Support for database migrations and data seeding
When to use this structure:
- For web applications with multiple features and user types
- When developing systems that will be deployed across different environments
- For projects with complex business logic
- When multiple developers or teams are working on the same codebase
- For applications that will evolve and grow over time
As your application continues to grow, you might find that even this structure becomes challenging to manage. At that point, consider domain-driven design or microservices architectures.
Advanced Structure: Domain-Driven Design
For very large and complex applications, especially those with multiple teams or complex business domains, Domain-Driven Design (DDD) offers a powerful organizing principle. DDD is like organizing a conglomerate with multiple businesses—each domain operates somewhat independently but shares corporate resources and standards.
In DDD, the codebase is organized around business domains rather than technical layers:
ecommerce_platform/
├── src/
│ ├── users/ # User domain
│ │ ├── __init__.py
│ │ ├── models.py # User domain models
│ │ ├── repositories.py # User data access
│ │ ├── services.py # User business logic
│ │ ├── routes.py # User API endpoints
│ │ └── exceptions.py # User domain exceptions
│ ├── products/ # Product domain
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── repositories.py
│ │ ├── services.py
│ │ ├── routes.py
│ │ └── exceptions.py
│ ├── orders/ # Order domain
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── repositories.py
│ │ ├── services.py
│ │ ├── routes.py
│ │ └── exceptions.py
│ ├── payments/ # Payment domain
│ │ ├── __init__.py
│ │ ├── models.py
│ │ ├── repositories.py
│ │ ├── services.py
│ │ ├── routes.py
│ │ └── gateways/ # External payment provider integrations
│ │ ├── __init__.py
│ │ ├── stripe.py
│ │ └── paypal.py
│ └── shared/ # Shared components
│ ├── __init__.py
│ ├── database.py # Database connection
│ ├── security.py # Security utilities
│ ├── events.py # Event bus
│ └── utils.py # Common utilities
├── app/ # Application assembly
│ ├── __init__.py
│ ├── config.py # Configuration
│ ├── api.py # API initialization
│ ├── web.py # Web interface initialization
│ └── cli.py # CLI commands
├── web/ # Web interface
│ ├── __init__.py
│ ├── templates/
│ └── static/
├── tests/ # Tests by domain
│ ├── users/
│ ├── products/
│ ├── orders/
│ └── payments/
├── infrastructure/ # Infrastructure concerns
│ ├── database/ # Database migration scripts
│ ├── logging/ # Logging configuration
│ ├── messaging/ # Message queue setup
│ └── monitoring/ # Application monitoring
└── scripts/ # Operational scripts
├── deploy.sh
├── backup.sh
└── monitor.sh
Key features of this structure:
- Organization by business domain rather than technical function
- Each domain module contains all related code regardless of type
- Shared module for cross-domain functionality
- Clear boundaries between domains
- Infrastructure concerns separated from business logic
DDD Concepts and Benefits
Domain-Driven Design is more than just a folder structure—it's a methodology for tackling complex domains:
Ubiquitous Language
DDD emphasizes creating a shared language between developers and domain experts. This language is reflected in the code, making it more intuitive and aligned with business concepts.
Example: Instead of generic terms like "User" and "Item", the code uses domain-specific terms like "Customer", "Product", and "ShippingAddress".
Bounded Contexts
A bounded context defines the boundaries within which a particular model or concept has a specific meaning. Different contexts might have different interpretations of the same term.
Example: In the "Orders" context, a "Product" might just have an ID, name, and price, while in the "Inventory" context, a "Product" might include detailed specifications, supplier information, and stock levels.
Aggregates and Entities
Aggregates are clusters of domain objects that are treated as a single unit, with one entity serving as the aggregate root. This helps maintain consistency boundaries.
Example: An "Order" aggregate might include the order entity itself plus associated line items, with rules that items can only be added or removed through the order.
Domain Events
Domain events capture significant occurrences within the domain. They enable loose coupling between components and support event-driven architectures.
Example: When an order is placed, an "OrderPlaced" event is published, which can trigger actions in other domains such as inventory updates, payment processing, and notification sending.
When to Use Domain-Driven Design
DDD is particularly valuable in these scenarios:
- Complex business domains with rich processes and rules
- Large applications where different components evolve at different rates
- Projects with multiple teams working on different aspects
- Systems where business logic is the core value proposition
- Applications with long expected lifespans that need to adapt to changing requirements
Real-World Applications: DDD is commonly used in enterprise systems like:
- E-commerce platforms
- Financial systems
- Healthcare applications
- Supply chain management
- Insurance systems
Implementation Considerations:
- DDD introduces higher initial complexity that pays off for complex domains
- It requires close collaboration with domain experts
- It may be overkill for simpler applications
- The structure should evolve as understanding of the domain deepens
- Teams need to invest in shared understanding of DDD principles
Domain-Driven Design provides a framework for managing complexity in large systems by aligning the code structure with business domains. This approach leads to more maintainable, evolvable applications that better serve business needs.
Choosing the Right Structure for Your Project
Selecting the appropriate project structure is a critical decision that will impact your project's maintainability, scalability, and developer experience. Use these guidelines to make an informed choice:
Project Size and Complexity Assessment
The scale of your project is a primary factor in structure selection:
| Size | Characteristics | Recommended Structure |
|---|---|---|
| Small |
|
Simple script structure |
| Medium |
|
Package-based structure |
| Large |
|
Application structure or DDD |
Project Type Considerations
Different types of projects have different structural needs:
| Type | Recommended Structure | Rationale |
|---|---|---|
| Command-line tool | Simple or medium structure with scripts/ directory | Command-line tools typically have simpler interfaces and focused functionality |
| Library/package | Medium structure with clear API boundaries | Libraries need well-defined interfaces and comprehensive tests |
| Web application | Large application structure with MVC-like organization | Web apps have multiple layers (presentation, business logic, data) that benefit from separation |
| Microservice | Domain-focused structure with clear boundaries | Microservices should be organized around business capabilities |
| Enterprise application | Domain-driven design | Complex business rules and multiple teams need domain-centered organization |
Team Considerations
Your team's characteristics also influence structure choice:
- Team Size: Larger teams need more structure to coordinate effectively. Small teams can work with simpler structures.
- Team Experience: Less experienced teams may benefit from more prescriptive structures with clear conventions. Experienced teams might prefer more flexibility.
- Team Organization: If teams are organized by feature or domain, a DDD approach may align better with your organizational structure.
- Distributed Teams: Geographically distributed teams benefit from clearer boundaries and interfaces between components.
Future Growth Expectations
Consider how your project might evolve:
- Short-term project: Simpler structure may be sufficient
- Long-term product: More sophisticated structure will pay off
- Expected feature growth: Choose a structure that accommodates expansion
- Team growth: Plan for onboarding new developers with clear organization
Evolution Strategy
Remember that project structure can evolve over time:
- Start with a structure suitable for your current needs
- Plan for incremental refactoring as the project grows
- Watch for signs that your structure is becoming insufficient:
- Files becoming too large
- Difficulty finding specific code
- Increasing merge conflicts
- New features requiring changes across many files
- Refactor toward a more sophisticated structure when needed
Tip: For new projects, choose a structure that's slightly more sophisticated than what you think you need right now. This provides room to grow without requiring immediate restructuring. It's easier to start with a bit more structure than to add it later.
A Practical Decision Framework
When choosing a structure, ask yourself these questions:
- Purpose: What is the primary purpose of this project? (Library, application, utility)
- Lifespan: How long will this project be maintained? (Days, months, years)
- Team: Who will work on this project? (Solo, small team, multiple teams)
- Complexity: How complex is the business domain? (Simple, moderate, complex)
- Growth: How much will this project grow? (Fixed scope, moderate growth, significant expansion)
The answers to these questions will guide you toward the most appropriate structure for your specific context.
Universal Project Structure Best Practices
Regardless of the specific structure you choose, these best practices will help keep your project organized and maintainable:
Follow the Single Responsibility Principle
Each module, class, or function should have one clear purpose. This principle applies at all levels of your project, from the highest-level packages down to individual functions.
Think of each component as a specialist rather than a generalist. Just as you wouldn't expect a cardiologist to also be an orthopedic surgeon, each module in your code should focus on doing one thing well.
Good: Separate auth_service.py, email_service.py, and payment_service.py
Bad: A single services.py with functions for authentication, email sending, and payment processing
Keep Related Files Together
Files that change together should be located near each other in the directory structure. This reduces the mental overhead of navigation and minimizes merge conflicts.
Think of your codebase like a kitchen—ingredients for a specific dish should be stored together for easy access. Similarly, all the code related to a specific feature or domain should be readily accessible without jumping around the codebase.
Good: Organizing user-related code (model, service, routes) in a users/ package
Bad: Spreading user-related code across many different directories based on technical type
Create Clear API Boundaries
Make it obvious what parts of your code are public interfaces vs. internal implementation details. This helps other developers use your code correctly and allows you to change implementations without breaking dependent code.
Think of this like the dashboard of a car—it presents a clean interface hiding the complex engine underneath. The driver doesn't need to understand the internal combustion process to operate the vehicle.
Good: Using __init__.py to expose only the public interface, using underscore prefixes for internal functions
Bad: No clear distinction between public API and private implementation details
Separate Configuration from Code
Configuration settings should be separated from application logic. This enables deploying the same code to different environments (development, testing, production) with environment-specific settings.
This is like having adjustable settings on a machine—the core mechanics stay the same, but the behavior can be modified through external controls without changing the internal design.
Good: Using environment variables, config files, or a dedicated config.py module
Bad: Hardcoding configuration values throughout the codebase
Keep the Root Directory Clean
The project root should contain only high-level files and directories. Implementation details should be pushed into subdirectories.
Think of your project root as the front page of a newspaper—it should provide an overview of what's important without overwhelming detail. The details come in the inner pages (subdirectories).
Good: Moving implementation details into subdirectories, keeping only essential files at the root
Bad: Dozens of Python files in the root directory
Use Common Conventions
Follow established naming and organization conventions to make your project more approachable to newcomers and maintain consistency.
This is like standardized road signs—once you learn them, you can navigate any highway system. Similarly, using standard conventions helps developers familiar with Python navigate your codebase.
Good: Using standard names like tests/, docs/, and README.md
Bad: Inventing unique naming schemes that deviate from community standards
Include Essential Project Files
Every project should include certain standard files that help users and developers understand and use your code.
Essential files:
README.md- Project overview, installation instructions, basic usage examplesLICENSE- The project's license termsrequirements.txtor equivalent - Project dependencies.gitignore- Files for Git to ignore- Setup or build script (
setup.py,pyproject.toml, etc.)
Document Your Structure
Make sure new team members can understand your project organization. Documentation is like a map that guides people through unfamiliar territory.
Good: Including a "Project Structure" section in your README or documentation that explains the organization and rationale
Bad: Assuming others will intuitively understand your organization without explanation
Test Directory Structure Should Mirror Source
Organize tests to reflect the structure of the code they test. This makes it easier to find tests for specific functionality and maintain test coverage as the codebase evolves.
Think of tests as shadows of your code—they should follow the same shape and organization.
Good: If you have app/models/user.py, create tests/test_models/test_user.py
Bad: Test organization that doesn't correspond to the source structure
Use Consistent Import Style
Establish and follow consistent rules for imports within your project. This reduces confusion and makes the codebase more predictable.
Good: Using absolute imports from the project root for clarity and consistency
Bad: Mixing absolute and relative imports inconsistently throughout the project
Plan for Error Handling
Design your project structure to facilitate consistent, comprehensive error handling. This includes defining custom exceptions and establishing error reporting mechanisms.
Good: Creating an exceptions.py module in each package with domain-specific error classes
Bad: Inconsistent error handling approaches across different modules
Balance Flexibility and Convention
Create enough structure to guide development without being overly restrictive. Your structure should support the workflow, not impede it.
Think of project structure as a garden trellis—it provides support and guidance for growth without constraining the plant's natural development.
Good: Establishing conventions while allowing for exceptions when they make sense
Bad: Extremely rigid rules that require workarounds for legitimate use cases
By following these universal best practices, you can create a project structure that enhances productivity, encourages collaboration, and remains maintainable as your project grows and evolves.
Common Structure Patterns in Python Projects
Several structural patterns appear across many Python projects. Understanding these can help you recognize and apply common organizational strategies:
Application Factory Pattern
Creating the application object via a factory function rather than globally. This pattern is especially common in Flask applications but can be applied to other frameworks as well.
Think of this like a car factory that can produce different models of cars based on specifications. Similarly, the application factory can create different instances of your application with varying configurations.
# app/__init__.py
def create_app(config_name='development'):
app = Flask(__name__)
# Configure the app based on config_name
app.config.from_object(config[config_name])
# Initialize extensions
db.init_app(app)
# Register components
register_blueprints(app)
register_error_handlers(app)
return app
Benefits:
- Enables different configurations for different environments
- Simplifies testing by allowing test-specific configurations
- Avoids circular import issues common in large applications
- Provides a clear initialization sequence
Blueprint Pattern
Organizing routes into modular components, often by feature area. Common in Flask and similar frameworks, this pattern helps manage routes in larger applications.
This is like organizing a large department store into sections—each blueprint manages a distinct area of functionality, making the whole system more navigable.
# app/routes/auth.py
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login')
def login():
# Login logic
@auth_bp.route('/logout')
def logout():
# Logout logic
# app/__init__.py
from app.routes.auth import auth_bp
app.register_blueprint(auth_bp, url_prefix='/auth')
Benefits:
- Modular organization of routes by feature or domain
- Separate URL prefixes for different areas
- Enables parallel development by different team members
- Routes can be conditionally registered based on configuration
Repository Pattern
Abstracting data access behind repository interfaces. This pattern separates business logic from data storage details, making the application more flexible and testable.
Like a library reference desk that handles book retrieval for you—you don't need to know where the books are physically stored, just what you're looking for.
# app/repositories/user_repository.py
class UserRepository:
def get_by_id(self, user_id):
# Query the database
return User.query.get(user_id)
def get_by_email(self, email):
# Query the database
return User.query.filter_by(email=email).first()
def create(self, user_data):
# Create a new user
user = User(**user_data)
db.session.add(user)
db.session.commit()
return user
def update(self, user_id, user_data):
# Update an existing user
user = self.get_by_id(user_id)
if not user:
return None
for key, value in user_data.items():
setattr(user, key, value)
db.session.commit()
return user
def delete(self, user_id):
# Delete a user
user = self.get_by_id(user_id)
if user:
db.session.delete(user)
db.session.commit()
return True
return False
Benefits:
- Abstracts data access details from business logic
- Makes testing easier through mocking
- Allows changing the data storage mechanism without affecting other code
- Centralizes data access logic
- Provides a consistent API for data operations
Service Layer Pattern
Encapsulating business logic in service modules separate from routes and models. This pattern creates a clear separation between your application's interface and its core functionality.
Similar to how a restaurant separates the dining room (interface) from the kitchen (business logic)—the waitstaff takes orders and delivers food without needing to know how to cook.
# app/services/auth_service.py
def authenticate_user(username, password):
"""Authenticate a user with username and password."""
user = user_repository.get_by_username(username)
if user and user.check_password(password):
# Update last login time
user.last_login = datetime.utcnow()
user_repository.update(user)
# Log successful login
logger.info(f"User {username} logged in successfully")
return user
# Log failed attempt
logger.warning(f"Failed login attempt for username: {username}")
return None
def register_user(username, email, password, **kwargs):
"""Register a new user."""
# Validate input
if user_repository.get_by_username(username):
raise ValueError(f"Username '{username}' is already taken")
if user_repository.get_by_email(email):
raise ValueError(f"Email '{email}' is already registered")
# Create user object
user_data = {
'username': username,
'email': email,
**kwargs
}
user = User(**user_data)
user.set_password(password)
# Save to database
user_repository.create(user)
# Send welcome email asynchronously
email_service.send_welcome_email(user)
# Log registration
logger.info(f"New user registered: {username}")
return user
Benefits:
- Separates business logic from HTTP/presentation concerns
- Makes business logic more testable and reusable
- Can be accessed from different entry points (web, API, CLI)
- Centralizes validation and business rules
- Promotes cleaner, more focused components
Command Pattern
Encapsulating operations as command objects. This pattern is common in CLI-heavy applications and task queues, where operations need to be serialized, queued, or logged.
Like writing down instructions for someone to follow later—commands encapsulate all the information needed to perform an action at a later time.
# app/commands/user_commands.py
class CreateUserCommand:
def __init__(self, username, email, password, **kwargs):
self.username = username
self.email = email
self.password = password
self.additional_data = kwargs
def execute(self):
# Check if user already exists
if User.query.filter_by(username=self.username).first():
raise ValueError(f"Username '{self.username}' is already taken")
if User.query.filter_by(email=self.email).first():
raise ValueError(f"Email '{self.email}' is already registered")
# Create and save the user
user = User(
username=self.username,
email=self.email,
**self.additional_data
)
user.set_password(self.password)
db.session.add(user)
db.session.commit()
return user
Benefits:
- Encapsulates operations in a clean, self-contained way
- Enables features like command queuing, logging, and undo operations
- Simplifies complex operation sequences
- Can be stored for later execution (asynchronous processing)
- Supports operation history and audit trails
Settings Module Pattern
Centralizing configuration in a dedicated module, often with environment-specific subclasses. This pattern provides a clean way to manage different configurations for different environments.
Like having different presets on a complex machine—each preset configures the machine for a specific use case.
# app/config.py
class Config:
"""Base configuration applicable to all environments."""
DEBUG = False
TESTING = False
SECRET_KEY = os.environ.get('SECRET_KEY', 'dev-key')
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Common settings
PAGINATION_PER_PAGE = 25
UPLOAD_FOLDER = os.path.join(basedir, 'uploads')
class DevelopmentConfig(Config):
"""Development environment configuration."""
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///dev.db'
SQLALCHEMY_ECHO = True
class TestingConfig(Config):
"""Testing environment configuration."""
TESTING = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
WTF_CSRF_ENABLED = False
class ProductionConfig(Config):
"""Production environment configuration."""
SQLALCHEMY_DATABASE_URI = os.environ.get(
'DATABASE_URL',
'postgresql://user:pass@localhost/dbname'
)
# Production-specific settings
SERVER_NAME = os.environ.get('SERVER_NAME')
# Configuration dictionary
config = {
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig,
'default': DevelopmentConfig
}
Benefits:
- Centralizes all configuration options
- Supports different environments with inheritance
- Makes configuration explicit and documented
- Separates sensitive values from code (via environment variables)
- Simplifies testing and deployment
Event-Driven Architecture Pattern
Using events to communicate between components. This pattern reduces coupling and enables more flexible, extensible systems.
Like a newspaper publishing news—the newspaper doesn't need to know who's reading; subscribers decide what to do with the information.
# app/events/event_bus.py
class EventBus:
def __init__(self):
self.subscribers = {}
def subscribe(self, event_type, callback):
if event_type not in self.subscribers:
self.subscribers[event_type] = []
self.subscribers[event_type].append(callback)
def publish(self, event):
event_type = event.__class__.__name__
if event_type in self.subscribers:
for callback in self.subscribers[event_type]:
callback(event)
# Creating the global event bus
event_bus = EventBus()
# app/events/user_events.py
class UserCreated:
def __init__(self, user_id, username, email):
self.user_id = user_id
self.username = username
self.email = email
self.timestamp = datetime.utcnow()
# app/services/user_service.py
def create_user(username, email, password):
# Create the user
user = User(username=username, email=email)
user.set_password(password)
db.session.add(user)
db.session.commit()
# Publish event
event = UserCreated(user.id, user.username, user.email)
event_bus.publish(event)
return user
# app/listeners/email_listener.py
def send_welcome_email(event):
"""Send welcome email when a user is created."""
user_id = event.user_id
email = event.email
# Generate email content
subject = "Welcome to Our Application!"
body = f"Hi {event.username},\n\nWelcome to our application!"
# Send the email
email_service.send_email(email, subject, body)
# Register the listener
event_bus.subscribe(UserCreated, send_welcome_email)
Benefits:
- Reduces coupling between components
- Enables adding new behaviors without modifying existing code
- Supports asynchronous processing
- Facilitates audit logging and event sourcing
- Creates more extensible, maintainable systems
These patterns are not mutually exclusive—most mature Python applications combine several of these patterns to create a cohesive, maintainable architecture. By understanding these common patterns, you can leverage tried-and-tested approaches rather than reinventing the wheel.
Tools for Project Structure Management
Several tools can help you create and maintain good project structures. These tools automate common tasks, enforce conventions, and provide scaffolding for new projects:
Project Templates and Generators
These tools provide pre-defined project structures and can generate boilerplate code, saving time and ensuring consistency:
- cookiecutter: A versatile project template engine that creates projects from templates
# Install cookiecutter pip install cookiecutter # Generate a Python package project cookiecutter https://github.com/audreyfeldroy/cookiecutter-pypackage # Generate a Flask application cookiecutter https://github.com/cookiecutter-flask/cookiecutter-flaskCookiecutter prompts you for project-specific information and then creates a complete directory structure with appropriate files.
- Framework CLI tools: Most major frameworks have their own tools to generate project structures
# Django project creation django-admin startproject myproject django-admin startapp myapp # Flask application with Flask-CLI pip install flask-cli flask create-app --name myapp # FastAPI project with FastAPI CLI pip install fastapi-cli fastapi startproject myprojectThese tools create structures optimized for their respective frameworks.
Code Quality and Structure Enforcement Tools
These tools help maintain code quality and structural consistency in your projects:
- pylint: A comprehensive linter that checks code quality and enforces structure conventions
pip install pylint pylint mypackagePylint can detect issues like improper imports, inconsistent naming, and module organization problems.
- flake8: A code linter that combines several tools (PyFlakes, pycodestyle, etc.)
pip install flake8 flake8 mypackageFlake8 focuses on style issues and common errors but is less opinionated than pylint.
- black: An uncompromising code formatter that ensures consistent style
pip install black black mypackageBlack automatically formats your code to a consistent style, eliminating debates about formatting.
- isort: A utility to sort and organize imports
pip install isort isort mypackageIsort groups and sorts imports according to conventions, improving readability and consistency.
- pre-commit: A framework for managing git pre-commit hooks
pip install pre-commit pre-commit installPre-commit can run linters, formatters, and other tools before each commit, ensuring code quality.
Documentation Tools
Documenting your project structure helps others understand and navigate it:
- Sphinx: Generate comprehensive documentation from docstrings and separate documentation files
pip install sphinx sphinx-quickstartSphinx can create HTML, PDF, and other formats from your documentation, including API references.
- mkdocs: A simpler, Markdown-based documentation generator
pip install mkdocs mkdocs new myprojectMkDocs creates clean, searchable documentation sites from Markdown files.
- pdoc: Automatic API documentation for Python projects
pip install pdoc pdoc --html mypackagePdoc generates API documentation directly from your Python modules, with minimal configuration required.
Project Structure Analysis
These tools help visualize and analyze your project structure:
- pyan3: Analyze and visualize dependencies between Python objects
pip install pyan3 pyan3 mypackage/*.py --dot > deps.dot dot -Tpng deps.dot -o deps.pngPyan creates visual graphs of dependencies, helping you understand and refactor your codebase.
- pyreverse: Generate UML diagrams from Python code (part of pylint)
pip install pylint pyreverse -o png mypackagePyreverse creates class and package diagrams that visualize your project's structure.
- pipdeptree: Visualize pip dependency trees
pip install pipdeptree pipdeptreePipdeptree shows the relationships between installed packages, helping manage dependencies.
Virtual Environment and Dependency Management
These tools help manage the environment in which your project runs:
- venv/virtualenv: Create isolated Python environments
# Using venv (built into Python 3) python -m venv myenv # Using virtualenv pip install virtualenv virtualenv myenvVirtual environments isolate dependencies for different projects, preventing conflicts.
- pipenv: Combines pip and virtualenv with dependency locking
pip install pipenv pipenv install requestsPipenv creates a Pipfile and Pipfile.lock for deterministic builds.
- poetry: Modern Python packaging and dependency management
pip install poetry poetry new myproject poetry add requestsPoetry handles dependencies, building, and publishing in a more integrated way.
Building and Packaging Tools
These tools help prepare your project for distribution:
- setuptools: Standard tool for packaging Python projects
python setup.py sdist bdist_wheelSetuptools builds source and binary distributions of your package.
- build: PEP 517 compatible build frontend
pip install build python -m buildBuild creates standardized packages using the new Python packaging standards.
- twine: Utility for publishing packages to PyPI
pip install twine twine upload dist/*Twine securely uploads your packages to the Python Package Index.
Using these tools effectively can significantly improve your project structure management workflow. They automate repetitive tasks, enforce best practices, and help you create more maintainable, professional Python projects.
Learning from Real-World Examples
Examining popular open-source projects can provide valuable insights into effective project structures. Let's analyze some notable examples:
Flask (Web Framework)
Flask uses a simple, package-based structure that's easy to understand:
flask/
├── docs/ # Comprehensive documentation
├── examples/ # Example applications
├── src/
│ └── flask/ # Main package
│ ├── __init__.py # Public API
│ ├── app.py # Application object
│ ├── blueprints.py # Blueprint support
│ ├── cli.py # Command-line interface
│ ├── config.py # Configuration handling
│ ├── ctx.py # Context locals
│ ├── globals.py # Global objects
│ ├── sessions.py # Session handling
│ ├── templating.py # Template support
│ └── ... # Other modules
├── tests/ # Test suite
├── setup.py # Package setup
└── pyproject.toml # Project metadata
Key Insights:
- Flask uses a flat module structure under the main package, with each module having a clear responsibility.
- The
__init__.pyfile carefully exposes the public API, controlling what users can import directly. - Tests are kept separate from the source code but mirror its organization.
- Examples help users understand how to use the framework in different contexts.
- Documentation is comprehensive and treated as a first-class citizen of the project.
Takeaway: Even for smaller projects, a clean separation of concerns and well-defined API boundaries can create maintainable, user-friendly packages.
Django (Web Framework)
Django uses a more complex structure reflecting its larger scope:
django/
├── django/ # Main package
│ ├── __init__.py
│ ├── apps/ # Application registry
│ ├── conf/ # Settings and configuration
│ ├── contrib/ # Bundled apps (admin, auth, etc.)
│ ├── core/ # Core functionality
│ ├── db/ # Database layer
│ │ ├── __init__.py
│ │ ├── backends/ # Database engine backends
│ │ ├── migrations/ # Migration system
│ │ ├── models/ # ORM components
│ │ └── ...
│ ├── http/ # HTTP handling
│ ├── template/ # Template system
│ ├── urls/ # URL routing
│ ├── utils/ # Utilities
│ └── views/ # View functions
├── docs/ # Documentation
├── tests/ # Test suite
├── scripts/ # Utility scripts
└── setup.py # Package setup
Key Insights:
- Django uses a hierarchical package structure, with subpackages for different areas of functionality.
- The
contribpackage contains optional, self-contained applications that extend Django's functionality. - The
utilspackage provides shared utilities used across different parts of the framework. - Each major subsystem (db, http, template) is a separate package with clear boundaries.
- Tests are comprehensive and organized in a way that reflects the package structure.
Takeaway: For large, feature-rich applications, a hierarchical structure with clear component boundaries helps manage complexity and allows different parts to evolve independently.
Requests (HTTP Library)
Requests uses a simpler structure appropriate for its focused purpose:
requests/
├── requests/ # Main package
│ ├── __init__.py # Public API
│ ├── api.py # API functionality
│ ├── auth.py # Authentication handlers
│ ├── cookies.py # Cookie handling
│ ├── exceptions.py # Custom exceptions
│ ├── models.py # Data models
│ ├── sessions.py # Session handling
│ ├── structures.py # Data structures
│ └── utils.py # Utilities
├── docs/ # Documentation
├── tests/ # Test suite
└── setup.py # Package setup
Key Insights:
- Requests uses a flat module structure, reflecting its focused purpose and cohesive functionality.
- The
__init__.pyfile exposes a clean, user-friendly API, hiding implementation details. - Each module has a clear, single responsibility (auth, cookies, sessions, etc.).
- Custom exceptions are centralized in
exceptions.py, creating a cohesive error handling system. - The project prioritizes simplicity and a great user experience over complex architecture.
Takeaway: For libraries with a focused purpose, a simple, flat structure with clear module responsibilities can be more maintainable than a complex hierarchy.
SQLAlchemy (ORM Library)
SQLAlchemy uses a more complex structure suitable for its comprehensive nature:
sqlalchemy/
├── lib/
│ └── sqlalchemy/ # Main package
│ ├── __init__.py
│ ├── engine/ # Database engine
│ ├── event/ # Event system
│ ├── ext/ # Extensions
│ ├── orm/ # Object-relational mapping
│ ├── pool/ # Connection pooling
│ ├── sql/ # SQL expression language
│ └── ... # Other modules
├── doc/ # Documentation
├── test/ # Test suite
├── examples/ # Example applications
└── setup.py # Package setup
Key Insights:
- SQLAlchemy uses a hierarchical structure with subpackages for major components.
- The
extpackage contains extensions that are useful but not core to the library's functionality. - The structure reflects the library's layered architecture, with clear separation between the SQL expression language, database engine, and ORM.
- Examples demonstrate different usage patterns and help users understand the library's capabilities.
- The documentation is extensive, reflecting the library's power and flexibility.
Takeaway: For complex libraries with multiple layers of functionality, a hierarchical structure with clear separation between layers helps users understand and use the library effectively.
FastAPI (Web Framework)
FastAPI uses a modern, focused structure:
fastapi/
├── fastapi/ # Main package
│ ├── __init__.py # Version and main imports
│ ├── applications.py # FastAPI application class
│ ├── background.py # Background tasks
│ ├── concurrency.py # Async utilities
│ ├── datastructures.py # Data structures
│ ├── dependencies.py # Dependency injection
│ ├── encoders.py # JSON encoders
│ ├── exception_handlers.py # Exception handling
│ ├── exceptions.py # Custom exceptions
│ ├── middleware.py # Middleware components
│ ├── openapi/ # OpenAPI schema generation
│ ├── params.py # Parameter models
│ ├── responses.py # Response models
│ ├── routing.py # URL routing
│ ├── security/ # Security utilities
│ └── templating.py # Template rendering
├── docs/ # Documentation
├── tests/ # Test suite
└── setup.py # Package setup
Key Insights:
- FastAPI uses a mostly flat structure with a few subpackages for complex subsystems.
- Each module has a clear, focused responsibility, making the codebase easy to navigate.
- The structure reflects modern Python practices, with type hints and clear interfaces.
- The project leverages Pydantic for data validation, showing how external dependencies can influence structure.
- Documentation is treated as a first-class citizen, with extensive examples and explanations.
Takeaway: Modern Python projects can benefit from a clean, focused structure that leverages the ecosystem of powerful libraries to reduce internal complexity.
Common Patterns Across Successful Projects:
- Clear Responsibilities: Each module or package has a well-defined purpose.
- Thoughtful API Design: Public interfaces are carefully designed and exposed.
- Comprehensive Testing: Tests are extensive and organized to mirror the code structure.
- Excellent Documentation: Documentation is treated as an essential part of the project.
- Examples: Real-world examples help users understand how to use the code.
- Separation of Concerns: Different aspects of functionality are cleanly separated.
- Utility Packages: Common utilities are organized in dedicated modules or packages.
By studying these successful open-source projects, you can gain insights into effective organizational patterns and apply them to your own projects, regardless of size or complexity.
Practical Exercise: Refactoring a Project Structure
Let's practice by refactoring a poorly structured project into a well-organized one. This exercise will help solidify the concepts we've discussed and demonstrate the process of improving an existing codebase.
The Starting Point: A Messy Flask Application
Here's our initial project structure—a typical example of a Flask application that has grown organically without much thought to organization:
messy_flask_app/
├── app.py # Contains everything: models, routes, business logic (2,000+ lines)
├── utils.py # Miscellaneous utilities
├── templates/ # HTML templates
│ ├── base.html
│ ├── index.html
│ ├── login.html
│ ├── register.html
│ ├── product_list.html
│ └── product_detail.html
├── static/ # Static assets
│ ├── css/
│ ├── js/
│ └── img/
├── test.py # All tests in one file
└── requirements.txt # Dependencies
The main issue is the monolithic app.py file that contains:
- Flask application setup and configuration
- Database models (User, Product, Order)
- Route handlers for all features
- Business logic for user authentication, product management, etc.
- Form validation
- Helper functions and utilities
- Hardcoded configuration values
This structure makes the application difficult to maintain, test, and extend. Let's refactor it into a well-organized Flask application.
Our Target: A Well-Structured Flask Application
We'll reorganize the code into this cleaner structure:
organized_flask_app/
├── app/ # Application package
│ ├── __init__.py # Application factory
│ ├── config.py # Configuration
│ ├── models/ # Database models
│ │ ├── __init__.py
│ │ ├── user.py
│ │ ├── product.py
│ │ └── order.py
│ ├── routes/ # Route handlers
│ │ ├── __init__.py
│ │ ├── auth.py
│ │ ├── main.py
│ │ └── products.py
│ ├── services/ # Business logic
│ │ ├── __init__.py
│ │ ├── auth_service.py
│ │ └── product_service.py
│ ├── forms/ # Form definitions and validation
│ │ ├── __init__.py
│ │ ├── auth_forms.py
│ │ └── product_forms.py
│ ├── templates/ # HTML templates
│ │ ├── base.html
│ │ ├── index.html
│ │ ├── auth/
│ │ │ ├── login.html
│ │ │ └── register.html
│ │ └── products/
│ │ ├── list.html
│ │ └── detail.html
│ ├── static/ # Static assets
│ │ ├── css/
│ │ ├── js/
│ │ └── img/
│ └── utils/ # Utilities
│ ├── __init__.py
│ └── helpers.py
├── tests/ # Test suite
│ ├── __init__.py
│ ├── conftest.py # Test fixtures
│ ├── test_models/
│ │ ├── test_user.py
│ │ └── test_product.py
│ ├── test_routes/
│ │ ├── test_auth.py
│ │ └── test_products.py
│ └── test_services/
│ ├── test_auth_service.py
│ └── test_product_service.py
├── migrations/ # Database migrations
├── .env.example # Environment variables template
├── .gitignore # Git ignore rules
├── README.md # Project documentation
├── requirements.txt # Dependencies
└── wsgi.py # WSGI entry point
Step-by-Step Refactoring Process
Let's walk through the process of transforming our messy project into this clean, organized structure:
Step 1: Create the Directory Structure
First, we'll create the necessary directories and files:
# Create directories
mkdir -p organized_flask_app/app/{models,routes,services,forms,utils}
mkdir -p organized_flask_app/app/templates/{auth,products}
mkdir -p organized_flask_app/app/static/{css,js,img}
mkdir -p organized_flask_app/tests/{test_models,test_routes,test_services}
# Create necessary __init__.py files
touch organized_flask_app/app/__init__.py
touch organized_flask_app/app/{models,routes,services,forms,utils}/__init__.py
touch organized_flask_app/tests/__init__.py
# Create other essential files
touch organized_flask_app/app/config.py
touch organized_flask_app/.env.example
touch organized_flask_app/.gitignore
touch organized_flask_app/README.md
touch organized_flask_app/wsgi.py
Step 2: Extract Database Models
Next, we'll extract the models from the monolithic app.py file:
app/models/user.py
"""User model module."""
from flask_login import UserMixin
from werkzeug.security import generate_password_hash, check_password_hash
from datetime import datetime
from app import db
class User(db.Model, UserMixin):
"""User model for authentication and profile information."""
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), unique=True, index=True)
email = db.Column(db.String(120), unique=True, index=True)
password_hash = db.Column(db.String(128))
created_at = db.Column(db.DateTime, default=datetime.utcnow)
# Relationships
orders = db.relationship('Order', backref='customer', lazy='dynamic')
def __repr__(self):
return f''
def set_password(self, password):
"""Set the user's password hash."""
self.password_hash = generate_password_hash(password)
def check_password(self, password):
"""Check if the provided password matches the hash."""
return check_password_hash(self.password_hash, password)
app/models/product.py
"""Product model module."""
from datetime import datetime
from app import db
class Product(db.Model):
"""Product model representing items for sale."""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), nullable=False)
description = db.Column(db.Text)
price = db.Column(db.Float, nullable=False)
stock = db.Column(db.Integer, default=0)
created_at = db.Column(db.DateTime, default=datetime.utcnow)
# Relationships
order_items = db.relationship('OrderItem', backref='product')
def __repr__(self):
return f''
app/models/order.py
"""Order model module."""
from datetime import datetime
from app import db
class Order(db.Model):
"""Order model representing customer purchases."""
id = db.Column(db.Integer, primary_key=True)
user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False)
status = db.Column(db.String(20), default='pending')
created_at = db.Column(db.DateTime, default=datetime.utcnow)
# Relationships
items = db.relationship('OrderItem', backref='order', lazy='dynamic')
def __repr__(self):
return f''
@property
def total(self):
"""Calculate the total cost of the order."""
return sum(item.price * item.quantity for item in self.items)
class OrderItem(db.Model):
"""OrderItem model representing individual items in an order."""
id = db.Column(db.Integer, primary_key=True)
order_id = db.Column(db.Integer, db.ForeignKey('order.id'), nullable=False)
product_id = db.Column(db.Integer, db.ForeignKey('product.id'), nullable=False)
quantity = db.Column(db.Integer, default=1)
price = db.Column(db.Float, nullable=False)
def __repr__(self):
return f''
app/models/__init__.py
"""Models package initialization."""
# Import models for easy access
from app.models.user import User
from app.models.product import Product
from app.models.order import Order, OrderItem
Step 3: Extract Route Handlers
Now we'll extract the route handlers into separate modules:
app/routes/main.py
"""Main application routes."""
from flask import Blueprint, render_template, current_app
from app.models.product import Product
from app.services.product_service import get_featured_products
main_bp = Blueprint('main', __name__)
@main_bp.route('/')
def index():
"""Home page view."""
featured_products = get_featured_products(limit=4)
return render_template('index.html', products=featured_products)
@main_bp.route('/about')
def about():
"""About page view."""
return render_template('about.html')
@main_bp.route('/contact')
def contact():
"""Contact page view."""
return render_template('contact.html')
app/routes/auth.py
"""Authentication routes."""
from flask import Blueprint, render_template, redirect, url_for, flash, request
from flask_login import login_user, logout_user, current_user, login_required
from app.forms.auth_forms import LoginForm, RegistrationForm
from app.services.auth_service import register_user, authenticate_user
auth_bp = Blueprint('auth', __name__)
@auth_bp.route('/login', methods=['GET', 'POST'])
def login():
"""User login view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = LoginForm()
if form.validate_on_submit():
user = authenticate_user(form.username.data, form.password.data)
if user:
login_user(user, remember=form.remember_me.data)
next_page = request.args.get('next')
if not next_page or not next_page.startswith('/'):
next_page = url_for('main.index')
return redirect(next_page)
else:
flash('Invalid username or password')
return render_template('auth/login.html', title='Sign In', form=form)
@auth_bp.route('/logout')
def logout():
"""User logout view."""
logout_user()
flash('You have been logged out')
return redirect(url_for('main.index'))
@auth_bp.route('/register', methods=['GET', 'POST'])
def register():
"""User registration view."""
if current_user.is_authenticated:
return redirect(url_for('main.index'))
form = RegistrationForm()
if form.validate_on_submit():
try:
user = register_user(
username=form.username.data,
email=form.email.data,
password=form.password.data
)
flash('Congratulations, you are now a registered user!')
return redirect(url_for('auth.login'))
except ValueError as e:
flash(str(e))
return render_template('auth/register.html', title='Register', form=form)
app/routes/products.py
"""Product management routes."""
from flask import Blueprint, render_template, redirect, url_for, request, flash
from flask_login import login_required, current_user
from app.models.product import Product
from app.forms.product_forms import ProductSearchForm
from app.services.product_service import get_products, get_product_by_id
products_bp = Blueprint('products', __name__)
@products_bp.route('/products', methods=['GET'])
def product_list():
"""Product listing view with optional search."""
form = ProductSearchForm(request.args)
page = request.args.get('page', 1, type=int)
filters = {}
if form.validate():
if form.search.data:
filters['search'] = form.search.data
if form.min_price.data:
filters['min_price'] = form.min_price.data
if form.max_price.data:
filters['max_price'] = form.max_price.data
products, total = get_products(page=page, **filters)
return render_template(
'products/list.html',
title='Products',
products=products,
form=form,
page=page,
total=total
)
@products_bp.route('/products/')
def product_detail(product_id):
"""Product detail view."""
product = get_product_by_id(product_id)
if not product:
flash('Product not found')
return redirect(url_for('products.product_list'))
return render_template(
'products/detail.html',
title=product.name,
product=product
)
app/routes/__init__.py
"""Routes package initialization."""
from app.routes.auth import auth_bp
from app.routes.main import main_bp
from app.routes.products import products_bp
def register_blueprints(app):
"""Register all blueprints with the application."""
app.register_blueprint(main_bp)
app.register_blueprint(auth_bp, url_prefix='/auth')
app.register_blueprint(products_bp)
Step 4: Extract Business Logic into Services
Next, let's move business logic into dedicated service modules:
app/services/auth_service.py
"""Authentication service module."""
from app import db
from app.models.user import User
def register_user(username, email, password):
"""
Register a new user.
Args:
username: User's username
email: User's email address
password: User's password
Returns:
User: The created user object
Raises:
ValueError: If username or email already exists
"""
# Check if username or email already exists
if User.query.filter_by(username=username).first():
raise ValueError(f"Username '{username}' is already taken")
if User.query.filter_by(email=email).first():
raise ValueError(f"Email '{email}' is already registered")
# Create the user
user = User(username=username, email=email)
user.set_password(password)
# Save to database
db.session.add(user)
db.session.commit()
return user
def authenticate_user(username, password):
"""
Authenticate a user with username and password.
Args:
username: User's username
password: User's password
Returns:
User: The user object if authentication is successful, None otherwise
"""
user = User.query.filter_by(username=username).first()
if user and user.check_password(password):
return user
return None
app/services/product_service.py
"""Product service module."""
from app import db
from app.models.product import Product
from sqlalchemy import or_
from flask import current_app
def get_products(page=1, per_page=None, search=None, min_price=None, max_price=None):
"""
Get products with optional filtering.
Args:
page: Page number (default: 1)
per_page: Items per page (default: from config)
search: Search term for name/description
min_price: Minimum price
max_price: Maximum price
Returns:
tuple: (list of products, total count)
"""
per_page = per_page or current_app.config.get('PRODUCTS_PER_PAGE', 10)
# Start with base query
query = Product.query
# Apply filters
if search:
search_term = f"%{search}%"
query = query.filter(
or_(
Product.name.ilike(search_term),
Product.description.ilike(search_term)
)
)
if min_price is not None:
query = query.filter(Product.price >= min_price)
if max_price is not None:
query = query.filter(Product.price <= max_price)
# Get paginated results
pagination = query.order_by(Product.name).paginate(
page=page,
per_page=per_page,
error_out=False
)
return pagination.items, pagination.total
def get_product_by_id(product_id):
"""
Get a product by ID.
Args:
product_id: Product ID
Returns:
Product: The product object or None if not found
"""
return Product.query.get(product_id)
def get_featured_products(limit=4):
"""
Get a list of featured products.
Args:
limit: Maximum number of products to return
Returns:
list: Featured products
"""
# In a real app, you might have a 'featured' flag or use another logic
# For this example, we'll just get the newest products
return Product.query.order_by(Product.created_at.desc()).limit(limit).all()
Step 5: Create Form Classes
Let's extract form handling into dedicated modules:
app/forms/auth_forms.py
"""Authentication form classes."""
from flask_wtf import FlaskForm
from wtforms import StringField, PasswordField, BooleanField, SubmitField
from wtforms.validators import DataRequired, Email, EqualTo, Length, ValidationError
from app.models.user import User
class LoginForm(FlaskForm):
"""User login form."""
username = StringField('Username', validators=[DataRequired()])
password = PasswordField('Password', validators=[DataRequired()])
remember_me = BooleanField('Remember Me')
submit = SubmitField('Sign In')
class RegistrationForm(FlaskForm):
"""User registration form."""
username = StringField('Username', validators=[
DataRequired(),
Length(min=3, max=64)
])
email = StringField('Email', validators=[
DataRequired(),
Email(),
Length(max=120)
])
password = PasswordField('Password', validators=[
DataRequired(),
Length(min=8)
])
password2 = PasswordField('Confirm Password', validators=[
DataRequired(),
EqualTo('password')
])
submit = SubmitField('Register')
def validate_username(self, username):
"""Validate that username is not already taken."""
user = User.query.filter_by(username=username.data).first()
if user:
raise ValidationError('Username already taken.')
def validate_email(self, email):
"""Validate that email is not already registered."""
user = User.query.filter_by(email=email.data).first()
if user:
raise ValidationError('Email already registered.')
app/forms/product_forms.py
"""Product-related form classes."""
from flask_wtf import FlaskForm
from wtforms import StringField, FloatField, SubmitField
from wtforms.validators import Optional, NumberRange
class ProductSearchForm(FlaskForm):
"""Product search and filter form."""
search = StringField('Search')
min_price = FloatField('Min Price', validators=[
Optional(),
NumberRange(min=0)
])
max_price = FloatField('Max Price', validators=[Optional()])
submit = SubmitField('Search')
def validate_max_price(self, max_price):
"""Validate max_price is greater than min_price if both are provided."""
if self.min_price.data and max_price.data:
if max_price.data < self.min_price.data:
raise ValidationError('Max price must be greater than min price.')
Step 6: Create Configuration Module
Let's set up a proper configuration system:
app/config.py
"""Application configuration module."""
import os
from dotenv import load_dotenv
# Load environment variables from .env file
basedir = os.path.abspath(os.path.dirname(__file__))
load_dotenv(os.path.join(os.path.dirname(basedir), '.env'))
class Config:
"""Base configuration class."""
# Flask settings
SECRET_KEY = os.environ.get('SECRET_KEY', 'dev-key-please-change')
# Database settings
SQLALCHEMY_DATABASE_URI = os.environ.get(
'DATABASE_URL',
f"sqlite:///{os.path.join(basedir, 'app.db')}"
)
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Application settings
PRODUCTS_PER_PAGE = 10
class DevelopmentConfig(Config):
"""Development environment configuration."""
DEBUG = True
SQLALCHEMY_ECHO = True
class TestingConfig(Config):
"""Testing environment configuration."""
TESTING = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
WTF_CSRF_ENABLED = False
class ProductionConfig(Config):
"""Production environment configuration."""
DEBUG = False
@classmethod
def init_app(cls, app):
# Log to stderr
import logging
from logging import StreamHandler
file_handler = StreamHandler()
file_handler.setLevel(logging.INFO)
app.logger.addHandler(file_handler)
# Configuration dictionary
config = {
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig,
'default': DevelopmentConfig
}
Step 7: Create Application Factory
Now let's create the application factory in __init__.py:
app/__init__.py
"""Application factory module."""
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_login import LoginManager
import logging
from logging.handlers import RotatingFileHandler
import os
# Initialize extensions
db = SQLAlchemy()
migrate = Migrate()
login_manager = LoginManager()
def create_app(config_name='development'):
"""Create and configure the Flask application."""
app = Flask(__name__)
# Load configuration
from app.config import config
app.config.from_object(config[config_name])
# Initialize extensions
db.init_app(app)
migrate.init_app(app, db)
login_manager.init_app(app)
# Set up login manager
from app.models.user import User
login_manager.login_view = 'auth.login'
@login_manager.user_loader
def load_user(user_id):
return User.query.get(int(user_id))
# Register blueprints
from app.routes import register_blueprints
register_blueprints(app)
# Set up logging
if not app.debug and not app.testing:
if not os.path.exists('logs'):
os.mkdir('logs')
file_handler = RotatingFileHandler(
'logs/app.log',
maxBytes=10240,
backupCount=10
)
file_handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'
))
file_handler.setLevel(logging.INFO)
app.logger.addHandler(file_handler)
app.logger.setLevel(logging.INFO)
app.logger.info('Application startup')
return app
Step 8: Create WSGI Entry Point
Let's create the WSGI entry point for production:
wsgi.py
"""WSGI entry point for the application."""
from app import create_app
app = create_app('production')
if __name__ == '__main__':
app.run()
Step 9: Create Basic Tests
Finally, let's set up some basic tests:
tests/conftest.py
"""Test fixtures and configuration."""
import pytest
from app import create_app, db
from app.models import User, Product
@pytest.fixture
def app():
"""Create and configure a Flask app for testing."""
app = create_app('testing')
# Create tables
with app.app_context():
db.create_all()
yield app
# Clean up
with app.app_context():
db.drop_all()
@pytest.fixture
def client(app):
"""A test client for the app."""
return app.test_client()
@pytest.fixture
def runner(app):
"""A test CLI runner for the app."""
return app.test_cli_runner()
@pytest.fixture
def init_database(app):
"""Initialize the database with test data."""
with app.app_context():
# Create a test user
user = User(username='test_user', email='test@example.com')
user.set_password('password')
db.session.add(user)
# Create some test products
products = [
Product(name='Test Product 1', description='Description 1', price=10.99, stock=10),
Product(name='Test Product 2', description='Description 2', price=20.99, stock=5),
]
db.session.add_all(products)
db.session.commit()
yield
tests/test_models/test_user.py
"""User model tests."""
import pytest
from app.models.user import User
def test_new_user(app):
"""Test user creation."""
with app.app_context():
user = User(username='test_user', email='test@example.com')
user.set_password('password')
assert user.username == 'test_user'
assert user.email == 'test@example.com'
assert user.password_hash is not None
assert user.check_password('password')
assert not user.check_password('wrong_password')
def test_user_representation(app):
"""Test user string representation."""
with app.app_context():
user = User(username='test_user', email='test@example.com')
assert str(user) == ''
tests/test_routes/test_auth.py
"""Authentication route tests."""
import pytest
from flask import url_for
from app.models.user import User
def test_login_page(client):
"""Test login page loads correctly."""
response = client.get('/auth/login')
assert response.status_code == 200
assert b'Sign In' in response.data
def test_register_page(client):
"""Test registration page loads correctly."""
response = client.get('/auth/register')
assert response.status_code == 200
assert b'Register' in response.data
def test_successful_registration(client, app):
"""Test user registration works."""
response = client.post('/auth/register', data={
'username': 'new_user',
'email': 'new@example.com',
'password': 'password123',
'password2': 'password123'
}, follow_redirects=True)
assert response.status_code == 200
assert b'Congratulations, you are now a registered user!' in response.data
# Verify user was created in database
with app.app_context():
user = User.query.filter_by(username='new_user').first()
assert user is not None
assert user.email == 'new@example.com'
def test_successful_login_logout(client, init_database):
"""Test login and logout functionality."""
# Login
response = client.post('/auth/login', data={
'username': 'test_user',
'password': 'password'
}, follow_redirects=True)
assert response.status_code == 200
# Check if user is logged in
response = client.get('/')
assert b'Log Out' in response.data
# Logout
response = client.get('/auth/logout', follow_redirects=True)
assert response.status_code == 200
assert b'You have been logged out' in response.data
# Check if user is logged out
response = client.get('/')
assert b'Sign In' in response.data
Step 10: Add Documentation
Let's create a comprehensive README.md file:
README.md
# Flask E-Commerce Application
A simple e-commerce application built with Flask.
## Features
- User authentication (register, login, logout)
- Product catalog with search and filtering
- Shopping cart functionality
- Order management
- Admin interface
## Project Structure
organized_flask_app/
├── app/ # Application package
│ ├── init.py # Application factory
│ ├── config.py # Configuration
│ ├── models/ # Database models
│ ├── routes/ # Route handlers
│ ├── services/ # Business logic
│ ├── forms/ # Form definitions
│ ├── templates/ # HTML templates
│ ├── static/ # Static assets
│ └── utils/ # Utilities
├── tests/ # Test suite
├── migrations/ # Database migrations
├── .env.example # Environment variables template
├── .gitignore # Git ignore rules
├── README.md # Project documentation
├── requirements.txt # Dependencies
└── wsgi.py # WSGI entry point
## Installation
1. Clone the repository:
git clone https://github.com/yourusername/flask-ecommerce.git
cd flask-ecommerce
2. Create a virtual environment:
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
3. Install dependencies:
pip install -r requirements.txt
4. Create a `.env` file:
cp .env.example .env
Then edit the `.env` file with your settings.
5. Initialize the database:
flask db init
flask db migrate
flask db upgrade
6. Run the development server:
flask run
## Testing
Run the test suite:
pytest
## Deployment
The application can be deployed using Gunicorn:
gunicorn wsgi
## License
This project is licensed under the MIT License - see the LICENSE file for details.
Benefits of the Refactored Structure
Our refactored project now has these advantages over the original structure:
- Separation of Concerns: Each part of the application has a clear, focused responsibility
- Modularity: Components can be developed, tested, and maintained independently
- Scalability: The structure can accommodate growth as new features are added
- Testability: The modular design makes it easier to write focused tests
- Configuration: Settings are centralized and environment-specific
- Maintainability: New developers can quickly understand the project organization
- Deployment: The application is ready for different deployment scenarios
This exercise demonstrates how a poorly structured project can be transformed into a well-organized, maintainable codebase by applying the principles and patterns we've discussed throughout this guide.
Building Better Python Projects
Throughout this guide, we've explored the art and science of structuring Python projects. From simple scripts to complex enterprise applications, good structure is the foundation of maintainable, scalable, and collaborative code.
Let's recap some key takeaways:
- Structure Matters: Project organization affects maintainability, scalability, collaboration, onboarding, testing, and deployment.
- Choose Appropriate Complexity: Your project structure should match the size and complexity of your application. Don't overengineer simple projects, but don't understructure complex ones.
- Follow the Single Responsibility Principle: Each component should have one clear purpose.
- Keep Related Files Together: Files that change together should be located near each other.
- Create Clear API Boundaries: Make it obvious what's public interface and what's implementation detail.
- Separate Configuration from Code: Allow your application to adapt to different environments without code changes.
- Learn from Successful Projects: Study well-established open-source projects for inspiration and patterns.
- Use Available Tools: Leverage project templates, linters, formatters, and other tools to maintain good structure.
- Be Consistent: Whatever structure you choose, apply it consistently throughout the project.
- Document Your Structure: Make sure new team members can understand your project organization.
- Evolve Gradually: Allow your structure to evolve as your project grows and requirements change.
Remember that project structure is not an end in itself but a means to create better software. The best structure is one that makes your development process more efficient, your code more maintainable, and your team more productive.
As you design and build your next Python project, take the time to thoughtfully consider its structure. The investment will pay dividends throughout the life of your application, making it easier to add features, fix bugs, onboard new developers, and deliver value to users.
In the words of Phil Karlton: "There are only two hard things in Computer Science: cache invalidation and naming things." We might add a third: structuring your code well. But with the principles and patterns we've explored in this guide, you're now better equipped to tackle this challenge.
Happy coding, and may your projects be well-structured!