Introduction to Handling Multiple Exceptions
Welcome to our exploration of handling multiple exceptions in Python! As we've learned, exceptions are Python's way of dealing with unexpected events during program execution. In real-world applications, multiple things can go wrong, and we need strategies for handling different types of errors appropriately.
Think of exception handling as the emergency response system for your code. Just like a city has different teams for handling fires, medical emergencies, and police matters, your code can have specific handlers for different types of exceptions.
Folder Structure for Today's Examples
multiple_exceptions/
├── basics/
│ ├── simple_multiple_except.py
│ ├── exception_groups.py
│ ├── exception_tuple.py
│ └── exception_hierarchy.py
├── advanced/
│ ├── exception_chaining.py
│ ├── unified_handler.py
│ └── exception_filtering.py
├── practical/
│ ├── file_processing.py
│ ├── network_operations.py
│ ├── database_access.py
│ └── data_conversion.py
└── exercises/
├── exercise1.py
├── exercise2.py
└── exercise3.py
Basic Approaches to Handling Multiple Exceptions
Python offers several ways to handle multiple exception types. Let's explore these approaches starting with the most basic.
Multiple except Blocks
# File: basics/simple_multiple_except.py
def divide_and_process_numbers(a, b):
try:
# Several operations that could raise different exceptions
result = a / b # Potential ZeroDivisionError
processed = int(result) # Potential ValueError
return processed
except ZeroDivisionError:
print("Error: Cannot divide by zero!")
return None
except ValueError:
print("Error: Could not convert the result to an integer!")
return None
except Exception as e:
# Catch-all for any other exceptions
print(f"Unexpected error: {e}")
return None
# Test cases
print(divide_and_process_numbers(10, 2)) # Works: 5
print(divide_and_process_numbers(10, 0)) # ZeroDivisionError
print(divide_and_process_numbers(10, 3.5)) # Works: 2 (truncated to int)
print(divide_and_process_numbers("10", 2)) # TypeError (caught by generic Exception)
This is the simplest approach, where each exception type has its own except block. Python tries each block in sequence until it finds a matching exception type.
Key points:
- The order of
exceptblocks matters. More specific exceptions should come first, followed by more general ones. - Once an exception is caught by a matching block, no other blocks are checked.
- Using
Exceptionas a catch-all should generally be the last block to avoid masking more specific errors.
Handling Multiple Exception Types in a Single Block
# File: basics/exception_tuple.py
def process_data(data):
try:
# Operations that might raise different exceptions
value = data['key'] # Potential KeyError
result = 100 / value # Potential ZeroDivisionError
return f"Processed result: {result}"
except (KeyError, TypeError) as e:
# Handle data structure issues together
print(f"Data structure error: {e}")
return None
except (ValueError, ZeroDivisionError) as e:
# Handle calculation issues together
print(f"Calculation error: {e}")
return None
# Test cases
print(process_data({'key': 5})) # Works: Processed result: 20.0
print(process_data({'different_key': 5})) # KeyError
print(process_data(None)) # TypeError
print(process_data({'key': 0})) # ZeroDivisionError
print(process_data({'key': '5'})) # TypeError (can't divide by string)
You can group similar exceptions by providing a tuple of exception types in a single except clause. This is useful when you want to handle related exceptions in the same way.
Advantages:
- Reduces code duplication when multiple exceptions require similar handling
- Groups logically related errors together
- Creates cleaner code with fewer repetitive blocks
Leveraging the Exception Hierarchy
# File: basics/exception_hierarchy.py
def read_config(filename):
try:
with open(filename, 'r') as file:
content = file.read()
config = eval(content) # Danger: eval is used for demonstration only!
return config
except FileNotFoundError:
# Specific handling for missing files
print(f"Config file not found: {filename}")
return {}
except OSError as e:
# Handles broader file system errors (parent of FileNotFoundError)
print(f"OS error when reading config: {e}")
return {}
except SyntaxError as e:
# Handle invalid Python syntax in the config
print(f"Invalid syntax in config file: {e}")
return {}
except Exception as e:
# Generic fallback
print(f"Unexpected error reading config: {e}")
return {}
# Show the exception hierarchy
import inspect
def show_exception_hierarchy(exception_class, indent=0):
print(" " * indent + exception_class.__name__)
for subclass in exception_class.__subclasses__():
show_exception_hierarchy(subclass, indent + 4)
# Show a partial hierarchy starting from OSError
print("Partial Exception Hierarchy:")
show_exception_hierarchy(OSError)
# Test with different files
print("\nTesting with different files:")
print(read_config("config.txt")) # Assuming it doesn't exist
print(read_config("/nonexistent_dir/config.txt")) # Permission error on some systems
Python's exceptions form a hierarchy, which you can leverage to handle exceptions at different levels of specificity. Understanding this hierarchy helps you structure your exception handling more effectively.
Hierarchical handling strategy:
- Catch specific exceptions first for specialized handling
- Use parent exception classes to handle groups of related exceptions
- Remember that child exceptions won't be caught by handlers that appear after their parent exception handler
For example, since FileNotFoundError is a subclass of OSError, a handler for OSError would catch FileNotFoundError too, unless the FileNotFoundError handler comes first.
Advanced Exception Handling Techniques
Let's explore more sophisticated approaches for managing multiple exceptions that provide better organization and error propagation.
Exception Chaining (Exception from Exception)
# File: advanced/exception_chaining.py
def parse_config_value(text):
try:
# Try to parse as an integer
return int(text)
except ValueError:
# Chain the original exception to a new, more specific one
raise ConfigParsingError(f"Invalid numeric value: {text}") from ValueError
def read_config_value(filename, key):
try:
with open(filename, 'r') as file:
for line in file:
if line.startswith(f"{key}="):
value_text = line.split('=', 1)[1].strip()
return parse_config_value(value_text)
# If we get here, the key wasn't found
raise KeyError(f"Config key not found: {key}")
except FileNotFoundError as e:
# Wrap the original exception with more context
raise ConfigError(f"Could not read config file: {filename}") from e
except KeyError as e:
# Propagate KeyError with the original as the cause
raise ConfigError(f"Missing configuration: {e}") from e
# Custom exception classes
class ConfigError(Exception):
"""Base class for configuration-related errors"""
pass
class ConfigParsingError(ConfigError):
"""Raised when a config value can't be parsed"""
pass
# Usage example
def demonstrate_exception_chaining():
try:
# Try to read a configuration value
value = read_config_value("settings.ini", "timeout")
print(f"Timeout setting: {value}")
except ConfigError as e:
print(f"Configuration error: {e}")
# Access the original exception that caused this one
if e.__cause__:
print(f"Original error: {e.__cause__}")
# Run the demonstration
demonstrate_exception_chaining()
Exception chaining allows you to raise a new exception while preserving the original exception as the cause. This is done using the raise ... from ... syntax.
Benefits of exception chaining:
- Preserves the original exception information
- Adds higher-level context to low-level exceptions
- Builds a clear chain of what went wrong
- Makes debugging easier by showing the complete error path
This is like telling someone not just that their package delivery failed, but exactly why: "Your delivery failed because the truck broke down because it ran out of gas."
Unified Exception Handling with Error Classification
# File: advanced/unified_handler.py
def classify_exception(e):
"""Classify an exception into a general category for handling."""
# Check for data access errors
if isinstance(e, (FileNotFoundError, PermissionError, IOError)):
return "DATA_ACCESS_ERROR", f"Could not access required data: {e}"
# Check for data format errors
elif isinstance(e, (ValueError, TypeError, AttributeError)):
return "DATA_FORMAT_ERROR", f"Invalid data format: {e}"
# Check for resource errors
elif isinstance(e, (MemoryError, TimeoutError, ConnectionError)):
return "RESOURCE_ERROR", f"System resource error: {e}"
# Default classification
else:
return "UNKNOWN_ERROR", f"An unexpected error occurred: {e}"
def process_with_unified_handling(func, *args, **kwargs):
"""
Execute a function with unified exception handling.
Args:
func: The function to execute
*args, **kwargs: Arguments to pass to the function
Returns:
A tuple of (success, result, error_code, error_message)
"""
try:
# Try to execute the function
result = func(*args, **kwargs)
return True, result, None, None
except Exception as e:
# Classify and handle the exception
error_code, error_message = classify_exception(e)
# Log the error (in a real system, use a proper logger)
print(f"ERROR [{error_code}]: {error_message}")
print(f"Exception type: {type(e).__name__}")
# Return failure result with error information
return False, None, error_code, error_message
# Example functions to test with
def read_data(filename):
with open(filename, 'r') as file:
return file.read()
def parse_json(text):
import json
return json.loads(text)
def calculate_average(numbers):
return sum(numbers) / len(numbers)
# Demonstrate unified handling
def demonstrate_unified_handling():
# Test with file not found error
success, data, error_code, error_msg = process_with_unified_handling(
read_data, "nonexistent_file.txt"
)
print(f"Operation succeeded: {success}")
if not success:
print(f"Error code: {error_code}")
print()
# Test with JSON parsing error
success, data, error_code, error_msg = process_with_unified_handling(
parse_json, "{ invalid json }"
)
print(f"Operation succeeded: {success}")
if not success:
print(f"Error code: {error_code}")
print()
# Test with division by zero
success, data, error_code, error_msg = process_with_unified_handling(
calculate_average, [] # Empty list will cause division by zero
)
print(f"Operation succeeded: {success}")
if not success:
print(f"Error code: {error_code}")
# Run the demonstration
demonstrate_unified_handling()
This approach creates a unified system for handling exceptions by classifying them into general categories. It's especially useful in larger applications where you want consistent error handling across many components.
Advantages of unified error handling:
- Consistent error responses throughout the application
- Centralizes error handling logic
- Makes it easier to standardize logging, reporting, and user feedback
- Simplifies code at the call site
This is like having a central dispatch service that routes all emergency calls to the appropriate department based on the type of emergency.
Exception Filtering with Predicates
# File: advanced/exception_filtering.py
def match_exception(e, **conditions):
"""
Check if an exception matches specific conditions.
Args:
e: The exception to check
**conditions: Key-value pairs of attributes to match
Returns:
True if all conditions match, False otherwise
"""
for attr, value in conditions.items():
# Check if the exception has the attribute
if not hasattr(e, attr):
return False
# Check if the attribute value matches
if getattr(e, attr) != value:
return False
# All conditions matched
return True
def handle_database_error(func):
"""
A decorator that handles database-related exceptions.
Args:
func: The function to wrap
Returns:
A wrapped function with error handling
"""
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
# Handle connection errors
if isinstance(e, ConnectionError) and match_exception(e, code='timeout'):
print("Database connection timed out. Please try again later.")
return None
elif isinstance(e, ConnectionError) and match_exception(e, code='auth'):
print("Database authentication failed. Please check credentials.")
return None
# Handle constraint violations
elif isinstance(e, ValueError) and 'constraint' in str(e).lower():
print("Database constraint violation. Please check your input data.")
return None
# Re-raise unhandled exceptions
raise
return wrapper
# Simulated database functions and exceptions
class DatabaseError(Exception):
"""Base class for database errors."""
pass
class ConnectionError(DatabaseError):
"""Database connection errors."""
def __init__(self, message, code=None, severity=None):
super().__init__(message)
self.code = code
self.severity = severity
class QueryError(DatabaseError):
"""Database query errors."""
def __init__(self, message, query=None, params=None):
super().__init__(message)
self.query = query
self.params = params
# Simulated database functions
@handle_database_error
def connect_to_database(host, user, password):
"""Simulate connecting to a database."""
if not host:
raise ConnectionError("No host specified", code='param')
if user == "invalid":
raise ConnectionError("Authentication failed", code='auth', severity='high')
if host == "slow-server":
raise ConnectionError("Connection timed out", code='timeout', severity='medium')
print(f"Connected to database at {host}")
return {"connection": "object"}
@handle_database_error
def execute_query(connection, query, params=None):
"""Simulate executing a database query."""
if not connection:
raise ConnectionError("Not connected to database", code='state')
if not query:
raise QueryError("Empty query", query=query, params=params)
if "INSERT" in query and params and 'value' in params and params['value'] > 100:
raise ValueError("Constraint violation: value must be <= 100")
print(f"Executed query: {query}")
return ["result1", "result2"]
# Demonstrate exception filtering
def demonstrate_exception_filtering():
# Test with different scenarios
print("Scenario 1: Valid connection")
conn = connect_to_database("db-server", "user", "password")
print()
print("Scenario 2: Authentication failure")
conn = connect_to_database("db-server", "invalid", "password")
print()
print("Scenario 3: Connection timeout")
conn = connect_to_database("slow-server", "user", "password")
print()
print("Scenario 4: Valid query")
if conn:
results = execute_query(conn, "SELECT * FROM users")
print()
print("Scenario 5: Constraint violation")
if conn:
results = execute_query(conn, "INSERT INTO items VALUES (:value)", {'value': 200})
print()
print("Scenario 6: Unhandled exception")
try:
if conn:
results = execute_query(None, "SELECT * FROM users")
except Exception as e:
print(f"Caught unhandled exception: {e}")
# Run the demonstration
demonstrate_exception_filtering()
Exception filtering lets you handle exceptions based not just on their type but also on their attributes or other conditions. This gives you fine-grained control over exception handling.
When to use exception filtering:
- When you need to differentiate between exceptions of the same type
- When exception details determine how to handle the error
- When working with APIs that use rich exception objects
- For implementing more sophisticated error recovery strategies
This is like a doctor not just identifying that you have an infection, but determining exactly what kind of infection and prescribing the appropriate treatment based on its specific characteristics.
Real-World Examples of Multiple Exception Handling
Let's explore some practical examples of handling multiple exceptions in common programming scenarios.
File Processing with Multiple Error Conditions
# File: practical/file_processing.py
import os
import json
import csv
from datetime import datetime
def process_data_file(input_file, output_file, error_log=None):
"""
Process a data file and write the results to an output file.
Args:
input_file: Path to the input file (JSON or CSV)
output_file: Path to the output file
error_log: Optional path to an error log file
Returns:
A dictionary with processing statistics
"""
stats = {
'input_file': input_file,
'output_file': output_file,
'start_time': datetime.now(),
'end_time': None,
'records_processed': 0,
'records_succeeded': 0,
'records_failed': 0,
'status': 'failed', # Default to failed, update on success
'error': None
}
# Create or open the error log file if specified
error_log_file = None
if error_log:
try:
error_log_file = open(error_log, 'a')
error_log_file.write(f"\n\n--- Processing {input_file} at {stats['start_time']} ---\n")
except Exception as e:
print(f"Warning: Could not open error log file: {e}")
try:
# Make sure the output directory exists
output_dir = os.path.dirname(output_file)
if output_dir and not os.path.exists(output_dir):
os.makedirs(output_dir)
# Determine the file type based on extension
file_extension = os.path.splitext(input_file)[1].lower()
# Read the input file
data = []
if file_extension == '.json':
# Handle JSON file
try:
with open(input_file, 'r') as f:
data = json.load(f)
# Ensure data is a list
if not isinstance(data, list):
if isinstance(data, dict) and 'records' in data:
# Handle common format where data is in a "records" field
data = data['records']
else:
# Wrap single object in a list
data = [data]
except json.JSONDecodeError as e:
raise ValueError(f"Invalid JSON format: {e}")
elif file_extension == '.csv':
# Handle CSV file
try:
with open(input_file, 'r', newline='') as f:
reader = csv.DictReader(f)
data = list(reader)
except csv.Error as e:
raise ValueError(f"Invalid CSV format: {e}")
else:
raise ValueError(f"Unsupported file type: {file_extension}")
# Process the data
output_data = []
for i, record in enumerate(data):
try:
# Process the record (in a real system, this would do something useful)
processed_record = {
'id': record.get('id', i),
'timestamp': datetime.now().isoformat(),
'original_data': record,
'processed': True
}
output_data.append(processed_record)
stats['records_succeeded'] += 1
except Exception as e:
# Log the error for this record
error_message = f"Error processing record {i}: {e}"
if error_log_file:
error_log_file.write(f"{error_message}\n")
else:
print(error_message)
stats['records_failed'] += 1
# Include the failed record with error information
output_data.append({
'id': record.get('id', i),
'timestamp': datetime.now().isoformat(),
'original_data': record,
'processed': False,
'error': str(e)
})
# Update the records processed count
stats['records_processed'] += 1
# Write the output data
output_extension = os.path.splitext(output_file)[1].lower()
if output_extension == '.json':
with open(output_file, 'w') as f:
json.dump(output_data, f, indent=2)
elif output_extension == '.csv':
if output_data:
# Get all possible field names from all records
fieldnames = set()
for record in output_data:
fieldnames.update(record.keys())
with open(output_file, 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=sorted(fieldnames))
writer.writeheader()
writer.writerows(output_data)
else:
raise ValueError(f"Unsupported output file type: {output_extension}")
# Update statistics
stats['status'] = 'success'
except FileNotFoundError as e:
stats['error'] = f"File not found: {e}"
if error_log_file:
error_log_file.write(f"ERROR: {stats['error']}\n")
raise
except PermissionError as e:
stats['error'] = f"Permission denied: {e}"
if error_log_file:
error_log_file.write(f"ERROR: {stats['error']}\n")
raise
except ValueError as e:
stats['error'] = f"Value error: {e}"
if error_log_file:
error_log_file.write(f"ERROR: {stats['error']}\n")
raise
except Exception as e:
stats['error'] = f"Unexpected error: {e}"
if error_log_file:
error_log_file.write(f"ERROR: {stats['error']}\n")
raise
finally:
# Update end time
stats['end_time'] = datetime.now()
# Close the error log file if it was opened
if error_log_file:
error_log_file.write(f"Completed with status: {stats['status']}\n")
error_log_file.write(f"Processed {stats['records_processed']} records: "
f"{stats['records_succeeded']} succeeded, "
f"{stats['records_failed']} failed\n")
error_log_file.write(f"Total time: {stats['end_time'] - stats['start_time']}\n")
error_log_file.close()
return stats
# Function to demonstrate the file processor
def demonstrate_file_processor():
# Create some sample data for testing
sample_data = [
{"id": 1, "name": "Alice", "score": 95},
{"id": 2, "name": "Bob", "score": "invalid"}, # Will cause an error during processing
{"id": 3, "name": "Charlie", "score": 85}
]
# Write sample data to a JSON file
os.makedirs("sample_data", exist_ok=True)
with open("sample_data/input.json", 'w') as f:
json.dump(sample_data, f)
# Write sample data to a CSV file
with open("sample_data/input.csv", 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=["id", "name", "score"])
writer.writeheader()
writer.writerows(sample_data)
# Process the files
print("Processing JSON file:")
try:
stats = process_data_file(
"sample_data/input.json",
"sample_data/output.json",
"sample_data/errors.log"
)
print(f"Processing completed with status: {stats['status']}")
print(f"Processed {stats['records_processed']} records: "
f"{stats['records_succeeded']} succeeded, "
f"{stats['records_failed']} failed")
except Exception as e:
print(f"Processing failed: {e}")
print("\nProcessing CSV file:")
try:
stats = process_data_file(
"sample_data/input.csv",
"sample_data/output.csv",
"sample_data/errors.log"
)
print(f"Processing completed with status: {stats['status']}")
print(f"Processed {stats['records_processed']} records: "
f"{stats['records_succeeded']} succeeded, "
f"{stats['records_failed']} failed")
except Exception as e:
print(f"Processing failed: {e}")
print("\nTesting error handling with nonexistent file:")
try:
stats = process_data_file(
"sample_data/nonexistent.json",
"sample_data/output.json",
"sample_data/errors.log"
)
except FileNotFoundError as e:
print(f"Expected error caught: {e}")
print("\nTesting error handling with invalid JSON:")
# Create an invalid JSON file
with open("sample_data/invalid.json", 'w') as f:
f.write("{this is not valid JSON}")
try:
stats = process_data_file(
"sample_data/invalid.json",
"sample_data/output.json",
"sample_data/errors.log"
)
except ValueError as e:
print(f"Expected error caught: {e}")
# Run the demonstration
# demonstrate_file_processor()
This example demonstrates a robust file processing system that handles multiple types of exceptions at different levels. It shows how to:
- Handle file-level errors (missing files, permission issues)
- Handle format-specific parsing errors (invalid JSON, CSV)
- Handle record-level errors to allow processing to continue
- Log errors for later analysis
- Provide detailed statistics on the processing results
This pattern is common in data processing pipelines, ETL (Extract, Transform, Load) processes, and batch processing systems.
Network Operations with Multiple Failure Modes
# File: practical/network_operations.py
import socket
import json
import time
import random
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
from urllib.parse import urlencode
class APIError(Exception):
"""Base class for API-related errors."""
pass
class ConnectionFailure(APIError):
"""Raised when connection to the API fails."""
pass
class AuthenticationError(APIError):
"""Raised when authentication with the API fails."""
pass
class RateLimitError(APIError):
"""Raised when the API rate limit is exceeded."""
pass
class APIClientError(APIError):
"""Raised when the API reports a client error (4xx status code)."""
pass
class APIServerError(APIError):
"""Raised when the API reports a server error (5xx status code)."""
pass
class SimpleAPIClient:
"""
A simple API client that demonstrates handling multiple network-related exceptions.
"""
def __init__(self, base_url, api_key=None, timeout=10, max_retries=3):
self.base_url = base_url.rstrip('/')
self.api_key = api_key
self.timeout = timeout
self.max_retries = max_retries
def request(self, endpoint, method='GET', params=None, data=None, headers=None):
"""
Send a request to the API.
Args:
endpoint: API endpoint (without base URL)
method: HTTP method (GET, POST, PUT, DELETE)
params: Query parameters
data: Request body data for POST/PUT
headers: Custom headers
Returns:
The parsed JSON response
Raises:
Various APIError subclasses depending on what went wrong
"""
# Build the full URL
url = f"{self.base_url}/{endpoint.lstrip('/')}"
# Add query parameters
if params:
query_string = urlencode(params)
url = f"{url}?{query_string}"
# Prepare headers
request_headers = headers or {}
if self.api_key:
request_headers['Authorization'] = f"Bearer {self.api_key}"
if data:
request_headers['Content-Type'] = 'application/json'
# Convert data to JSON if needed
json_data = None
if data:
json_data = json.dumps(data).encode('utf-8')
# Create the request
request = Request(
url=url,
data=json_data,
headers=request_headers,
method=method
)
# Try to send the request with retries
retries = 0
while True:
try:
# Attempt to send the request
response = urlopen(request, timeout=self.timeout)
# Parse the response
response_data = response.read().decode('utf-8')
# Return the parsed JSON data
return json.loads(response_data) if response_data else {}
except HTTPError as e:
# Handle HTTP error responses
if e.code == 401 or e.code == 403:
raise AuthenticationError(f"Authentication failed: {e}")
elif e.code == 429:
if retries < self.max_retries:
# Rate limited, wait and retry
retry_after = int(e.headers.get('Retry-After', 5))
print(f"Rate limited. Waiting {retry_after} seconds before retry...")
time.sleep(retry_after)
retries += 1
continue
else:
raise RateLimitError(f"API rate limit exceeded: {e}")
elif 400 <= e.code < 500:
# Client errors
raise APIClientError(f"Client error (HTTP {e.code}): {e}")
elif 500 <= e.code < 600:
# Server errors
if retries < self.max_retries:
# Server error, wait and retry
backoff = (2 ** retries) + random.uniform(0, 1)
print(f"Server error. Retrying in {backoff:.2f} seconds...")
time.sleep(backoff)
retries += 1
continue
else:
raise APIServerError(f"Server error (HTTP {e.code}): {e}")
# Re-raise any other HTTP errors
raise APIError(f"HTTP error: {e}")
except URLError as e:
# Handle URL errors (connection problems)
if isinstance(e.reason, socket.timeout):
if retries < self.max_retries:
# Timeout, wait and retry
backoff = (2 ** retries) + random.uniform(0, 1)
print(f"Request timed out. Retrying in {backoff:.2f} seconds...")
time.sleep(backoff)
retries += 1
continue
else:
raise ConnectionFailure(f"Connection timed out after {self.max_retries} retries")
elif "connection refused" in str(e.reason).lower():
raise ConnectionFailure(f"Connection refused: {e.reason}")
# Re-raise other URL errors
raise ConnectionFailure(f"Connection failed: {e.reason}")
except socket.timeout:
# Handle socket timeouts
if retries < self.max_retries:
# Timeout, wait and retry
backoff = (2 ** retries) + random.uniform(0, 1)
print(f"Request timed out. Retrying in {backoff:.2f} seconds...")
time.sleep(backoff)
retries += 1
continue
else:
raise ConnectionFailure(f"Connection timed out after {self.max_retries} retries")
except json.JSONDecodeError as e:
# Handle invalid JSON responses
raise APIError(f"Invalid JSON response: {e}")
except Exception as e:
# Handle any other unexpected errors
raise APIError(f"Unexpected error: {e}")
def get(self, endpoint, params=None, headers=None):
"""Send a GET request to the API."""
return self.request(endpoint, method='GET', params=params, headers=headers)
def post(self, endpoint, data=None, params=None, headers=None):
"""Send a POST request to the API."""
return self.request(endpoint, method='POST', params=params, data=data, headers=headers)
def put(self, endpoint, data=None, params=None, headers=None):
"""Send a PUT request to the API."""
return self.request(endpoint, method='PUT', params=params, data=data, headers=headers)
def delete(self, endpoint, params=None, headers=None):
"""Send a DELETE request to the API."""
return self.request(endpoint, method='DELETE', params=params, headers=headers)
# Function to demonstrate the API client
def demonstrate_api_client():
# Create an API client for a public test API
client = SimpleAPIClient(
base_url="https://jsonplaceholder.typicode.com",
timeout=5,
max_retries=2
)
# Example 1: Successful GET request
try:
print("Example 1: Successful GET request")
response = client.get("/posts/1")
print(f"Response: {response}")
except APIError as e:
print(f"API Error: {e}")
print()
# Example 2: Resource not found (404)
try:
print("Example 2: Resource not found (404)")
response = client.get("/nonexistent-endpoint")
print(f"Response: {response}")
except APIClientError as e:
print(f"Expected client error: {e}")
except APIError as e:
print(f"API Error: {e}")
print()
# Example 3: POST request
try:
print("Example 3: POST request")
new_post = {
"title": "Test Post",
"body": "This is a test post",
"userId": 1
}
response = client.post("/posts", data=new_post)
print(f"Response: {response}")
except APIError as e:
print(f"API Error: {e}")
print()
# Example 4: Connection failure (invalid hostname)
try:
print("Example 4: Connection failure (invalid hostname)")
bad_client = SimpleAPIClient("https://nonexistent-api-host.invalid")
response = bad_client.get("/endpoint")
print(f"Response: {response}")
except ConnectionFailure as e:
print(f"Expected connection failure: {e}")
except APIError as e:
print(f"API Error: {e}")
print()
# Example 5: Handle a timeout
try:
print("Example 5: Handle a timeout")
# Simulate a timeout by using a very short timeout value
timeout_client = SimpleAPIClient(
base_url="https://httpbin.org/delay/3", # This endpoint delays 3 seconds
timeout=1, # Only wait 1 second
max_retries=1
)
response = timeout_client.get("")
print(f"Response: {response}")
except ConnectionFailure as e:
print(f"Expected timeout: {e}")
except APIError as e:
print(f"API Error: {e}")
# Run the demonstration
# demonstrate_api_client()
This example demonstrates handling the many types of errors that can occur during network operations and API calls, including:
- Connection failures (timeouts, refused connections)
- HTTP errors (client errors, server errors)
- Authentication failures
- Rate limiting
- JSON parsing errors
The example also shows how to implement retry logic for transient errors, with exponential backoff to avoid overwhelming the server.
Best Practices for Handling Multiple Exceptions
Key Principles
- Order exceptions from most specific to most general - Always catch the most specific exceptions first, followed by more general ones.
- Don't catch exceptions you can't handle properly - If you can't take appropriate action for an exception, let it propagate up the call stack.
- Keep error handling separate from business logic - This makes both the error handling and the main code easier to understand.
- Use custom exceptions for domain-specific errors - Create a hierarchy of exception classes that makes sense for your application.
- Be specific about which exceptions you catch - Avoid catching
Exceptionor worse, all exceptions with a bareexcept:clause. - Always clean up resources - Use
finallyblocks or context managers (withstatement) to ensure resources are properly released. - Provide context in exceptions - Include relevant information that helps understand what went wrong and how to fix it.
- Consider retries for transient errors - Some errors (network timeouts, rate limiting) are temporary and can be resolved by trying again.
- Log exceptions with appropriate detail - Include enough information to debug the issue, but be careful about sensitive data.
- Use appropriate abstraction levels - Lower-level exceptions should be caught and translated to higher-level exceptions that make sense in the current context.
Common Anti-Patterns to Avoid
- Bare except: clauses - This catches all exceptions including keyboard interrupts and system exits.
- Catching Exception too broadly - Unless you're at a very high level in your application, this probably catches too much.
- Empty except blocks - Silently ignoring errors makes debugging nearly impossible.
- Long code blocks in try - Keep try blocks focused on the specific operations that might raise exceptions.
- Handling exceptions in the wrong place - Catch exceptions where you have enough context to handle them properly.
- Raising string exceptions - Always raise instances of the Exception class or its subclasses, not strings.
- Catching and re-raising without adding value - If you're just going to re-raise, consider whether you need to catch the exception at all.
- Using exceptions for flow control - Exceptions should be for exceptional conditions, not normal program flow.
Before and After: Improving Exception Handling
# BEFORE: Problematic exception handling
def problematic_function(filename, value):
try:
# Long block with multiple potential exceptions
file = open(filename, 'r')
data = file.read()
result = int(data) / value
file.close()
return result
except:
# Bare except, no specific handling
print("An error occurred")
# No cleanup
return None
# AFTER: Improved exception handling
def improved_function(filename, value):
try:
# Using a context manager for proper resource handling
with open(filename, 'r') as file:
data = file.read()
# Separate try block for different operation
try:
return int(data) / value
except ValueError:
# Specific handling for integer conversion
print(f"Error: File does not contain a valid integer: {data}")
return None
except ZeroDivisionError:
# Specific handling for division by zero
print("Error: Cannot divide by zero")
return float('inf') # Or another appropriate default
except FileNotFoundError:
# Specific handling for missing file
print(f"Error: File not found: {filename}")
return None
except PermissionError:
# Specific handling for permission issues
print(f"Error: Permission denied for file: {filename}")
return None
except Exception as e:
# General handler as last resort, with useful information
print(f"Unexpected error processing file {filename}: {e}")
return None
# Test both functions
def compare_functions():
print("Testing problematic function:")
print(f"Missing file: {problematic_function('nonexistent.txt', 5)}")
print(f"Invalid data: {problematic_function('invalid_data.txt', 5)}")
print(f"Zero value: {problematic_function('valid_data.txt', 0)}")
print()
print("Testing improved function:")
print(f"Missing file: {improved_function('nonexistent.txt', 5)}")
print(f"Invalid data: {improved_function('invalid_data.txt', 5)}")
print(f"Zero value: {improved_function('valid_data.txt', 0)}")
# Create test files
def create_test_files():
with open('valid_data.txt', 'w') as f:
f.write('10')
with open('invalid_data.txt', 'w') as f:
f.write('not a number')
# Uncomment to run
# create_test_files()
# compare_functions()
This example contrasts poor exception handling with a better approach, highlighting several key improvements:
- Using context managers (
withstatement) for resource management - Separating different operations into different try/except blocks
- Catching specific exceptions rather than using a bare
except - Providing informative error messages
- Handling different error conditions differently
- Using
Exceptionas a last resort, not the primary handler
Exercises to Reinforce Learning
Exercise 1: Database Connection Wrapper
Create a database connection wrapper class that handles multiple types of database-related exceptions, providing a unified interface for error handling and retries.
# File: exercises/database_connection.py
class DatabaseConnection:
"""
A wrapper for database connections with unified error handling.
Your task:
1. Implement the connect, execute, and close methods
2. Add appropriate exception handling for different database error scenarios
3. Implement retry logic for transient errors
4. Ensure proper resource cleanup
"""
def __init__(self, connection_string, max_retries=3):
self.connection_string = connection_string
self.max_retries = max_retries
self.connection = None
def connect(self):
"""
Establish a connection to the database.
Returns:
True if connection successful, False otherwise
Raises:
Various exceptions depending on what went wrong
"""
# Your implementation here
pass
def execute(self, query, params=None):
"""
Execute a query on the database.
Args:
query: SQL query to execute
params: Parameters for the query
Returns:
Query results
Raises:
Various exceptions depending on what went wrong
"""
# Your implementation here
pass
def close(self):
"""
Close the database connection.
"""
# Your implementation here
pass
def __enter__(self):
"""Support for use as a context manager."""
self.connect()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Clean up resources when used as a context manager."""
# Your implementation here
pass
Exercise 2: Robust Configuration Manager
Create a configuration manager that can read configuration from multiple sources (environment variables, config files, defaults) and handle various error conditions gracefully.
# File: exercises/config_manager.py
class ConfigManager:
"""
A configuration manager that reads from multiple sources with error handling.
Your task:
1. Implement methods to read configuration from different sources
2. Handle different types of errors (missing files, invalid formats, etc.)
3. Implement fallback mechanisms
4. Provide meaningful error messages
"""
def __init__(self, app_name, config_files=None, env_prefix=None):
self.app_name = app_name
self.config_files = config_files or []
self.env_prefix = env_prefix or app_name.upper()
self.config = {}
def load_config(self):
"""
Load configuration from all sources.
Returns:
The merged configuration dictionary
"""
# Your implementation here
pass
def load_from_file(self, filename):
"""
Load configuration from a file.
Args:
filename: Path to the configuration file
Returns:
Configuration dictionary from the file
Raises:
Various exceptions depending on what went wrong
"""
# Your implementation here
pass
def load_from_env(self):
"""
Load configuration from environment variables.
Returns:
Configuration dictionary from environment variables
"""
# Your implementation here
pass
def get(self, key, default=None):
"""
Get a configuration value.
Args:
key: Configuration key
default: Default value if key not found
Returns:
The configuration value or default
"""
# Your implementation here
pass
def set(self, key, value):
"""
Set a configuration value.
Args:
key: Configuration key
value: Configuration value
"""
# Your implementation here
pass
def save(self, filename=None):
"""
Save the current configuration to a file.
Args:
filename: Path to the output file (optional)
Returns:
True if successful, False otherwise
"""
# Your implementation here
pass
Exercise 3: HTTP Request Validator
Create a function that validates HTTP requests, checking various aspects of the request and raising appropriate exceptions for different validation failures.
# File: exercises/request_validator.py
class ValidationError(Exception):
"""Base class for request validation errors."""
pass
class InvalidMethodError(ValidationError):
"""Raised when the HTTP method is not allowed."""
pass
class InvalidContentTypeError(ValidationError):
"""Raised when the content type is not supported."""
pass
class MissingRequiredFieldError(ValidationError):
"""Raised when a required field is missing."""
pass
class InvalidFieldValueError(ValidationError):
"""Raised when a field has an invalid value."""
pass
class RequestTooLargeError(ValidationError):
"""Raised when the request payload is too large."""
pass
def validate_request(method, headers, body, allowed_methods=None, max_size=None, required_fields=None, field_validators=None):
"""
Validate an HTTP request.
Args:
method: HTTP method (GET, POST, etc.)
headers: Dictionary of HTTP headers
body: Request body (dictionary)
allowed_methods: List of allowed HTTP methods
max_size: Maximum allowed request body size
required_fields: List of required fields in the body
field_validators: Dictionary mapping field names to validator functions
Returns:
None if validation succeeds
Raises:
Various ValidationError subclasses depending on what's invalid
"""
# Your implementation here
pass
# Example validator functions
def validate_email(value):
"""Validator for email fields."""
if '@' not in value:
raise InvalidFieldValueError("Invalid email format")
def validate_age(value):
"""Validator for age fields."""
try:
age = int(value)
if age < 0 or age > 120:
raise InvalidFieldValueError("Age must be between 0 and 120")
except (ValueError, TypeError):
raise InvalidFieldValueError("Age must be a number")
# Test the validator
def test_validator():
"""Test cases for the request validator."""
# Example 1: Valid request
try:
validate_request(
method="POST",
headers={"Content-Type": "application/json"},
body={"name": "Alice", "email": "alice@example.com", "age": 30},
allowed_methods=["POST", "PUT"],
max_size=1024,
required_fields=["name", "email"],
field_validators={"email": validate_email, "age": validate_age}
)
print("Example 1: Valid request passed validation")
except ValidationError as e:
print(f"Example 1 failed: {e}")
# Example 2: Invalid method
try:
validate_request(
method="DELETE",
headers={"Content-Type": "application/json"},
body={"name": "Alice", "email": "alice@example.com"},
allowed_methods=["POST", "PUT"],
required_fields=["name", "email"]
)
print("Example 2: Should have failed but didn't")
except InvalidMethodError as e:
print(f"Example 2: Correctly caught {type(e).__name__}: {e}")
except ValidationError as e:
print(f"Example 2: Caught wrong error type: {type(e).__name__}: {e}")
# Additional test cases for other validation errors...
# Uncomment to run the tests
# test_validator()
Summary
In this comprehensive guide to handling multiple exceptions in Python, we've covered:
- The basic approaches to handling different exception types using multiple
exceptblocks and exception tuples - Leveraging Python's exception hierarchy for more elegant error handling
- Advanced techniques like exception chaining, unified error handling, and exception filtering
- Real-world examples of handling multiple exceptions in file processing and network operations
- Best practices and common anti-patterns in exception handling
Effective exception handling is key to building robust, maintainable Python applications. By understanding the different approaches and when to use them, you can write code that gracefully handles errors, provides helpful feedback, and ensures your applications remain stable even when unexpected situations arise.
Remember that good exception handling is not just about preventing crashes—it's about making your code more resilient, easier to debug, and ultimately more reliable. By applying the principles and techniques covered in this guide, you'll be well-equipped to handle the diverse range of errors that can occur in real-world applications.