Introduction to Error Handling
Welcome to our deep dive into error handling in Python! Today, we'll explore one of the most practical and essential aspects of programming: how to gracefully handle errors and exceptions. This skill separates novice programmers from professionals - no matter how well-designed your code is, unexpected situations will occur in real-world applications.
This tutorial will be stored in your course files as: /week2_thursday_error_handling.html
Understanding Errors and Exceptions
Before we dive into handling errors, let's understand what they are. In Python, there are generally two kinds of errors:
- Syntax errors: Errors in the structure of your code that prevent it from running (like missing colons, unclosed parentheses, etc.)
- Exceptions: Errors that occur during execution when something unexpected happens
Syntax errors need to be fixed before your code will run, but exceptions can be caught and handled, allowing your program to continue running even when something goes wrong.
Real-world Analogy: Driving a Car
Think of exceptions like unexpected events while driving:
- A syntax error is like trying to start a car with no key - you won't even get moving.
- An exception is like a flat tire while driving - an unexpected problem that requires handling.
- Exception handling is like having a spare tire and knowing how to change it - you've prepared for the problem and can continue your journey.
Why Error Handling Matters
Error handling is not just about preventing crashes - it's about creating robust, user-friendly software. Here's why it's crucial:
- Robustness: Programs that can handle unexpected situations without crashing
- User experience: Friendly error messages instead of cryptic technical details
- Debugging: Better error information makes finding and fixing bugs easier
- Security: Prevents exploitation of error conditions and information leakage
- Data integrity: Ensures operations complete fully or not at all
Real-world Application: Web Form Submission
When a user submits a form on your website:
- Without error handling: If the database connection fails, the user sees a server error page with confusing technical details, has no idea if their data was saved, and your application crashes.
- With error handling: If the database connection fails, the user sees a friendly message like "We couldn't process your submission right now. Your data has been saved and we'll try again soon." Your application logs the error and continues serving other users.
Basic try/except Structure
Python's primary mechanism for handling exceptions is the try/except block:
try:
# Code that might raise an exception
result = 10 / 0 # This will raise a ZeroDivisionError
except:
# Code to handle the exception
print("An error occurred!")
The basic structure works like this:
- The code inside the
tryblock is executed. - If no exception occurs, the
exceptblock is skipped. - If an exception occurs, the rest of the
tryblock is skipped and theexceptblock is executed. - After either case, execution continues after the try/except block.
Warning: Avoid Bare Except!
Using a bare except: without specifying which exceptions to catch is generally considered bad practice. It can hide bugs and catch exceptions you didn't intend to handle. Always specify which exceptions you expect to catch.
Catching Specific Exceptions
In practice, you should always catch specific exceptions rather than using a bare except:
try:
number = int(input("Enter a number: "))
result = 10 / number
print(f"10 divided by {number} is {result}")
except ZeroDivisionError:
print("You can't divide by zero!")
except ValueError:
print("That's not a valid number!")
This approach has several advantages:
- Different exceptions can be handled differently
- Unexpected exceptions still propagate (which is good for debugging)
- Your code more clearly communicates what exceptions you're expecting
Common Built-in Exceptions
| Exception | Description | Example Scenario |
|---|---|---|
ValueError |
Inappropriate value for operation | Converting a string like "hello" to an integer |
TypeError |
Operation or function applied to inappropriate type | Adding a string to an integer without conversion |
ZeroDivisionError |
Division or modulo by zero | Dividing any number by zero |
FileNotFoundError |
Trying to access a file that doesn't exist | Opening a file that was deleted or moved |
IndexError |
Sequence index out of range | Accessing list[10] in a list with only 5 elements |
KeyError |
Dictionary key not found | Accessing dict['key'] when 'key' isn't in the dictionary |
AttributeError |
Object has no such attribute or method | Calling string.append() when strings don't have an append method |
The Exception Hierarchy
Python's exceptions form a hierarchy, with the base class BaseException at the top, and Exception as the parent class for most exceptions you'll catch. This hierarchy is important because when you catch an exception, you also catch all its subclasses.
try:
# This code could raise various types of exceptions
with open("data.txt", "r") as file:
data = file.read()
value = int(data)
result = 100 / value
except (ValueError, ZeroDivisionError):
# Catch multiple specific exceptions
print("Invalid data in file!")
except OSError as e:
# The 'as' clause captures the exception object
print(f"File error: {e}")
Exception Hierarchy (Simplified)
- BaseException
- SystemExit
- KeyboardInterrupt
- Exception (almost all other exceptions derive from this)
- ArithmeticError
- ZeroDivisionError
- OverflowError
- LookupError
- IndexError
- KeyError
- OSError
- FileNotFoundError
- PermissionError
- ValueError
- TypeError
- And many more...
- ArithmeticError
Note: When catching exceptions, order matters! Always catch more specific exceptions before more general ones.
The else and finally Clauses
The try/except statement can include two additional clauses: else and finally.
The else Clause
The else clause executes only if the try block completes without raising an exception:
try:
number = int(input("Enter a number: "))
result = 100 / number
except ValueError:
print("That's not a valid number!")
except ZeroDivisionError:
print("You can't divide by zero!")
else:
# This runs only if no exceptions were raised
print(f"The result is {result}")
# You can also put code here that might raise different exceptions
The else clause is useful when you want some code to run only if no exceptions occur, but you still want to catch exceptions in the try block.
The finally Clause
The finally clause always executes, whether an exception was raised or not (even if there's a return statement in the try or except blocks):
try:
file = open("important_data.txt", "w")
file.write("Critical information")
# Some code that might raise an exception
result = calculate_something()
except Exception as e:
print(f"An error occurred: {e}")
finally:
# This ALWAYS runs, ensuring the file is closed
file.close()
print("File has been closed")
The finally clause is perfect for cleanup code that must run regardless of whether an exception occurred.
Try/Except/Else/Finally Analogy: Baking a Cake
- try: Attempt to bake a cake following a recipe
- except: Handle specific problems (out of eggs? use a substitute)
- else: If the cake baked perfectly, add frosting and decoration
- finally: No matter what happened, clean up the kitchen
Complete Example: File Processing
Let's pull everything together in a practical example of processing a data file:
def process_data_file(filename):
processed_data = []
file = None
try:
# Attempt to open and process the file
file = open(filename, "r")
for line_number, line in enumerate(file, 1):
try:
# Process each line (might fail for various reasons)
data = line.strip().split(',')
if len(data) != 3:
raise ValueError(f"Line {line_number}: Expected 3 values but got {len(data)}")
name = data[0]
age = int(data[1])
score = float(data[2])
if age < 0 or age > 120:
raise ValueError(f"Line {line_number}: Invalid age {age}")
if score < 0 or score > 100:
raise ValueError(f"Line {line_number}: Invalid score {score}")
processed_data.append({
'name': name,
'age': age,
'score': score
})
except ValueError as e:
# Handle data format errors for this line
print(f"Error processing line {line_number}: {e}")
# Continue processing the next line
except FileNotFoundError:
print(f"The file {filename} does not exist.")
except PermissionError:
print(f"No permission to read the file {filename}.")
except Exception as e:
print(f"An unexpected error occurred: {e}")
else:
print(f"Successfully processed {len(processed_data)} records.")
return processed_data
finally:
# Ensure file is closed even if an exception occurred
if file is not None:
file.close()
print(f"File {filename} has been closed.")
# This only executes if an exception occurred in the main try block
return None
# Example usage
data = process_data_file("student_data.txt")
This example demonstrates several key concepts:
- Nested try/except blocks for different levels of error handling
- Catching and handling specific exceptions differently
- Using the else clause for code that runs only on success
- Using finally for cleanup that always happens
- Custom validation with raised exceptions
- Gracefully continuing when possible (skipping bad lines)
Raising Exceptions
Sometimes you need to raise exceptions yourself, either to report errors in your code or to re-raise caught exceptions:
def divide(a, b):
if b == 0:
raise ZeroDivisionError("Cannot divide by zero")
return a / b
def calculate_average(numbers):
if not numbers:
raise ValueError("Cannot calculate average of empty list")
return sum(numbers) / len(numbers)
# Re-raising with additional information
def process_user_data(user_id):
try:
user = get_user_from_database(user_id)
return process_user(user)
except DatabaseError as e:
# Add context to the exception
raise RuntimeError(f"Error processing user {user_id}") from e
The raise ... from ... syntax creates an exception chain, preserving the original exception as the cause of the new one. This is very helpful for debugging because it shows the complete exception history.
When to Raise Exceptions
- Invalid arguments: When a function receives arguments it can't work with
- Impossible states: When your program reaches a state that shouldn't be possible
- API contracts: When callers of your code violate expected conditions
- Re-raising: To add context to caught exceptions before propagating them
Creating Custom Exceptions
For complex applications, it's often valuable to create your own exception types:
class ApplicationError(Exception):
"""Base class for all exceptions in this application."""
pass
class ConfigurationError(ApplicationError):
"""Raised when there's a problem with application configuration."""
pass
class DatabaseConnectionError(ApplicationError):
"""Raised when the application cannot connect to the database."""
pass
class UserNotFoundError(ApplicationError):
"""Raised when a requested user doesn't exist."""
def __init__(self, user_id):
self.user_id = user_id
super().__init__(f"User with ID {user_id} not found")
Custom exceptions have several benefits:
- More precise exception handling specific to your application
- Clearer error messages and better debugging
- Logical organization of different error types
- Ability to attach additional contextual data to exceptions
Using Custom Exceptions
def get_user(user_id):
try:
# Attempt to get user from database
if not valid_connection():
raise DatabaseConnectionError("Database connection failed")
user = find_user(user_id)
if user is None:
raise UserNotFoundError(user_id)
return user
except ApplicationError as e:
# We can catch all our custom exceptions together
log_error(e)
# Or we could have specific handling for different subclasses
raise
Context Managers and the with Statement
Python's with statement provides a clean way to handle resource cleanup, even if exceptions occur:
# Without with statement
file = open("data.txt", "r")
try:
data = file.read()
# Process data...
finally:
file.close()
# With the with statement (much cleaner!)
with open("data.txt", "r") as file:
data = file.read()
# Process data...
# File is automatically closed, even if an exception occurs
The with statement works with context managers - objects that define __enter__ and __exit__ methods. When the with block ends (normally or due to an exception), the __exit__ method is called, ensuring cleanup.
Common Context Managers
open()for file handling- Database connections and transactions
- Network connections
- Locks and semaphores
- Temporarily changing settings
Creating Your Own Context Managers
You can create custom context managers using a class or the contextlib module:
from contextlib import contextmanager
# Using a class
class Timer:
def __enter__(self):
import time
self.start_time = time.time()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
import time
self.end_time = time.time()
print(f"Operation took {self.end_time - self.start_time:.2f} seconds")
# Return False to let any exceptions propagate
return False
# Using the contextmanager decorator
@contextmanager
def timer():
import time
start_time = time.time()
try:
# The yield statement is where the with block's content executes
yield
finally:
end_time = time.time()
print(f"Operation took {end_time - start_time:.2f} seconds")
# Usage
with Timer():
# Code to time...
process_large_file()
# Or with the decorator version
with timer():
# Code to time...
process_large_file()
Error Handling Best Practices
Do's
- Catch specific exceptions, not bare
except:clauses - Handle exceptions at the right level - where you have enough context to take appropriate action
- Use else for success-only code that might raise different exceptions
- Use finally or with for cleanup to ensure resources are properly managed
- Include useful information in exception messages to aid debugging
- Document exceptions your functions might raise
Don'ts
- Don't catch exceptions you can't handle meaningfully
- Don't silence exceptions without good reason (avoid
except: pass) - Don't catch Exception or BaseException unless you really mean to catch everything
- Don't use exceptions for flow control if regular control structures would work
- Don't overuse custom exceptions - use built-in types when they're appropriate
Common Error Handling Patterns
1. Look Before You Leap (LBYL) vs. Easier to Ask Forgiveness than Permission (EAFP)
# LBYL style (check conditions first)
if os.path.exists(filename):
with open(filename) as f:
data = f.read()
else:
data = default_data
# EAFP style (try it and handle exceptions)
try:
with open(filename) as f:
data = f.read()
except FileNotFoundError:
data = default_data
Python generally favors the EAFP style, which can be more concise and avoids race conditions.
2. Reraise Pattern
try:
result = complex_operation()
except SomeError as e:
logger.error(f"Operation failed: {e}")
raise # Re-raise the caught exception
This pattern lets you take action (like logging) while still propagating the exception.
3. Conversion Pattern
try:
result = api.get_data()
except APIError as e:
# Convert to a more appropriate exception for this layer
raise DataUnavailableError("Could not retrieve data") from e
This pattern translates low-level exceptions into more appropriate ones for your application layer.
Error Handling in Web Development
In web applications, error handling is particularly important. Here are some specific applications:
Form Validation and User Input
def process_user_registration(data):
try:
# Validate and process user data
username = data['username']
if not username:
raise ValueError("Username cannot be empty")
email = data['email']
if '@' not in email:
raise ValueError("Invalid email format")
age = int(data.get('age', 0))
if age < 18:
raise ValueError("User must be at least 18 years old")
# ... more validation
# If everything is valid, create user
return create_user(username, email, age)
except KeyError as e:
# Missing required field
return {"success": False, "error": f"Missing required field: {e}"}
except ValueError as e:
# Invalid data format or validation error
return {"success": False, "error": str(e)}
except DatabaseError as e:
# Database error
log_error(f"Database error during registration: {e}")
return {"success": False, "error": "Registration failed due to a system error. Please try again later."}
except Exception as e:
# Unexpected error
log_error(f"Unexpected error during registration: {e}")
return {"success": False, "error": "An unexpected error occurred. Please try again later."}
API Responses and Status Codes
@app.route('/api/users/')
def get_user_api(user_id):
try:
user = get_user_from_database(user_id)
return jsonify(user), 200
except UserNotFoundError:
return jsonify({"error": "User not found"}), 404
except DatabaseConnectionError:
return jsonify({"error": "Database connection error"}), 503
except Exception as e:
log_error(f"Unexpected error in get_user_api: {e}")
return jsonify({"error": "Internal server error"}), 500
Graceful Fallbacks in Templates
# In your Flask route
@app.route('/dashboard')
def dashboard():
try:
user_stats = get_user_statistics()
recent_activity = get_recent_activity()
notifications = get_notifications()
except DataUnavailableError:
# Fallback to empty data if not available
user_stats = {}
recent_activity = []
notifications = []
return render_template(
'dashboard.html',
user_stats=user_stats,
recent_activity=recent_activity,
notifications=notifications
)
Security Considerations
Always be careful with exception messages in production:
- Don't expose sensitive details in user-facing error messages
- Log detailed exceptions for debugging, but show users generic messages
- Be aware that exception stack traces can reveal implementation details
- Consider using different error handling in development vs. production
Debugging with Exceptions
Exceptions are not just for error handling - they're also valuable debugging tools:
Assertion Statements
def calculate_discount(price, rate):
# Verify inputs meet expectations
assert price >= 0, f"Price must be non-negative, got {price}"
assert 0 <= rate <= 1, f"Rate must be between 0 and 1, got {rate}"
return price * rate
Assertions are checked only when Python is run without the -O (optimize) flag, making them perfect for development-time checks.
Logging Exceptions
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s',
filename='app.log'
)
try:
result = complex_calculation()
except Exception as e:
# Log the exception with traceback
logging.exception("Error in complex_calculation")
# This is equivalent to:
# logging.error("Error in complex_calculation", exc_info=True)
# Handle the exception
result = fallback_value
Proper logging of exceptions provides crucial information for debugging production issues.
Debugging with the traceback Module
import traceback
try:
problematic_function()
except Exception as e:
# Print detailed traceback
traceback.print_exc()
# Or get traceback as a string for logging/display
traceback_str = traceback.format_exc()
log_error(traceback_str)
Practice Exercises
Exercise 1: File Reader
Write a function that safely reads a file, handles potential errors, and returns the content or a default value.
Solution
def safe_read_file(filename, default=""):
try:
with open(filename, 'r') as file:
return file.read()
except FileNotFoundError:
print(f"Warning: File {filename} not found.")
return default
except PermissionError:
print(f"Warning: No permission to read {filename}.")
return default
except Exception as e:
print(f"Unexpected error reading {filename}: {e}")
return default
Exercise 2: Division Calculator
Create a function that divides two numbers, handling all potential errors gracefully.
Solution
def safe_divide():
try:
num1 = float(input("Enter the numerator: "))
num2 = float(input("Enter the denominator: "))
result = num1 / num2
print(f"{num1} divided by {num2} is {result}")
return result
except ValueError:
print("Error: Please enter valid numbers.")
except ZeroDivisionError:
print("Error: Cannot divide by zero.")
except KeyboardInterrupt:
print("\nOperation cancelled by user.")
except Exception as e:
print(f"Unexpected error: {e}")
# If we get here, an exception occurred
return None
Exercise 3: Custom Exception
Create a custom exception for a banking application that validates withdrawals.
Solution
class BankingError(Exception):
"""Base exception for banking operations."""
pass
class InsufficientFundsError(BankingError):
"""Raised when a withdrawal would result in a negative balance."""
def __init__(self, account_id, requested, available):
self.account_id = account_id
self.requested = requested
self.available = available
self.deficit = requested - available
super().__init__(
f"Account {account_id} has insufficient funds: "
f"requested ${requested:.2f}, available ${available:.2f}, "
f"deficit ${self.deficit:.2f}"
)
def withdraw(account_id, amount):
# Get account balance (in a real app, this would be from a database)
balance = get_account_balance(account_id)
if amount <= 0:
raise ValueError("Withdrawal amount must be positive")
if balance < amount:
raise InsufficientFundsError(account_id, amount, balance)
# Perform withdrawal
new_balance = perform_withdrawal(account_id, amount)
return new_balance
# Usage
try:
new_balance = withdraw("ACC123456", 500.00)
print(f"Withdrawal successful. New balance: ${new_balance:.2f}")
except ValueError as e:
print(f"Invalid withdrawal: {e}")
except InsufficientFundsError as e:
print(f"Withdrawal failed: {e}")
print(f"Would you like to withdraw ${e.available:.2f} instead?")
except BankingError as e:
print(f"Banking error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
Exercise 4: Context Manager
Create a context manager that measures and logs the execution time of a block of code.
Solution
import time
import logging
from contextlib import contextmanager
# Configure logging
logging.basicConfig(level=logging.INFO)
@contextmanager
def timing(description):
start_time = time.time()
try:
yield
finally:
elapsed_time = time.time() - start_time
logging.info(f"{description}: {elapsed_time:.4f} seconds")
# Usage
with timing("Data processing"):
# Code to time
data = process_large_dataset()
with timing("API request"):
# Another operation to time
response = api.get_data()
Assignment: Robust Data Processing Pipeline
Create a robust data processing pipeline that:
- Reads data from a CSV file
- Validates and transforms the data
- Processes the transformed data
- Writes results to an output file
- Handles all potential errors gracefully
- Creates a comprehensive log of the operation
Requirements:
- Implement proper exception handling at all stages
- Create at least one custom exception type
- Use context managers for file operations
- Implement the try/except/else/finally pattern
- Include appropriate logging
- Add clear user feedback
- Handle partial success (some records processed, some failed)
Tips:
- Break down the problem into smaller functions
- Think about what can go wrong at each stage
- Consider which exceptions to handle locally vs. propagate
- Test with various error scenarios
Submission location: /week2_thursday_error_handling_assignment.py
Conclusion
Robust error handling is what separates professional applications from hobby projects. By mastering Python's exception handling, you'll create software that's more reliable, user-friendly, and maintainable.
Remember these key points:
- Always catch specific exceptions rather than using bare except
- Use the try/except/else/finally pattern appropriately
- Take advantage of context managers for resource management
- Create custom exceptions when they add clarity
- Log detailed errors for debugging but show user-friendly messages
- Handle errors at the appropriate level of abstraction
As you develop more complex applications, particularly web applications, these techniques will become an essential part of your toolkit, ensuring your software can handle the unpredictability of the real world.
"Defensive programming is about expecting the unexpected and being prepared for it."