Python Logging - A Tutorial with Examples

Logging is an important aspect of software development. This applies to Python as well as all other programming languages. It helps developers debug their code by providing information about what happened during execution, including error messages, warnings or other relevant details that can be helpful in debugging.

Additionally, logs can track the flow and state of a program at different points in time, which is particularly useful when dealing with complex systems where understanding how data flows through various components is essential.

Logging timestamps or performance counters also enables developers to analyze the performance of their code to identify bottlenecks or areas for optimization. Furthermore, logs can provide valuable information about errors that occur in production environments, helping developers identify patterns and trends in error occurrence, which can lead to more effective bug-fixing strategies.

Python Logging Module
Basic Usage
Logging Levels
Logging Exceptions
Logging Configurations
Using the SMTPHandler for critical errors
Best Practices
Conclusion

Python Logging Module

Python’s built-in logging module provides a powerful and flexible way to handle logs in your applications. It allows you to track events that happen when some software runs, making it easier to understand what is happening under the hood.

Software developers add logging calls to their code to indicate that certain events have occurred. These calls cause messages to be written to a log, which can then be used for various purposes.

Event tracking: Logging is used to track events that occur when software runs. This can include things like function calls, errors, and other important events.
Debugging assistance: Logs can be very helpful for debugging purposes. By looking at the log, a developer can see exactly what happened leading up to an error or other problem.
Auditing and monitoring: Logs can be used to keep track of who is using the software and how they are using it. This can be useful for auditing and monitoring purposes.
Error reporting: If something goes wrong in the software, the logging module can be used to generate an error report. This report can then be sent to the developer so they can fix the problem.
Information sharing: Logs can be used to share information about the software’s operation with other developers, users, or systems.

The logging module in Python offers a versatile system for creating log messages at various levels of severity, including debug, warning, info, error, and critical. It also allows you to direct log messages to various outputs, such as the console, a file, or a network connection.

This article will provide an overview of Python logging, including its basic usage, advanced features, and best practices for using it effectively.

Basic Usage

The logging module provides several functions and classes capable of reporting events that occur during normal software operation. The most common use case is to log messages to a file or the console. Here’s an example:


import logging

# Create a logger object
logger = logging.getLogger(__name__)

# Set the level of severity for the logger
logger.setLevel(logging.INFO)

# Create a handler to write log messages to a file
handler = logging.FileHandler('example.log')

# Set the formatter for the handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)

# Log some messages
logger.debug('This is a debug message')
logger.info('This is an info message')

In this example, we first create a Logger object with the name of our module (__name__). We then set its level to INFO, meaning that only messages at or above this level will be logged.

We create a FileHandler and set its formatter to include the time of creation, logger’s name, severity level, and the message itself. This handler is then added to our logger. Finally, we log some messages using the debug() and info() methods.

Logging Levels

Python defines several standard logging levels: DEBUG, INFO, WARNING, ERROR, and CRITICAL. Each level includes all levels above it in its hierarchy. For example, if you set the logger’s level to INFO, only messages at or above this level will be logged.

DEBUG: This level is used for detailed information about the program state, typically used during development or troubleshooting.
INFO: This level is used for informational messages that describe the progress or state of the program.
WARNING: This level is used to indicate a potential issue or warning that does not require immediate attention but should be monitored.
ERROR: This level is used to indicate that an error has occurred, which may cause the program to fail or produce incorrect results.
CRITICAL: This level is used to indicate a severe error that causes the program to fail immediately.

Here’s an example of how to use these logging levels in Python:


import logging

# Create a logger object
logger = logging.getLogger(__name__)

# Set the logging level
logger.setLevel(logging.DEBUG)

# Create a console handler
console_handler = logging.StreamHandler()

# Set the logging level for the console handler
console_handler.setLevel(logging.INFO)

# Create a formatter
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

# Add the formatter to the console handler
console_handler.setFormatter(formatter)

# Add the console handler to the logger
logger.addHandler(console_handler)

# Use the logger object to log messages with different levels
logger.debug('This is a debug message')
logger.info('This is an info message')
logger.warning('This is a warning message')
logger.error('This is an error message')
logger.critical('This is a critical message')

In this example, we create a logger object and set its logging level to DEBUG. We then create a console handler and set its logging level to INFO, which means that any log messages with a level of DEBUG or higher will be printed to the console. Finally, we add the formatter and console handler to the logger object and use it to log messages with different levels.

Note that you can also create custom logging levels in Python by defining a new integer value and adding it to the logging module using the addLevelName() method.

Logging Exceptions

You can log exceptions using the exception() method of a logger. This logs an error message with a stack trace:


import logging

# Create a logger object
logger = logging.getLogger(__name__)

try:
    100 / 0
except ZeroDivisionError as e:
    logger.exception('An exception occurred')

In this example, if a ZeroDivisionError is raised when trying to divide by zero, it will be caught and logged with the message "An error occurred". The logger.exception() method logs an ERROR level message along with the traceback of the exception.


import logging

# Create a logger object
logger = logging.getLogger(__name__)

try:
    # Some code that may raise a TypeError
    x = "1" + 20
except Exception as e:
     # Log the exception
    logger.exception("An error occurred")

In this example, if a TypeError is raised when trying to concatenate a string and an integer, it will be caught and logged with the message "An error occurred". The logger.exception() method logs an ERROR level message along with the traceback of the exception.

Logging Configurations

You can configure logging using a configuration file or dictionary, which is useful for deploying your application in different environments. Here’s an example of how to do this with a dictionary:


import logging.config

logging_config = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
    },
    'handlers': {
        'default': {
            'level': 'INFO',
            'formatter': 'standard',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        '': {
            'handlers': ['default'],
            'level': 'INFO',
            'propagate': False
        }
    }
}

logging.config.dictConfig(logging_config)

The given Python logging configuration, logging_config, is a dictionary that contains various parameters to configure the logging mechanism in Python. Here’s a detailed explanation of each parameter:

version: This parameter indicates the version of the logging configuration format. A value of 1 represents the basic configuration format.
disable_existing_loggers: This parameter, when set to False, ensures that any existing loggers are not disabled and can still be used. If it were set to True, all existing loggers would be disabled, and you would have to define new ones.
formatters: This parameter is a dictionary containing various formatter classes for the log records. In this case, there’s only one formatter defined, named ‘standard’. Its format string is as follows:
- %(asctime)s: The time when the log record was created.
- [%(levelname)s]: The level of the log record (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL).
- %(name)s: The name of the logger that generated this log record.
- : %(message)s: The message passed to the logger.
handlers: This parameter is a dictionary containing various handler classes for the log records. In this case, there’s only one handler defined, named ‘standard’. Its parameters are as follows:
- level: The minimum level of log records that this handler will handle. Here, it’s set to ‘INFO’, meaning any record with a level lower than INFO (i.e., DEBUG) won’t be handled by this handler.
- formatter: The formatter class associated with this handler. Here, the ‘standard’ formatter is used.
- class: The handler class that will process log records. In this case, it’s set to ‘logging.StreamHandler’, which writes log messages to streams (e.g., sys.stdout, sys.stderr).
loggers: This parameter is a dictionary containing various logger classes for the log records. Here, there’s only one logger defined, without a name, which means it will be used as the root logger. Its parameters are as follows:
- handlers: The handlers associated with this logger. In this case, the ‘standard’ handler is used.
- level: The minimum level of log records that this logger will handle. Here, it’s set to ‘INFO’, meaning any record with a level lower than INFO (i.e., DEBUG) won’t be handled by this logger.
- propagate: When set to False, this logger will not propagate its log records to the parent loggers in the hierarchy.

Using the SMTPHandler for critical errors

One of the most powerful features is its ability to send log messages via email using an SMTPHandler. This can be very useful when you want to get notified about critical errors or other important events that might occur during runtime.

Here’s a simple example:


import logging
from logging.handlers import SMTPHandler

# Create a logger object
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)

# Define the SMTPHandler
smtp_handler = SMTPHandler(
    mailhost=("smtp.example.com", 587),  # replace with your SMTP server details
    fromaddr="from@example.com",  # replace with your email address
    toaddrs=["to@example.com"],  # replace with recipient's email address
    subject="Application Error",
    credentials=("username", "password"),  # replace with SMTP username and password
)
smtp_handler.setLevel(logging.ERROR)

# Add the handler to the logger
logger.addHandler(smtp_handler)

In this example, we’re creating a SMTPHandler that will send an email whenever an error occurs in our application. The mailhost parameter is a tuple containing the SMTP server address and port number. The fromaddr parameter is the email address from which the emails are sent. The toaddrs parameter is a list of recipient email addresses.

The subject parameter is the subject line for the email, and the credentials parameter is a tuple containing the username and password for authenticating with the SMTP server.

We then add this handler to our logger object using the addHandler() method. We set the level of the handler to logging.ERROR so that it only sends emails when an error occurs.

The reason we might want to use this feature is because email notifications can be a quick and easy way to get notified about important events in your application. This can save you from having to constantly check logs or rely on other monitoring tools, which can be especially useful during production where errors may occur at any time.

Best Practices

Best practices can help you create efficient, maintainable, and informative logs that meet your application’s needs. Here are some best practices to consider when using the Python logging module:

Use named loggers: It is a good practice to create and use named loggers instead of using the root logger (logging.getLogger()). Named loggers allow you to configure them individually, set different levels, handlers, or formatters for each logger, and even disable them if needed.
Set appropriate logging levels: Choose an appropriate logging level for your application and its components. The recommended practice is to start with a higher logging level (e.g., WARNING) during development and then lower it as needed when you are ready to deploy. Remember that DEBUG-level logs can generate significant output, making it harder to find important information in the log stream.
Use meaningful logger names: Use descriptive and meaningful logger names to help identify where the log messages are coming from. This is especially useful when working with multiple modules or components in your application.
Configure handlers for specific needs: Choose appropriate handler classes based on your logging requirements, such as StreamHandler for console output, FileHandler for file-based logging, or HTTPHandler for sending log messages over HTTP. Also, consider using different formatters for different handlers to present logs in a more readable and informative way.
Use conditionals for handler configuration: Use conditional statements while configuring handlers to handle specific situations, such as logging only warnings and errors during production or enabling debug logs when a specific flag is set.
Propagate log messages: Decide whether to propagate log messages up the logger hierarchy by setting the propagate parameter in the logger configuration. Propagation allows log messages to be handled by parent loggers, which can be useful for centralized logging or when you want to handle logs from child loggers at a higher level.
Rotate and manage log files: Use log rotation techniques to manage large log files and prevent them from consuming too much disk space. The logging.handlers module provides several classes, such as RotatingFileHandler, TimedRotatingFileHandler, and WatchedFileHandler, that can help you implement log file rotation based on size or time.
Keep configuration separate: Keep the logging configuration in a separate configuration file or a dedicated module to make it easier to maintain, update, and share across your application or different applications.
Log structured data: When formatting log messages, consider using structured logs that contain key-value pairs. Structured logs can be easily parsed, filtered, and analyzed by log processing tools, making it simpler to extract valuable insights from your logs.
Test logging configuration: Test your logging configuration thoroughly during development to ensure that the correct log messages are generated and handled as expected. This includes testing different logging levels, handlers, formatters, and propagation settings.

Conclusion

Remember that logging should be done with care, as it can have performance implications if not managed properly. It’s often a good idea to use a logging framework or library that provides features like log levels (e.g., debug, info, warning, error), configurable output destinations, and other useful features.

Python’s logging module is a powerful tool that can help you understand what’s happening inside your application. By following these best practices and using its advanced features, you can make the most of logging to improve the quality and maintainability of your code.

Python Logging – A Tutorial with Examples

Table of Contents