Right, so you’ve graduated from print() statements. Good for you. Now let’s talk about doing it properly. Logging to the console is fine for a quick script, but for anything that runs longer than five minutes, you need persistence. You need logs that survive a reboot, that you can grep through at 2 AM when things are on fire, and that don’t fill up your disk and bring the whole operation to a grinding halt. Let’s get into it.

Logging to a File: Your First Step to Sanity

The absolute basics. You’re not just dumping messages into the void anymore; you’re writing them to a file. This is logging 101. The logging.basicConfig() function is your friend here, but we’re going to use it properly, not just for the console.

import logging

# This sets up the root logger. It's a one-time config, usually done at the start.
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler("my_app.log"),  # The important part
        logging.StreamHandler()  # Also log to the console, because why not?
    ]
)

logger = logging.getLogger(__name__)

def main():
    logger.info("Application started. Let's get this bread.")
    # ... your amazing code ...
    try:
        1 / 0
    except ZeroDivisionError:
        logger.error("Well, that was predictably disastrous.", exc_info=True)

if __name__ == "__main__":
    main()

Why this is better: The FileHandler does the heavy lifting. The exc_info=True argument in the logger.error() call is a lifesaver—it automatically captures and logs the full stack trace. You’re no longer guessing what went wrong. Notice the format includes the timestamp (asctime) and the logger’s name (name), which is the __name__ of your module. This is crucial when you have a multi-module application, as it lets you trace a message back to its source.

The Perils of a Single, Gigantic Log File

So now you have my_app.log. It grows. And grows. And grows. You tail -f the thing and it’s like watching the Matrix code waterfall. Eventually, it consumes all available disk space, your application crashes, and you’re left with a 500GB log file that Notepad++ will definitely not open. This is where the brilliant concept of log rotation comes in.

RotatingFileHandler: Because Your Disk Space Isn’t Infinite

The logging.handlers module has your back. RotatingFileHandler is the classic workhorse. It lets a log file grow to a specified maximum size, then closes it and renames it (e.g., adds a .1, .2, etc.), starting a fresh new log file. You also tell it how many of these backups to keep before it starts deleting the oldest ones.

import logging
from logging.handlers import RotatingFileHandler

# Create a dedicated logger for this module, not the root logger.
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# Create the rotating handler. Let's get specific.
handler = RotatingFileHandler(
    filename='my_app.log',
    maxBytes=5 * 1024 * 1024,  # 5 Megabytes. Not gigabyte, *megabyte*.
    backupCount=3  # Keep my_app.log.1, .log.2, and .log.3. Then start over.
)
handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))

logger.addHandler(handler)

# Now just use your logger as normal.
for i in range(10000):
    logger.info("This is message number %d. How thrilling.", i)

Run that. Watch as it creates my_app.log, then my_app.log.1, then .log.2, and so on, cycling through your backups. It’s beautiful. It means you can be as chatty as you want in your logs without the fear of a storage-pocalypse.

Pitfall Alert: There’s a tiny, almost absurd race condition on POSIX systems (Linux, macOS). If multiple processes are logging to the same file with a RotatingFileHandler, you’re gonna have a bad time. The rotation logic is process-specific, not system-wide. For multi-process apps (e.g., with Gunicorn), you’ll want to use WatchedFileHandler or a dedicated logging service that handles this for you.

TimedRotatingFileHandler: For When Time is More Important Than Size

Sometimes, you want a new log file every day at midnight, not when it hits a certain size. Enter TimedRotatingFileHandler. It’s the same idea, but based on time intervals.

from logging.handlers import TimedRotatingFileHandler
import logging

logger = logging.getLogger(__name__)
handler = TimedRotatingFileHandler(
    filename='timed_app.log',
    when='midnight',  # The magic keyword. Also try 'H' (hourly), 'W0' (weekly on Monday)
    interval=1,
    backupCount=7  # Keep a week's worth of logs
)
logger.addHandler(handler)
logger.info("This log entry is dated. Very fancy.")

The when parameter is brilliantly powerful and slightly cryptic. 'midnight' is obvious. 'H' is for hourly. 'W0' rolls over on Monday (W1 is Tuesday, etc.). Check the docs for this one, but it’s incredibly useful for creating daily logs you can archive or analyze.

Shipping Logs to External Services: Because grep is So 2004

For a serious production system, you don’t want to be SSHing into machines to read text files. You want your logs aggregated, searchable, and alertable in a central place. This is where services like Datadog, Loggly, Sentry, and Elasticsearch (the ELK stack) come in.

You could write a custom handler that sends log messages via an HTTP API. But you don’t have to, because someone else already did. This is a case where you absolutely should use a well-maintained library.

For example, to send your logs to Datadog:

pip install datadog

from ddtrace import patch_all; patch_all(logging=True)  # Patches logging to auto-integrate
import logging
from datadog_logger import DatadogLogHandler

logger = logging.getLogger(__name__)
dd_handler = DatadogLogHandler(
    api_key='your_api_key_here',
    service='my-python-app'
)
logger.addHandler(dd_handler)

logger.info("This message is now in the cloud. Look at us, we're modern.")

The Big Caveat: Network I/O is slow and unreliable compared to writing to a local file. A naive implementation can block your main application thread every time you log a message. The best practice is to use a queuing or asynchronous handler. Many external logging libraries offer a handler that buffers log messages and sends them in batches on a background thread. Always, always look for this feature. If you’re rolling your own, use a QueueHandler and QueueListener as outlined in the Python docs. It’s more setup, but it prevents your logging from becoming the reason your app is slow. Never let your observability tools become the source of the problem they’re meant to solve.