At the heart of Python’s iteration capabilities lies the Iterator Protocol, a formalized contract that any object can adhere to in order to be used in a for loop or with functions like next(). This protocol is elegantly simple, requiring just two methods: __iter__() and __next__(). Understanding this protocol is fundamental to grasping not only custom iteration but also how generators, which simplify its implementation, work under the hood.

An iterable is any object capable of returning an iterator. It implements the __iter__() method, which is called implicitly by the for loop or the iter() function. An iterator is the object that actually performs the iteration; it implements both __iter__() (returning itself) and __next__() (returning the next value). This distinction is crucial: an iterable is a container of items, while an iterator is the cursor that traverses those items.

The __iter__ Method

The __iter__ method’s primary responsibility is to return an iterator object. For an object that is its own iterator, this method simply returns self. This is the common pattern for iterators. For an iterable that is not its own iterator (like a list, which can have multiple independent iterators), __iter__ must return a new instance of an iterator object. This ensures that each for loop gets a fresh starting point.

class CountUpTo:
    """An iterable that is not its own iterator."""
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        # Returns a *new* iterator instance
        return CountUpToIterator(self.n)

class CountUpToIterator:
    """The iterator for CountUpTo."""
    def __init__(self, n):
        self.current = 1
        self.n = n

    def __iter__(self):
        return self  # Iterators return themselves

    def __next__(self):
        if self.current > self.n:
            raise StopIteration
        result = self.current
        self.current += 1
        return result

# Usage
my_iterable = CountUpTo(3)
for num in my_iterable: # First for loop calls __iter__(), gets a new iterator
    print(num) # Outputs 1, 2, 3
for num in my_iterable: # Second for loop calls __iter__() again, gets another new iterator
    print(num) # Outputs 1, 2, 3 again

The __next__ Method

The __next__ method is the workhorse of the protocol. It is responsible for returning the “next” value in the sequence each time it is called. When there are no more items to return, it must raise the StopIteration exception. This is not an error; it’s the designated signal to the looping construct that iteration is complete. The for loop catches this exception silently and terminates. Forgetting to raise StopIteration will result in an infinite iterator, a common pitfall.

class InfiniteRange:
    """A simple iterator that is also its own iterable."""
    def __init__(self, start=0):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        value = self.current
        self.current += 1
        return value  # This will never raise StopIteration

# WARNING: This will create an infinite loop!
# for number in InfiniteRange():
#     print(number)

The Iteration Process in Detail

When you write for item in my_object:, Python performs a precise series of steps:

  1. It calls iter(my_object), which in turn calls my_object.__iter__().
  2. The __iter__ method returns an iterator object.
  3. It repeatedly calls next() on this iterator object, which calls the iterator’s __next__() method.
  4. Each value returned by __next__() is assigned to item and the loop body executes.
  5. When __next__() raises StopIteration, the loop exits gracefully.

Best Practices and Common Pitfalls

A key best practice is to separate the iterable from the iterator when you need to support multiple independent iterations over the same data. If an object is its own iterator, after the first for loop exhausts it, subsequent loops will find it already in a “completed” state (StopIteration will be raised immediately). This is why list is an iterable that returns a new iterator (list_iterator object), while a zip object is a single-use iterator itself.

Another critical pitfall is modifying a mutable iterable during iteration. If you alter a list’s length (by adding or removing items) while iterating over it, you will likely encounter unexpected behavior or a RuntimeError. The solution is to iterate over a copy of the list (e.g., for item in my_list[:]:) or to create a new list altogether.

my_list = [1, 2, 3, 4]
# This can cause problems
for item in my_list:
    if item % 2 == 0:
        my_list.remove(item) # Modifying the list being iterated upon
print(my_list) # Output might be [1, 3] but behavior is undefined.

# Correct approach: iterate over a copy
my_list = [1, 2, 3, 4]
for item in my_list[:]: # Creates a shallow copy for iteration
    if item % 2 == 0:
        my_list.remove(item)
print(my_list) # Output is consistently [1, 3]

The iterator protocol provides the powerful, flexible foundation for all iteration in Python. While implementing it manually is educational, in practice, generators provide a far more concise and readable way to create iterator objects, abstracting away the explicit management of state and the StopIteration exception.