With statements

Programs often do things that need to be undone. You might open a file, database connection or network connection that needs to be closed afterwards so that other programs can use it. You might grab a large block of memory that needs to be freed up afterwards.

It isn't just about resources. Parts of our program might need to set a particular processing context - for example it might need to change the default mathematical precision, or catch runtime warnings. These states need to be undone when you are finished, otherwise the rest of the code might not function exactly as expected.

It is possible to undo these states manually, but there are dangers to this. If the code throws an exception, the code block might exit without cleaning up properly. If the code is very complex, with multiple exit points, you might miss a case, and exit without restoring the initial state.

With statements provide a neat way of handing this. When you execute a with statement, it creates a context manager that creates the new context. When your code exits the body of the with statement, for any reason, the context manager is called again to clean up.

In this article we will look at the most common use, ensuring that files are closed correctly after use.

Manually closing files

The most commonly used system resources are probably file handles. Every time you open a file in Python, it uses grabs file handle to tell the system that it is using the file. When Python closes the file, it gives that handle back. Here is some typical code that does this:

f = open('text.txt', 'r') # Grabs a file handle
s = f.read()
f.close()                 # Gives the handle back

File handles are global resources, shared by every program running on the system, and there are only a finite number of them. Modern computers have quite a lot of file handles, but even so it is good practice to always close a file when you have finished with it.

In addition, if you fail to close a file when you have finished with it, it can cause problems if other programs need to access the same file.

Manually closing the file, similar to the code above, works most of the time. But there are cases when it can fail:

• If the code throws an exception (for example, if the file read operation failed), the code would jump straight to the exception handler and close would never be called. Note that this would happen with any kind of exception, not just a file exception. If there was extra code in that block that threw a divide by zero exception, it could still prevent close being called, which could cause problems.
• In more complex code, there might be more than one way to exit the code block - for example there might be return statements or break statements. It can be difficult to be certain that every possible case has been handled, and the code will always call close.

This is where a with statement is useful.

Using with statements

In the example above, we could use a with statement, like this:

with open('text.txt', 'r') as f:
    s = f.read()

The with statement calls open and assigns the returned file object to the variable f - just like our previous open call in the original code.

When Python exits the with block for any reason the file f will get closed. this happens if there is an exception, a return, a break, or if the code block just ends normally. In every case, close will be called.

Notice that we don't even need to call close explicitly in our code (we could, but it would be pointless). We just let the with block do its thing.

We can nest with blocks, for example to do a file copy:

with open('text.txt', 'r') as fin:
    with open('out.txt', 'w') as fout:
        s = fin.read()
        fout.write(s)

When the inner block ends, both the files will be closed. In addition, if the output file failed to open, the input file would still be closed when its with block terminated.

An alternative way to create a nested with loop is this:

with open('text.txt', 'r') as fin, open('out.txt', 'w') as fout:
        s = fin.read()
        fout.write(s)

Which form you choose is a matter of personal choice, although of course if it results in excessive line lengths that should usually be avoided.

The context manager

A with statement operates with a context manager:

with contextmanager as x:
    with_block

A context manager is an object that implements __enter__ and __exit__ methods. The with statement calls the enter method as it enters the with code block, then calls the exit method when control leaves the with code block for any reason.

The file example works because a Python File object is a context manager. In particular, it has and exit method that closes the file.

A later article will cover other scenarios.

See also

• List comprehensions
• Objects and variables
• Objects and identity
• Immutable objects
• Global variables
• Data types
• Lists vs tuples
• Sequences
• Named tuples
• Operators
• Short circuit evaluation
• Walrus Operator
• For loops
• For loop using range vs iterables
• Changing the loop order
• Using enumerate in a for loop
• Using zip in a for loop
• Looping over multiple items (old article)
• Looping over selected items
• Functions
• Declaring functions
• Calling functions
• Function objects and lambdas
• Function decorators
• Exception handling
• String functions
• Built-in functions
• Optimisation
• Optimisation good practice
• Low level code optimisation
• Structural optimisation