Programs often do things that need to be undone. You might open a file, database connection or network connection that needs to be closed afterwards so that other programs can use it. You might grab a large block of memory that needs to be freed up afterwards.
It isn't just about resources. Parts of our program might need to set a particular processing context - for example it might need to change the default mathematical precision, or catch runtime warnings. These states need to be undone when you are finished, otherwise the rest of the code might not function exactly as expected.
It is possible to undo these states manually, but there are dangers to this. If the code throws an exception, the code block might exit without cleaning up properly. If the code is very complex, with multiple exit points, you might miss a case, and exit without restoring the initial state.
With statements provide a neat way of handing this. When you execute a with statement, it creates a context manager that creates the new context. When your code exits the body of the with statement, for any reason, the context manager is called again to clean up.
In this article we will look at the most common use, ensuring that files are closed correctly after use.
Manually closing files
The most commonly used system resources are probably file handles. Every time you open a file in Python, it uses grabs file handle to tell the system that it is using the file. When Python closes the file, it gives that handle back. Here is some typical code that does this:
f = open('text.txt', 'r') # Grabs a file handle s = f.read() f.close() # Gives the handle back
File handles are global resources, shared by every program running on the system, and there are only a finite number of them. Modern computers have quite a lot of file handles, but even so it is good practice to always close a file when you have finished with it.
In addition, if you fail to close a file when you have finished with it, it can cause problems if other programs need to access the same file.
Manually closing the file, similar to the code above, works most of the time. But there are cases when it can fail:
- If the code throws an exception (for example, if the file read operation failed), the code would jump straight to the exception handler and
closewould never be called. Note that this would happen with any kind of exception, not just a file exception. If there was extra code in that block that threw a divide by zero exception, it could still prevent
closebeing called, which could cause problems.
- In more complex code, there might be more than one way to exit the code block - for example there might be return statements or break statements. It can be difficult to be certain that every possible case has been handled, and the code will always call
This is where a with statement is useful.
Using with statements
In the example above, we could use a with statement, like this:
with open('text.txt', 'r') as f: s = f.read()
The with statement calls
open and assigns the returned file object to the variable
f - just like our previous open call in the original code.
When Python exits the with block for any reason the file
f will get closed. this happens if there is an exception, a return, a break, or if the code block just ends normally. In every case,
close will be called.
Notice that we don't even need to call
close explicitly in our code (we could, but it would be pointless). We just let the with block do its thing.
We can nest with blocks, for example to do a file copy:
with open('text.txt', 'r') as fin: with open('out.txt', 'w') as fout: s = fin.read() fout.write(s)
When the inner block ends, both the files will be closed. In addition, if the output file failed to open, the input file would still be closed when its with block terminated.
An alternative way to create a nested with loop is this:
with open('text.txt', 'r') as fin, open('out.txt', 'w') as fout: s = fin.read() fout.write(s)
Which form you choose is a matter of personal choice, although of course if it results in excessive line lengths that should usually be avoided.
The context manager
A with statement operates with a context manager:
with contextmanager as x: with_block
A context manager is an object that implements
__exit__ methods. The with statement calls the enter method as it enters the with code block, then calls the exit method when control leaves the with code block for any reason.
The file example works because a Python File object is a context manager. In particular, it has and exit method that closes the file.
A later article will cover other scenarios.
- List comprehensions
- Objects and variables
- Objects and identity
- Immutable objects
- Global variables
- Data types
- Lists vs tuples
- Named tuples
- Short circuit evaluation
- Walrus Operator
- For loops
- For loop using range vs iterables
- Changing the loop order
- Using enumerate in a for loop
- Using zip in a for loop
- Looping over multiple items (old article)
- Looping over selected items
- Declaring functions
- Calling functions
- Function objects and lambdas
- Function decorators
- Exception handling
- String functions
- Built-in functions
- Optimisation good practice
- Low level code optimisation
- Structural optimisation
Join the PythonInformer Newsletter
Sign up using this form to receive an email when new content is added:
2d arrays abstract data type alignment and angle animation arc array arrays bar chart bar style behavioural pattern bezier curve built-in function callable object chain circle classes clipping close closure cmyk colour combinations comparison operator comprehension context context manager conversion count creational pattern data science data types decorator design pattern device space dictionary drawing duck typing efficiency ellipse else encryption enumerate fill filter font font style for loop formula function function composition function plot functools game development generativepy tutorial generator geometry gif global variable gradient greyscale higher order function hsl html image image processing imagesurface immutable object in operator index inner function input installing iter iterable iterator itertools join l system lambda function len lerp line line plot line style linear gradient linspace list list comprehension logical operator lru_cache magic method mandelbrot mandelbrot set map marker style matplotlib monad mutability named parameter numeric python numpy object open operator optimisation optional parameter or pandas partial application path pattern permutations pie chart pil pillow polygon pong positional parameter print product programming paradigms programming techniques pure function python standard library radial gradient range recipes rectangle recursion reduce regular polygon repeat rgb rotation roundrect scaling scatter plot scipy sector segment sequence setup shape singleton slice slicing sound spirograph sprite square str stream string stroke structural pattern subpath symmetric encryption template text text metrics tinkerbell fractal transform translation transparency triangle truthy value tuple turtle unpacking user space vectorisation webserver website while loop zip zip_longest