Python 3.11: New Features

We just covered the most intriguing new features in this post; for a complete list of updates and changes, refer to the official release documentation.

The release of Python 3.11 occurred on October 24, 2022. Python's most recent version is quicker and easier to use. It has been under development for seventeen months and is now prepared for widespread usage.

Every version of Python has enhancements and changes, and 3.11 is no exception. The manual includes a list of each one. Discover the most interesting and useful new features right here.

You'll discover new features and enhancements in this course, such as:

• Improved error messages with more detailed tracebacks
• Task and exception groups make it easier to work with asynchronous code.
• Numerous new type features increase Python's static typing support.
• Native TOML support for working with configuration files.
• Faster code execution resulted from significant effort in the Faster CPython project.

Use Python 3.11 if you plan to follow any of the examples in this tutorial. How to Install a Pre-Release Version of Python and The Python 3 Installation & Setup Guide? Several alternatives for installing a new version of Python on your PC.

You'll learn more about the upcoming language updates and tips on what to think about before updating to the new version. To obtain code samples showcasing Python 3.11's new features, click the link below:

More Informative Error Tracebacks

With its accessible syntax and robust data structures, Python is frequently regarded as a suitable first programming language. Reading the Traceback that is shown when Python encounters an error can be difficult for anyone, but it can be especially difficult for people new to Python.

Python's error messages were considerably enhanced in version 3.10. The most anticipated feature of Python 3.11 will likewise improve your developer experience in a similar manner. The tracebacks have decorative annotations that can speed up error message interpretation.

What changes are included in the upcoming version?

Prolog

I've built up two docker containers to compare versions 3.10 and 3.11 differences.

First container for version 3.10:

Second container for version 3.11:

We can utilize vs small code container to connect to the two operating containers once they are both running. We can then run my Python code in each environment to compare the results.

Let's first demonstrate a code example in the following sections and then demonstrate how the two versions differ.

1. Error Location

Output in version 3.10:

1
100
Traceback (most recent call last):
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 86, in _run_code
    exec(code, run_globalss)
  File "/root//py310//myapp/__main__.py", line 4, in <module>
    print(d["key_11"])
KeyError: 'key_11'

Output in version 3.11

1
100
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/py311/myapp/__main__.py", line 4, in <module>
    print(d["key_11"])
          ~^^^^^^^^^^
KeyError: 'key_11'

An improved error location is disclosed to the developer in Python 3.11, creating a fantastic developer experience.

2. The 'self' type

The self-type was previously introduced in the module on typing extensions, and it is now elevated to the standard typing library.

The structure of a directory is depicted in the code above. The definition is recursive because directories have subdirectories.

Python 3.10 Output looks like this:

Traceback (most recent call last):
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 86, in _run_code
    exec(code, run_globalss)
  File "/root//py310//myapp/__main__.py", line 2, in <module>
    from typing import List, Tuple, Self

Python 3.11 Output looks like this:

{'content': (['a.txt', 'b.txt'],
             [{'content': (['file1', 'file2'], None), 'name': 'dir1'}]),
 'name': 'dir2'}

Note: The class name can be used to annotate the nature of the sub dir. However, all annotation references would need to be updated if the parent class's name changed.

You can import this code as shown below to make it functional in both 3.11 and 3.10:

3. Exception note

An updated __note__ class attribute on the BaseException class has a default value of None.

You can override it and add more information using any string.

Python 3.10 Output looks like this:

Traceback (most recent call last):
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 86, in _run_code
    exec(code, run_globalss)
  File "/root//py310//myapp/__main__.py", line 6, in <module>
    raise MyException("some exception")
__main__.MyException: some exception

Python 3.11 Output looks like this:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/py311/myapp/__main__.py", line 6, in <module>
    raise MyException("some exception")
    ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 
MyException: some exception
this is my note :)

Developers can now communicate exceptions more effectively thanks to the note that has been added to the output, as can be seen.

4. Groups of exception

To throw a collection of exceptions and manage them in an except clause, version 3.11 adds a new exception type called ExceptionGroup.

There is also a new except* syntax introduced...

Python 3.10 Output looks like this:

File "/usr /local /lib/ python3.10 /rrunpy.py", line 189, in _run_module_as_main
    mod_name, mod_spec, code = _get_main_module_details(_Error)
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 223, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 157, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen importlib._bootstrap_external>", line 1017, in get_code
  File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root//py310//myapp/__main__.py", line 18
    except * (ToYoungException, EmailIsInvalidException) as exception_group_1:

Python 3.11 Output looks like this:

validations failed
  + Exception Group Traceback (most recent call last):
  |   File "", line 198, in _run_module_as_main
  |   File "", line 88, in _run_code
  | ExceptionGroup:  (1 sub-exception)
  +-+ - - - - - - - - - - - - - - - - 1 - - - - - - - - - - - - - - - -
    | Exception Group Traceback (most recent call last):
    |   File "/root/py311/myapp/__main__.py", line 14, in 
    |     raise ExceptionGroup(
    |     ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 
    | ExceptionGroup: Data validations (2 sub-exceptions)
    +-+---------------- 1 ----------------
      | ToYoungException: Age must be over 18 - age is 11
      +- - - - - - - - - - - - - - - - - - - - - - - - - - - - 
2 ----------------
      | EmailIsInvalidException: Email must be valid some_wannabe_email
      +- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "/root/py311/myapp/__main__.py", line 20, in 
    |     raise ValueError from exception_group_1
    |     ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 
    | ValueError
    +- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

As we can see, it is really useful when discussing several failure reasons.

5. Nested Async Comprehensions

Python 3.10 Output looks like this:

Traceback (most recent call last):
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 189, in _run_module_as_main
    mod_name, mod_spec, code = _get_main_module_details(_Error)
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 223, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr /local /lib/ python3.10 /rrunpy.py", line 157, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen importlib._bootstrap_external>", line 1017, in get_code
  File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root//py310//myapp/__main__.py", line 11
    return { n: [x async for x in elements(n)] for n in range(3)}
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: asynchronous comprehension outside of an asynchronous function

Python 3.11 Output looks like this:

{0: [1], 1: [1, 1], 2: [1, 2, 4], 3: [1, 3, 9, 27], 4: [1, 4, 16, 64, 256]}

The code is aware of the comprehension's current "function context" when it enters a comprehension block.

Asynchronous statements cannot be used in the inner code block if the comprehension is not asynchronous.

Nevertheless, in Python 3.11, comprehensions containing inner async statements that support inner async comprehensions are automatically made asynchronous.

TOML parser

TOML processing is now included in the standard library, much like JSON and CSV.

Python 3.10 Output looks like this:

Python console:

Python 3.11 Output looks like this:

{'clients': {'data': [['gamma', 'delta'], [1, 2]], 'hosts': ['alpha', 'omega']},
 'database': {'connection_max': 5000,
              'enabled': True,
              'ports': [8000, 8001, 8002],
              'server': '192.168.1.1'},
 'owner': {'dob': datetime.datetime(1979, 5, 27, 7, 32, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600))),
           'name': 'Tom Preston-Werner'},
 'servers': {'alpha': {'dc': 'eqdc10', 'ip': '10.0.0.1'},
             'beta': {'dc': 'eqdc10', 'ip': '10.0.0.2'}},
 'title': 'TOML Example'}

7. Performance Enhancement

Python 3.11 is said to run 10-60% quicker than Python 3.10, according to some. The benchmarking area of this website has further information on it.

Performance and developer experience have both been improved with Python 3.11, which will be released soon.

8. Faster Execution of Source Code

Python is known for being a sluggish language. A normal loop, for instance, runs orders of magnitude slower in Python than it would in C. Code execution speed is frequently less significant than programmer productivity. There are numerous strategies to overcome this disadvantage.

Moreover, Python is excellent at encasing libraries created in faster languages. For instance, NumPy calculations are substantially faster than equivalent plain Python calculations. Thanks to this, Python is a strong contender in the data science field, coupled with the simplicity of writing code.

The fundamental Python language is still being worked on to make it faster. Mark Shannon proposed a number of performance enhancements that could be added to Python in the fall of 2020. The Shannon Plan, a very ambitious plan, aims to make Python five times quicker over the course of multiple releases.

Mark Shannon and Guido van Rossum, the inventor of Python, are among the programmers working on the Faster CPython project, which Microsoft has joined and is now funding. Based on the Faster CPython project, Python 3.11 has a lot of enhancements. You will discover more about the specialized adaptive interpreter in this section. Also, you'll discover quicker startup times and exceptions with no costs in later sections.

A specialized adaptable interpreter is described in PEP 659. The basic goal is to accelerate running code by optimizing frequently performed procedures. Just-in-time (JIT) compilation and this are comparable. Nonetheless, it has no impact on a compilation. Rather, Python's bytecode is modified or altered as needed.

Before being executed, Python code is converted to bytecode. Each line of Python code is split up into many bytecode statements because bytecode contains more fundamental instructions than conventional Python code.

You can use it to view the bytecode for Python. Consider a function that can convert between feet and metres as an illustration:

Python console:

Information about one bytecode instruction can be found on each line. The five columns are the line number, byte address, operation code name, operation parameters, and an interpretation of the parameters enclosed in parentheses.

To write Python, you can learn about something other than bytecode. However, it may assist you in comprehending the internal workings of Python.

The bytecode generation now includes a new step called quickening. Adaptive instructions are substituted for instructions that could be optimized at runtime by this. Each of these classes will examine how it is used and may tailor itself accordingly.

Once a function is called a certain number of times, the quickening begins. This occurs after eight invocations in CPython 3.11. You can watch the interpreter adapt bytecode by calling dis() and setting the adaptive parameter. First, a function must be defined and then called seven times with arguments of floating-point numbers:

Python console:

Next, take a look at the feet to meters() bytecode:

You won't see anything noteworthy just yet. The non-adaptive bytecode is still present in this version. This changes after the ninth call to feet to meters():

A number of the original guidelines have now been changed to more specific ones. BINARY OP has been specialized to BINARY OP MULTIPLY FLOAT to multiply two float integers more quickly.

Even though feet to meters() has been modified to be more efficient when feet are a float parameter, it continues to function normally for other parameter types by reverting to the previous bytecode instruction. Although the internal processes have changed, your code will continue to operate precisely as it did before.

The specific instructions continue to adapt. Fifty-two additional calls to your function, but this time with an integer argument:

Python console:

The two float numbers can still be multiplied by the Python interpreter. One more call to feet to meters() with an integer causes it to quit and go back to a general, adaptable instruction:

Python console:

Because one of the operations, 0.3048, is always a floating-point number, the bytecode instruction is altered to BINARY OP ADAPTIVE rather than BINARY OP MULTIPLY INT in this instance.

It is more difficult to optimize multiplications between integers of the same type than to multiply integers and floating point numbers. There currently needs to be dedicated instruction for multiplying floats and integers.

It would be best to understand how the adaptive specialized interpreter functions from this example. In general, you shouldn't be concerned about altering your current code to benefit from it. The majority of your code will execute faster as is.

There are a few situations where you might be able to restructure your code to enable more effective specialization. The Brandt Bucher specialist is a tool that illustrates how the interpreter handles your code. A manual code improvement example is provided in the tutorial. Listen to the Speak Python to Me podcast for further information.

The following principles are crucial for the Faster CPython project:

Python won't undergo any fundamental changes as a result of this research, and most code should run faster overall.

According to tests, Python 3.11 is generally 25% quicker than Python 3.10. (Source). Nevertheless, rather than how well Python 3.11 performs on benchmarks, you should be more concerned with how it performs on your code. To learn more about how to evaluate the effectiveness of your code, expand the box below:

Check your code's performance. Show/Hide

The Faster CPython project is still active, and Python 3.12, due out in October 2023, will include several optimizations. The project can be followed on GitHub. See the conversations and presentations below:

• EuroPython 2022: How We Are Making Python 3.11 Faster by Mark Shannon
• Speak Python to Me: Making Python Faster With Guido and Mark
• Brandt Bucher's Python Perf: Specialized, Adaptive Interpreter
• Pablo Galindo Salgado's "Faster CPython Project: How We're Making Python 3.11 More Quickly" presentation at PyCon ES.

A significant project that affects every aspect of Python is faster CPython. One component of the effort is the adaptive specialized interpreter. You'll discover two further optimizations later in this tutorial: quicker startup and zero-cost exceptions.

Learn these upgrades, and you'll be a better programmer!