What's new in Python 3.11

What’s new in Python 3.11
Henry Schreiner - Princeton RSE meeting
https://iscinumpy.dev/post/python-37
And older versions
And packaging quick-start
10-19-2022

Modern Python Packaging
with Hatchling
# contents of pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "package"
version = "0.1.0"
0 to working package

More realistic project table
Still static
and readable!
[project]
name = "example_package_YOUR_USERNAME_HERE"
version = "0.0.1"
authors = [
{ name="Example Author", email="author@example.com" },
]
description = "A small example package"
readme = "README.md"
requires-python = ">=3.7"
classifiers = [
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]
[project.urls]
"Homepage" = "https://github.com/pypa/sampleproject"
"Bug Tracker" = "https://github.com/pypa/sampleproject/issues"

Packaging recommendations
https://packaging.python.org/en/latest/tutorials/packaging-projects/
https://scikit-hep.org/developer
https://iscinumpy.dev

Quick overview of Python 3.6
Path protocol
__fspath__
Class keywords much easier
__init_subclass__
Descriptor protocol: names
__set_name__
Underscores in numeric literals
1_000_000
f-strings
f"Value {here!r}”
Variable annotations
timestamp: int
Past EoL!

Future annotations
from __future__ import annotations
Module methods
__dir__, __getattr__
Core typing support
__class_getitem__
__mro_entries__
__orig_bases__
Breakpoint
breakpoint()
dataclasses
contextvars
importlib.resources
Nanosecond time functions
Extra args for subprocess.run
Asyncio basically usable, finally
Type annotations in functools.singledispatch
dict officially required to be ordered
Thread-local storage C-API
async/await full keywords
import typing 7x faster
-x dev mode
-x importtime mode
255+ arguments now allowed
Deprecation warnings improved
Oldest supported
Python

Walrus operator
while x := get_line():
Positional only arguments
def f(x, ):
F-strings for debugging
f"{x = }"
Typing enhancements
Literal, Final,
Protocol, TypedDict
Typed AST
importlib.metadata
dict __slots__ for docs
math & statitistics additions
Better SyntaxErrors sometimes
multiprocessing.shared_memory
reversed(dict(…))
Window DLL loading changes
Single ABI debug & release
Runtime audit hooks
New vectorcall API
Pickle protocol 5 (out of band)
__code__.replace(…)
NEP29 / SPEC 0
Oldest scientific Python
First yearly release!

Dict merging operator
{"a": 1} | {"b": 2}
Runtime generics
list[int]()
zoneinfo
graphlib (topological sort)
ast.unparse
asyncio.to_thread
os.pidfd_open() (race/signal free)
typing.Annnotated
str.removeprefix / str.removesuffix
concurrent.futures cancel_future option
sys.platlibdir
__file__ is always absolute
New default parser (PEG)
Any expression can be a decorator

First release with major new syntax since 3.6!
(First new block since 2.5 & 2.6 added with!)
First release with a release logo :)
Used by Pyodide today!

Structural Pattern Matching
match value:
case {"x": x, "y": y}:
print(f"{x}, {y}")
case x, y:
print(f"{x}, {y}")
case Point(x=x, y=y):
print(f"{x}, {y}")
Better errors
Did you mean?
Unclosed parens
Syntax errors & indentation
Precise line numbers
Dataclasess: kw_only & slots
Typing: adding a parameter
Parenthesized context managers
zip(…, strict=True)
aiter() / anext()
itertools.pairwise()
inspect.get_annotations()
Type aliases
Functions cache .__builtins__
Warning for encoding missing
sys.stdlib_module_names
sys.orig_argv
__ipow__ works like other __i*__
int.bit_count()
Pathlib Windows .resolve() bug fixed!
Typing Unions
int | None

tomllib
AsyncIO
WebAssembly
Typing Errors
Faster
3.11
Releases
Oct 24, 2022!
Final
release candidate
available now!
Textual CSS
releases this
day, too!

Faster CPython Project
5%-60% faster - 25% on average!
Value
Axis
0
0.25
0.5
0.75
1
Python 3.8
Raytracing example, from speed.python.org
*: Python master Sep 8, 2022
(may include some 3.12 speedups)
Python 3.8 Python 3.9
Python 3.10
Python 3.11*
3.11
Zero cost exceptions (if not thrown)
10% faster re & atomic grouping, possessive qualifiers
10-15% faster startup
Faster function calls
C-style formatting sometimes as fast as f-string
Less memory for string keys in dicts
Specializing adaptive interpreter
And more!
Future
Major focus for the next several releases
Simple JIT planned eventually
The main driver behind C API changes

Specializing Adaptive Interpreter
Can be visualized with the specialist tool!
After running the same code a few times, Python specializes bytecodes for some operations!
def f(x):
return x*x
r = list(range(100))
for y in r:
f(y)

>>> dis.dis(f)
1 0 RESUME 0
2 2 LOAD_FAST 0 (x)
4 LOAD_FAST 0 (x)
6 BINARY_OP 5 (*)
10 RETURN_VALUE
def f(x):
return x*x
for y in r:
f(y)

>>> dis.dis(f)
1 0 RESUME 0
2 2 LOAD_FAST 0 (x)
4 LOAD_FAST 0 (x)
6 BINARY_OP 5 (*)
10 RETURN_VALUE
>>> dis.dis(f, adaptive=True)
1 0 RESUME_QUICK 0
2 2 LOAD_FAST__LOAD_FAST 0 (x)
4 LOAD_FAST 0 (x)
6 BINARY_OP_MULTIPLY_INT 5 (*)
10 RETURN_VALUE
def f(x):
return x*x
for y in r:
f(y)

Hot code identified - “quickened” after multiple runs
Adaptive instructions replace original (slightly slower)
Adaptive instructions can adapt to fast versions
>>> dis.dis(f, adaptive=True)
1 0 RESUME_QUICK 0
2 2 LOAD_FAST__LOAD_CONST 0 (x)
4 LOAD_CONST 1 (2)
6 BINARY_OP_MULTIPLY_INT 5 (*)
10 LOAD_CONST 2 (2.0)
12 BINARY_OP_ADAPTIVE 0 (+)
16 RETURN_VALUE
def f(x):
return x*2 + 2.0
Quickened (ran multiple times)
Fast load of const
Fast int - int multiply
Slow adaptive (no int - float)

Better Error messages
>>> obj = {"a": {"b": None}}
>>> obj["a"]["b"]["c"]["d"]
Traceback (most recent call last):
File "tmp.py", line 3, in <module>
obj["a"]["b"]["c"]["d"]
Before 3.11
TypeError: 'NoneType' object is not subscriptable

Better Error messages
>>> obj = {"a": {"b": None}}
>>> obj["a"]["b"]["c"]["d"]
obj["a"]["b"]["c"]["d"]
After 3.11
TypeError: 'NoneType' object is not subscriptable
~~~~~~~~~~~~~^^^^^

Exception notes
try:
import skbuild
except ModuleNotFoundError as err:
err.add_note("Please pip or conda install 'scikit-build', or upgrade pip.")
raise
import skbuild
^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'skbuild'
Please pip or conda install 'scikit-build', or upgrade pip.
Note!
Also
sys.exception()

Exception groups
try:
raise ExceptionGroup(
"multiexcept",
[TypeError(1), KeyError("two")],
)
except* TypeError as err:
print("Got", *(repr(r) for r in err.exceptions))
except* KeyError as err:
print("Got", *(repr(r) for r in err.exceptions))
Got TypeError(1)
Got KeyError('two')
Backported for 3.7+ as
pip install exceptiongroups
(minus the new syntax)

Example: converter
import dataclasses
@dataclasses.dataclass
class Config:
count: int
time: float
info: str
data = {"count": "12x", "time": "1.2y", "info": "one point two"}
conv = {
item.name: item.type(data[item.name])
for item in dataclasses.fields(Config)
}
res = Config(**conv)
print(res)

Example: single error
File "…/tmp.py", line 13, in <module>
conv = {item.name: item.type(data[item.name]) for item in dataclasses.fields(Config)}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "…/tmp.py", line 13, in <dictcomp>
conv = {item.name: item.type(data[item.name]) for item in dataclasses.fields(Config)}
^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not convert string to int: '12x'
But what about my float error? I have to find and fix each one.

Example: ExceptionGroups
conv = {}
errs = []
for item in dataclasses.fields(Config):
try:
conv[item.name] = item.type(data[item.name])
except Exception as err:
err.add_note(f"Could not convert {item.name!r}: {data[item.name]!r}")
errs.append(err)
if errs:
raise ExceptionGroup("Converting failed", errs)

Example: output
+ Exception Group Traceback (most recent call last):
| File "…/tmp2.py", line 23, in <module>
| raise ExceptionGroup("Converting failed", errs)
| ExceptionGroup: Converting failed (2 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| conv[item.name] = item.type(data[item.name])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
| ValueError: invalid literal for int() with base 10: '12x'
| Could not convert 'count': '12x'
+---------------- 2 ----------------
| conv[item.name] = item.type(data[item.name])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
| ValueError: could not convert string to float: '1.2y'
| Could not convert 'time': '1.2y'
+------------------------------------

Already used by cattrs!
+-+---------------- 1 ----------------
| Exception Group Traceback (most recent call last):
| File "<cattrs generated structure scikit_build_core.file_api.model.index.Index>", line 15, in structure_Index
| res['reply'] = __c_structure_reply(o['reply'], __c_type_reply)
| File "<cattrs generated structure scikit_build_core.file_api.model.index.Reply>", line 24, in structure_Reply
| if errors: raise __c_cve('While structuring Reply', errors, __cl)
| cattrs.errors.ClassValidationError: While structuring Reply (2 sub-exceptions)
| Structuring class Index @ attribute reply
+-+---------------- 1 ----------------
| res['cache_v2'] = __c_structure_cache_v2(o['cache_v2'], __c_type_cache_v2)
| KeyError: 'cache_v2'
| Structuring class Reply @ attribute cache_v2
+---------------- 2 ----------------
| res['cmakefiles_v1'] = __c_structure_cmakefiles_v1(o['cmakefiles_v1'], __c_type_cmakefiles_v1)
| KeyError: 'cmakefiles_v1'
| Structuring class Reply @ attribute cmakefiles_v1
All conversion errors are collected and shown at once!
(This example is using the backport and Python 3.7-3.10)
← Note!
← Note!

AsyncIO: TaskGroups
from rich.progress import Progress
import asyncio
async def lots_of_work(n: int, progress: Progress) -> None:
for i in progress.track(range(n), description=f"[red]Computing {n}..."):
await asyncio.sleep(0.05)
async def main():
with Progress() as progress:
async with asyncio.TaskGroup() as g:
g.create_task(lots_of_work(120, progress))
g.create_task(lots_of_work(90, progress))
asyncio.run(main())
Enabled by ExceptionGroups!

Static Typing
Variadic Generics
Great for ND arrays
Array[float, width, height]
Self type
Works for class methods too!
def unit(self) -> Self:
LiteralString
SQL Injection protection
f-strings propagate literalness
Exhaustiveness checking
Perfect for pattern matching
typing.assert_never(x)
Dataclass transforms (Not)Required for TypedDict typing.reveal_type()
Generic TypedDict/NamedTuple Any is subclassable typing.assert_type()
And more!

Tomllib / WebAssembly
Reading TOML files in the stdlib!
Identical to tomli
Still use tomli-w to write TOML
Run Python in your browser!
Currently support level tier 3
Tier 2 (binaries) planned for 3.12!
Will also make Pyodide -> PyScript easier to support
(and the webassembly forge project, etc)

Other things
contextlib.chdir()
PYTHONSAFEPATH
operator.call()
functools.single_dispatch supports Unions
More Python 2 compat removals
* unpacking inside for
PyBuffer added to Limited API / Stable ABI
New deprecations of rarely used modules

Support
Python is ABI stable RC 1
cibuildwheel optional support in betas
Full default support started RC 1
NumPy shipped binaries during RC 1
(due to now using cibuildwheel)
SciPy might release binaries in a (few) week(s)
(due to moving to cibuildwheel)
AwkwardArray shipped binaries during RC 1
(due to cibuildwheel + numpy)
Matplotlib shipped binaries during RC 2
(due to using cibuildwheel)
GitHub Actions has been shipping dev versions
We are often testing on 3.11-dev!
Many of these libraries have never shipped wheels before a Python release!

Support
Python is ABI stable RC 1
cibuildwheel optional support in betas
Full default support started RC 1
NumPy shipped binaries during RC 1
(due to now using cibuildwheel)
SciPy might release binaries in a (few) week(s)
(due to moving to cibuildwheel)
AwkwardArray shipped binaries during RC 1
(due to cibuildwheel + numpy)
Matplotlib shipped binaries during RC 2
(due to using cibuildwheel)
GitHub Actions has been shipping dev versions
We are often testing on 3.11-dev!
Many of these libraries have never shipped wheels before a Python release!
More of the ecosystem will be ready on release day than ever before!

The future
3.12 will remove distutils
Avoid direct usage of distutils
Consider a modern unaffected builder, like Hatchling
Watch the scikit-build project for compiling!
3.15 will make unicode default!
Always specify an encoding with any open function
You can use “native” encoding in 3.11+ (but, why?)
open(…, encoding="utf-8")
Path(…).read_txt(encoding="utf-8")
3.12 faster CPython plans
Trace optimizer (mutli-bytecode ops)
Multithreaded parallelism (sub-interpreters)
More specialization bytecodes
Smaller PyObject
Regular memory
JIT plans
Probably 3.13 or 3.14
https://github.com/faster-cpython/ideas/wiki/Python-3.12-Goals

What's new in Python 3.11

More Related Content

Similar to What's new in Python 3.11 (20)

More from Henry Schreiner (20)

Recently uploaded (20)

What's new in Python 3.11