Python 3.9 lands with union operators and type hinting generics

Python 3.9 lands with union operators and type hinting generics

After pulling support for the freshly retired Python 3.5, the Python team has concluded its work on version 3.9 of the programming language and made the result generally available.

The new release includes quite a few noteworthy features, though most users will surely be most interested in the syntax additions, such as operators | and |= for merging and updating dictionaries. According to language creator Guido van Rossum, the correct way of combining two dicts wasn’t something a typical Python user – let alone he himself – would have thought of right away, and the hope is that the new operators are more intuitive.

Better usability is also the rationale behind newly added type hinting generics in standard collections. Through it, developers can use built-in collection types such as list and dict “as generic types instead of importing the corresponding capitalized types [..] from typing”. This is meant to make programs easier to annotate and, with the duplicated collection hierarchy in the typing module gone, the language logic easier to comprehend.

The relaxation of grammar restrictions on decorators should help the cause further. Decorators can now be any valid expression instead of following the pattern of dotted name followed by a single call. Python 3.9 comes with new string methods str.removeprefix(prefix) and str.removesuffix(suffix) to remove an unneeded prefix or a suffix, and includes an Annotated type in the typing module for decorating types with context-specific metadata.

Another much highlighted feature of the current release is a new CPython parser. The latter uses a parsing expression grammar (PEG) instead of the left-to-right (LL(1)) top-down approach the previous parser followed. LL(1) “parses the input from left to right, performing leftmost derivation of the sentence, with just one token of lookahead”, which – though efficient to implement – made some common Python constructs hard to express and led to some ambiguity.

The new PEG-based parser meanwhile uses a grammar reflecting the way the parser will operate more closely and only knows one valid parse-tree per string. On the downside, it uses more memory, but given the reduction of maintenance costs and flexibility for new features, the new parser promises the trade-off was something the Python team was willing to accept.

Developers who don’t want to use the new parser yet can get the old one back by using -X oldparser on the command line. It will, however, be deleted in Python 3.10, so it’s probably best to get comfortable with the PEG parser sooner rather than later. 

A full list of changes can be found in the project’s release notes. Users should be aware that Python 3.9 is the last version to provide some Python 2 backward-compatibility layers, which is why the development team recommends you check for deprecation warnings and update code before everything will be removed in Python 3.10. 

The fact that some of these warnings have been around since 2012 shows that Python users usually get a lot of time for transitioning to a new version. A few days before 3.9 was released, the Python team announced the end of support for version 3.5 and recommended an upgrade to the latest iteration of the language. Python 3.5 was first released in 2015, 3.5.10 is the last official version of the series. 

To help developers plan their code migration efforts better, the Python team also decided on a new fixed release cadence with yearly updates. Following that, the next sort of major version, Python 3.10, is planned to land in October 2021.

A certain excitement about the new version seems to have helped Python’s ranking in the October edition of the TIOBE Index, which sees the language “getting closer to the second position”. TIOBE Software CEO Paul Jansen wrote on the topic: “Since Java is approaching its all time low since 2001, Python and Java are getting pretty close. The gap is now less than 1.3%. Since the start of the TIOBE index, C and Java have always held the 2 top positions. So it would be a unique event, if Python would reach position #2.”

The TIOBE Programming Community Index is calculated by counting the hits for the search query +”<language> programming” in 25 search engines and is supposed to give some sort of indication for a language’s popularity. New release or major bugs tend to have quite an impact on this metric, which for one means the unique event Jansen mentions could happen soon but also has to be taken with a grain of salt.

Python’s popularity is nevertheless widely recognised with the language regularly clocking in amongst the Top 5 in all sorts of rankings from StackOverflow’s Developer Survey to the PYPL Popularity of Programming Language Index to the RedMonk Programming Language Rankings.