The Python Steering Council intends to remove the Global Interpreter Lock (GIL) from Python, speeding concurrent code, but will be careful with backward compatibility to avoid “another Python 3 situation.”

In January this year, Meta research engineer Sam Gross, who is co-author of the popular PyTorch machine learning framework, proposed PEP (Python Enhancement Proposal) 703, entitled “Making the Global Interpreter Lock Optional in CPython.” The Steering Council has now stated that it intends to accept the proposal, following a poll of the core development team with two questions: were members in favor of making Python free-threaded, and were they willing to accept the cost “in terms of complexity and maintenance.” The votes were 87% in support of the first, and 63% in support of the second, despite the comment in the poll that “We don’t know how much code implicitly relies on the GIL, or is thread-unsafe in subtle ways that are masked by the GIL. We don’t know how much work it’s going to be to deal with the fallout of introducing free-threading.”

The Steering Council though is wary of the implications and keen to avoid a repeat of the painful progression from Python 2 to Python 3, where lack of compatibility delayed the upgrade for many users. “This is not Python 4,” said the post from core developer Thomas Wouters, posting for the whole Steering Council. 

The intent is to progress the proposal in three stages. The first is to add no-GIL as an experimental build, to enable the community to try it. Next, the no-GIL build would become supported, but not the default, for a period of “at least a year or two, possibly more.” Third, the teams wants no-GIL to be the default and “to remove any vestiges of the GIL”. There is no enthusiasm for having both GIL and no-GIL builds supported long-term, with the double-testing that implies, but this third stage is “probably 5+ years” away.

There remains uncertainty. “We want to be able to change our mind if it turns out, any time before we make no-GIL the default, that it’s just going to be too disruptive for too little gain. Such a decision could mean rolling back all of the work,” acknowledged the Steering Council, adding later in the post that “we want to be able to call off PEP 703 and find another solution if it looks to become problematic.”

Gross has been promoting the notion of a no-GIL CPython for some time, and spoke on the subject at the EuroPython event in July 2022, where he said he had been working on the project for the “past few years.” The GIL, he said, “ensures that only one thread can execute Python code at a time.” It was introduced in 1992 to enable multi-threaded coding in Python subject to the constraints of the GIL, which made such code safer and easier to implement. However as both CPUs and operating systems advanced their support of multi-threading, the GIL became an increasing obstacle to performance. Gross developed a proof-of-concept no-GIL CPython and published it on GitHub.

In PEP 703 Gross cites several reasons for removing the GIL, including that it makes many types of parallelism difficult to express, that it affects Python library usability, that GPU-heavy workloads require multi-core processing, and that the GIL makes deploying Python AI models difficult.

The rationale is compelling; but the compatibility issues are also real, and it seems that the Steering Council is keenly aware of both aspects.