Like C# before it, breaking change in Go may fix ‘design decision that makes programs incorrect’

The Go team has posted about a problematic language feature which, says Google Distinguished Engineer Russ Cox, is “the only design decision I know of in Go that makes programs incorrect more often than it makes them correct”.

The current version of Go is 1.19 (though version 2.0 is in planning), and the key question is whether it is better to fix the feature and break existing code, or to retain it and live with continued surprising behaviour.

The issue in question is loop variables. In the following code:

var all []*Item

for _, item := range items {

              all = append(all, &item)

}

The “all” array ends up being an array of identical objects, not an array of all the different objects in the source collection. All the values will be that of the last object assigned to the ”item“ variable, because this is a single variable whose value changes, not a new variable for each iteration. This is unlikely to be what the developer intended. The problem is usually fixed, Cox explains, by adding an assignment to a new variable within the loop:

var all []*Item

for _, item := range items {

              item := item

              all = append(all, &item)

}

The Go team would like to change this behavior, especially since it impacts closures that have implicit variables, but “we gave the general rule that language redefinitions like what I just described are not permitted,” says Cox. Nevertheless, he reckons the case is strong enough that he thinks it right to “motivate a one-time exception.”

How bad is it? “I suspect every Go programmer in the world has made this mistake in one program or another. I certainly have done it repeatedly over the past decade, despite being the one who argued for the current semantics and then implemented them. (Sorry!),” says Cox, adding that “the current cures for this problem are worse than the disease.”

One mitigating factor is that Go modules have go.mod file which includes the minimum version of Go required by a module. Cox proposes that the language semantics could change based on the version. “If we hypothetically made the change in go 1.30, then modules that say “go 1.30” or later get the per-iteration variables, while modules with earlier versions get the per-loop variables,” he says.

Microsoft’s C# language had the same problem, up until C# 5.0 which was released in 2012. The reason for the original mistake was the same: it copied the behavior of the for loop in C, but in C it is not such a problem. A member of the C# team, Jared Parsons, popped up to comment to Cox’s post, saying: “The C# 5 rollout unconditionally changed the foreach loop variable to be per iteration. At the time of C# 5 there was no equivalent to Go’s putting go 1.30 in a go.mod file so the only choice was break unconditionally or live with the behavior.” It was worse for C#, because at the time there was no equivalent to go.mod, though there is now an <LangVersion> property.

Changing something as fundamental as loop variable scope seems risky, and it is, but not as much as it first appears. Eric Lippert, then at Microsoft but now at Facebook, said at the time of the C# change, “Any developers who depend on this feature, who require the closed-over variable to contain the last value of the loop variable, would be broken. I can only hope that the number of such people is vanishingly small; this is a strange thing to depend on. Most of the time, people do not expect or depend on this behavior.”

In other words, where behavior changes because of the breaking change, it was most likely a bug in the first place. Cox confirmed this in tests run by Google on Go code. Using a set of 100,000 tests, “there were only 58 failures,” he says, and “we found only 2 instances out of the 58 where code correctly depended on per-loop semantics and was actually broken by the change.” The exercise in fact exposed numerous “previously undiagnosed actual bugs.”

That should be enough to ensure that that change does happen, and most in the community support it. “I am very in favor of this change even considering the breaking aspect of it,” said one Go developer, and another, “I like this change and think it would eliminate one of the biggest footguns in Go.”