proposal: spec: local func declarations

[adonovan]

Background: In Go, named functions must be declared at package level. The scope of a func declaration includes the function body, so named functions may be directly or mutually recursive.

func f() { ... f() ... }

Local functions, which may have free variables, must be anonymous. Therefore they cannot call themselves recursively. Instead, we must resort to the var trick, in which a local variable is first declared, then assigned the value of an anonymous function whose body refers to the variable:

func f() {
    var g func()
    g = func() { ... g() ... }
    ... g() ...
}

This limitation has three consequences:
(1) The source code for local functions is unnecessarily verbose.
(2) The generated code for local functions is unnecessarily inefficient. Because all calls to g are superficially dynamic, the compiler generates dynamic CALL instructions and inlining is defeated. (To be clear, this is problem is entirely fixable in the compiler.)
(3) Local functions cannot be generic, because there is no way to declare type parameters. In some cases this forces a number of declarations that could otherwise all be local to have to be spilled to the top level, polluting the package namespace.

Proposal: We propose to permit func declarations as statements, so the above code could be written:

func f() {
    func g() { ... g() ... }
    ... g() ...
}

This would allow local functions to be generic:

func f() {
    func g[T any]() { ... g() ... }
    ... g() ...
}

This does create a minor parsing ambiguity between a DeclStmt of a FuncDecl, and an ExprStmt whose leftmost subtree is a FuncLit, for example:

func f() {
    func() { ... } () // immediately applied lambda
    func g() { ... } // local func declaration
}

This ambiguity is easily resolved with an extra token of lookahead: when parsing a statement, if func is followed by an identifier, it's a DeclStmt, otherwise, it's an ExprStmt.

Initially at least, we should require that references to local generic functions need explicit instantiation, to avoid the issue that appears in ML where type inference is exponential in the depth of nested lets. Perhaps that can be relaxed.

I expect the most difficult part of the implementation to be the elimination of assumptions that FuncDecls are not syntactically nested, throughout the compiler and x/tools. It is of course a breaking change to clients of go/ast.

[fzipp]

Would this have the same scoping and shadowing rules as the other three declaration keywords (const, var, type)?

func f() {
    func g(s string) {
        // different g in g
        func g(x int) { ... }
    }

    { // new block, new g
        func g() { ... }
    }
}

[fzipp]

Would this allow methods for local types?

func f() {
    type T struct{}
    func (t T) method() { ... }
}

[randall77]

Can a declaration have no body, and be implemented in assembly? If so, what is its name?

Also #71562, methods on function-local types?

[fzipp]

This ambiguity is easily resolved with an extra token of lookahead: when parsing a statement, if func is followed by an identifier, it's a DeclStmt, otherwise, it's an ExprStmt.

Re. my previous question: This probably prevents methods for local types.

[fzipp]

This would capture outer variables like a closure, right?

for n := range 10 {
	func square() int { return n * n }
	fmt.Println(square())
}

[fzipp]

Are these function definitions fixed like package-level functions and cannot be re-assigned in the same scope?

func f() {
	func g() int { return n * n }
	g = func() int { return n + n } // not possible?
}

[adonovan]

In answer to all the questions:

• same shadowing rules.
• no local methods; maybe one day, but it's tricky; see proposal: spec: allow method declarations on function-local types #71562.
• no missing body. But there's no benefit to it, since an assembly implementation can't access the lexical environment.
• definitions are fixed, just like a local type declaration.

[AndrewHarrisSPU]

I'm in agreement that there's some real utility on the table with more efficient code generation, easier recursion, and type parameterization of a function scoped within a function. Also, just a preference for local functions as @adonovan suggests in #71562 (comment) is IMHO very reasonable in Go. However, the more I thought about comments like #33167 (comment) the more I thought I don't have a complete grasp on how many small changes are needed for this, or where, or what the limitations will necessarily be. So I'm not opposed to this, I think it's worth attention, I just wonder about the total impact.

Another syntax change would be to allow const for function variables. This would not require the lookahead tweak for resolving Decl / Expr ambiguity.

func f() {
    const g[T any]() := { ... g()[T] ... }
    ... g()[int] ...
}

Maybe the compiler could identify the code generation opportunity here (I think that shouldn't be in the spec, just implementation)? I could hand-wave an intuitive argument about why g()[int] requires an explicit type parameter. It might be more immediately intuitive why mutual recursive arrangements won't fall out trivially.