Constant Optimization

[mferguson9215]

Hello, I'm wondering if there are any plans to add constant optimization during the training process. If you do not plan to do so, do you have any suggestions about how I might convert a sympy model into an evogp tree object? Although it's not the most optimal implementation and will impact training time, I'm interested in experimenting with a constant optimization procedure like this:

1. Intermittently pause training.
2. Take sympy representations of the functions in the algorithm's forest.
3. Optimize the functions using something like scipy or optax.
4. Convert optimized functions back into sympy representations.
5. Update the trees in the forest of functions.
6. Resume the training process.

[Kelvin-Doremi]

Thank you for your valuable suggestion.

Your idea of intermittently pausing training, extracting the Sympy expressions, optimizing constants with an external optimizer (like scipy or optax), and then converting them back into EvoGP trees is very inspiring. Although it may increase training time, it could be an interesting direction to explore constant optimization in practice.

To help you get started, here’s a small example showing how you can convert a Sympy expression into an EvoGP Tree object. With this, you can experiment with taking optimized Sympy models and reinserting them into the forest:

import sympy as sp
import torch
from evogp.tree import Tree
from evogp.tree.utils import *

INV_SYMPY_MAP = {v: k for k, v in SYMPY_MAP.items()}

def from_sympy_expr(expr, symbol_names=None):
    node_type = []
    node_value = []
    subtree_size = []

    def build_tree(e):
        if isinstance(e, sp.Symbol):
            idx = symbol_names.index(str(e)) if symbol_names else int(str(e)[1:])
            node_type.append(NType.VAR)
            node_value.append(idx)
            subtree_size.append(1)
            return 1

        elif isinstance(e, sp.Number):
            node_type.append(NType.CONST)
            node_value.append(float(e))
            subtree_size.append(1)
            return 1

        else:
            f = e.func
            args = e.args

            if len(args) == 1:
                node_type.append(NType.UFUNC)
            elif len(args) == 2:
                node_type.append(NType.BFUNC)
            elif len(args) == 3:
                node_type.append(NType.TFUNC)
            else:
                raise ValueError(f"Unsupported arity: {len(args)}")

            if f not in INV_SYMPY_MAP:
                raise KeyError(f"Function {f} not in SYMPY_MAP")
            node_value.append(INV_SYMPY_MAP[f])

            child_sizes = [build_tree(arg) for arg in args]
            sz = 1 + sum(child_sizes)
            subtree_size.append(sz)
            return sz

    build_tree(expr)

    node_type = torch.tensor(node_type, dtype=torch.int32)
    node_value = torch.tensor(node_value, dtype=torch.float32)
    subtree_size = torch.tensor(subtree_size[::-1], dtype=torch.int32)

    return node_value, node_type, subtree_size


# Example: convert Sympy expression into EvoGP tree
x0, x1 = sp.symbols("x0 x1")
expr = 2.0 * x0 + sp.sin(x1)

node_value, node_type, subtree_size = from_sympy_expr(expr, symbol_names=["x0", "x1"])
newtree = Tree(2, 1, node_value, node_type, subtree_size)
print(newtree)

[mferguson9215]

Thank you for the response, I'll give it a try! I appreciate the quick assistance!