Design Doc: support optim() and optimize()

Owner: @fritzo | Pull Request: 318

Objective

Compile R code that uses optim() and optimize(), such that the compiled function directly uses the underlying C interface.

Outline of processing flow ideas

We start with a call like (R syntax): ans <- optim(par, fn, gr, ..., method = 'Nelder-Mead', lower, upper, control, hessian)

Ideas will follow sequence of processing steps.

1. Do we need keyword processing? In general, keyword processing is necessary when we need to insert new setup code. It is sometimes convenient for simple code transformations even if we don't need new setup code, but in general I prefer to have such transformations at later steps (like size processing, which is misnamed because we do more than size processing there).

In the current prototype, keyword processing creates something like the following

{
  vPtrFor_innerNF <- voidPtr(innerNF,nimbleFunction)
  vPtrFor_yProvided <- voidPtr(ARG2_yProvided_,double(1))
  vPtrFor_2p5 <- voidPtr(Interm_6,double(=0,default=2.5))
  bareBonesOptim(initPar=ARG1_xInit_,optFun=OPTIMREADY_nfObjective,voidNimFunPtr=vPtrFor_innerNF,numAdditionalArgs=3,y=vPtrFor_yProvided,z=vPtrFor_2p5,w=vPtrFor_2p5)
}

It also creates a line of newSetupCode (I think to create an object named OPTIMREADY_nfObjective whose role is really to trigger some other steps later).

Some observations about this:

• We originally thought the "method" would have to be baked in at compile time and become a different keyword (e.g. bareBonesOptim. We can evaluate if that is necessary. We can probably have run-time flexibility for method.

• Lifting of arguments to temporary variables (voidPtrs) is typically done in size processing, not keyword processing.

• voidPtr is an example of an internal part of the DSL. It's valid, but we don't document it because there's nothing to do with it as a programmer.

• The prototype only works if the objective function is the run function of a nimbleFunction class (i.e. with setup code). We probably, ideally, want optim to work with 3-4 cases: an RCfunction (aka simple nimbleFunction, without setup code), a member function of the same class in which optim is being used, or a method of another class (potentially nested: method of a nimbleFunction contained in another nimbleFunction, optim(par, fn = A$B$foo)). The prototype works only because (using our example) keyword processing for nf2Gen$run can expect that innerNF is already populated in the symbolTable and we can look up its run symbol. But for some of the desired cases, we won't be able to find another nimbleFunction's symbolTable until size processing. E.g. if we call optim(par, fn = foo) and foo is an RCfunction, we would normally not know anything about foo during keyword processing, but we would know (or be able to find out) about it during size processing. Summary: I bet we'll want to move much (all?) of what keyword processing does in the prototype to size processing.

Task summary

• Expunge Cliff's original optim() code to prepare for reimplementation.

• Add a new keyword fakeOptim(-) with trivial behavior (fakeOptim(x) = x).

  • Make fakeOptim become nimFakeOptim.

  • Add fakeOptim where needed to various steps.

    • Size processing. See details below

  • (Skip eigenization steps for now).

  • (Skip genCpp steps because default will be to emit nimFakeOptim as the function name. Later if needed we can make an entry in cppOutputCalls in genCpp_generateCpp.R)

• Return a nimbleList from fakeOptim().

  • Define a optimResultNimbleList for result type of optim()

  • Plumb fakeOptim() to return its nimbleList type.

• Plumb fakeOptim() to input two arguments (par, fn), assuming method = 'Nelder-Mead' (say), and fn is an RCfunction or a local method or another nimbleFunction's method. Perform the following steps in the size processor (as initially prototyped in keyword processing):

  • CANCELLED Create voidPtrs (or add them directly to the symbolTable instead of constructing code);

  • Record types needed for optim in the neededTypes system. For a nimbleFunction, neededTypes is a list of objects (e.g. nfProcessing) that represent the types needed to compile the current code.

    • CANCELLED Rename the RCfunProc list from neededRcFuns to NeededTypes. (This renaming is useful for both nimbleLists and for fakeOptim. Ideally we'll do this separately and you can just grab the changes.)

• Create a C++ wrapper nimOptim() around R's C implementation of optim().

• Swap out the C++ implementation to use nimOptim().

• Replace all occurrences of fakeOptim with optim.

• Allow nfMethods as objective functions.

• Support gradient arguments gr

• Support bound arguments lower and upper

• Support other optimization methods: "Nelder-Mead", "BFGS", "CG", "L-BFGS-B", "SANN", "Brent"

• Define optimControlNimbleList for control parameters of optim()

• Allow RCfunctions to use optim (using Perry's nimbleList-RCfun branch.

• Support control parameter fnscale for maximization problems.

• Fall back to numerical gradient when gr argument is missing.

• Support extra arguments via nimOptim(par, fn, ...)

• Support 1-dimensional optimization with optimize().

Task details

Size processing

See size processing docs. Name the new size processor sizeOptim. Add a new entry in sizeCalls, like nimFakeOptim = 'sizeOptim'. As the following steps progress, sizeOptim will gain more steps. It will need to label it's return type as a nimbleList, ensure the sizeExprs is an appropriate nimbleList symbol (see other cases, talk to NM), and probably self-lift out of non-assignment callers. Also it will probably process some of the nimOptim arguments.