State saving and serialization (#785)

* Enable saving and restoring subsimulator state (#765)

This is the first step towards closing #756.  I've added functions
corresponding to FMI 2.0's `fmi2{Get,Set,Free}FMUstate()` throughout the
various layers of subsimulator interfaces and implementations:

* `cosim::slave` and its implementation in
   `cosim::fmi::v2::slave_instance`
* `cosim::simulator` and its implementation in `cosim::slave_simulator`

This led me to also remove the `slave_state` and `state_guard` stuff
that was in `slave_simulator.{hpp,cpp}`. The overloading of the "state"
terminology became confusing, and it seemed like it was a lot of code
for very little gain. (It was supposed to be a check of correct API
usage, but I can't remember it ever actually catching a bug.)

* Enable exporting and importing subsimulator state (#769)

This is a follow-up to #765 and the second and final step to close #756.

Here, I've implemented functionality to export the internal state of
individual subsimulators in a generic, structured form, and to import
them again later.

This exported form is intended as an intermediate step before
serialisation and disk storage.  The idea was to create a type that can
be inspected and serialised to almost any file format we'd like.

The type is defined by `cosim::serialization::node` in
`cosim/serialization.hpp`.  It is a hierarchical, dynamic data type with
support for a variety of primitive scalar types and a few aggregate
types: strings, arrays of nodes, dictionaries of nodes, and binary
blobs. (Think JSON, only with more types.) It is based on
Boost.PropertyTree

* Enable saving and restoring full simulation state (#777)

This closes #768. Some changes may warrant a bit of extra explanation:

**`execution` and `algorithm`:** I have taken one step towards prohibiting adding/removing sub-simulators after the co-simulation has begun, as decided in #771.  In particular, I've added the `execution::initialize()` function, which marks the point where such changes are no longer allowed.  (This is a backwards-compatible change, because it gets automatically called by `execution::step()` if it hasn't been called manually.)

**`slave_simulator`**: The changes in `slave_simulator.cpp` really ought to have been included in #769. Unfortunately, I didn't realise they were necessary before it was all put into the context of saving algorithm and execution state.  Basically, I thought I could get away with just saving each FMU's internal state, but it turns out that we also need to save the `slave_simulator` "get" and "set" caches.

(For those interested in the nitty-gritties, this is due to some subtleties regarding exactly when the cache values are set in the course of a co-simulation, relative to when the values are passed to the FMU. At the end of an "algorithm step", the "set cache" is out of sync with the FMUs input variables, and won't be synced before the next step. Simply performing an additional sync prior to saving the state is not sufficient, because that could have an effect on the FMUs output variables, thus invalidating the "get cache". That could in principle be updated too, but then the `slave_simulator` is in a whole different state from where it was when we started to save the state.)

* Feature/779 state serialization (#780)

* Enable saving and restoring full simulation state

Closes #768

* Added tinycbor based serialization method

* Debug test

* Replaced tinycbor -> libcbor (ver 0.1)

* deserialization using cbor_reader

* Separated ctx to own struct

* Tag as std::optional in ctx

* State serialization test

* Removed comments

* Addressed Linux build issue

* Added BouncingBall reference model from modelica for adding byte vector serialization test

* Added BouncingBall license in README

* Added comments

* Deleted state_serialization_test.cpp

* Added libcbor as PRIVATE for `target_link_libraries`

* Controlling fixed precision for logging float point values (#775)

* Added an option to file_observer_config to set
fixed precision value.

* Fixed build failure

* Reading canGetAndSetFMUstate and canSerializeFMUstate from model description file

* Added transitive_libs for libcbor to avoid cmake error

* linking libcbor statically.

* Resetting time series observer upon state restore

* State saving support with proxyfmu

* Split proxyfmu from the save_state_test.

* Uses upgraded thrift/boost

* Fixed boost header issue

* Fixed boost header issue

* Fixed boost header issue

* Update dependency

* Updated action

* Upgraded libzip

* Upgraded libzip

* libzip deprecated func fix

* Observer fix

* Addressing comments
1. Defining exported sourced explicitly
2. Try to use shared libcbor.

* made libcbor PUBLIC in cmake (may require the consumer also declare libcbor for linking libcosim)

* Addressed comments

* Addressed comments

* Addressed comments - 02

* Addressed comments - 03

* Updated cosim_capabilities -> simulator_capabilities. Using cosim::serialization::format::cbor to choose which serialization/deserialization to use

* update

* pretty_print format added

* Updated error message

* Updated error message

* Updated error message

* Using fmu::model_description::capabilities in copy_current_state() and export_state()

* Updated error message

* Don't require importing def_types.h

* Using updated proxyfmu that hides thrift related libs

* Updated field names for the model description

* Updated proxyfmu dependency

---------

Co-authored-by: Lars T. Kyllingstad <lars.kyllingstad@sintef.no>

* Fixed an issue from the merge

---------

Co-authored-by: Lars T. Kyllingstad <lars.kyllingstad@sintef.no>

Author: davidhjp01

.github/workflows/ci-conan.yml

@@ -48,6 +48,7 @@ jobs:
             --settings="build_type=${{ matrix.build_type }}" \
             --options="${{ matrix.option_proxyfmu }}" \
             --options="${{ matrix.option_shared }}" \
+            --update \
             --build=missing \
             --user=osp \
             --channel="${CHANNEL}" \
@@ -103,6 +104,7 @@ jobs:
             --settings="build_type=${{ matrix.build_type }}" \
             --options="${{ matrix.option_proxyfmu }}" \
             --options="${{ matrix.option_shared }}" \
+            --update \
             --build=missing \
             --user=osp \
             --channel="${CHANNEL}" \

CMakeLists.txt

@@ -113,6 +113,7 @@ find_package(libzip REQUIRED)
 find_package(Microsoft.GSL REQUIRED)
 find_package(yaml-cpp REQUIRED)
 find_package(XercesC MODULE REQUIRED)
+find_package(libcbor REQUIRED)
 if(LIBCOSIM_WITH_PROXYFMU)
     find_package(PROXYFMU CONFIG REQUIRED)
 endif()

conanfile.py

@@ -39,19 +39,21 @@ def set_version(self):
     def requirements(self):
         self.tool_requires("cmake/[>=3.19]")
         self.requires("fmilibrary/[~2.3]")
-        self.requires("libzip/[>=1.7 <1.10]") # 1.10 deprecates some functions we use
+        self.requires("libcbor/0.11.0")
+        self.requires("libzip/[~1.11]")
         self.requires("ms-gsl/[>=3 <5]", transitive_headers=True)
+        self.requires("boost/[~1.85]", transitive_headers=True, transitive_libs=True)  # Required by Thrift
         if self.options.proxyfmu:
-            self.requires("proxyfmu/0.3.2@osp/stable")
-            self.requires("boost/[~1.81]", transitive_headers=True, transitive_libs=True) # Required by Thrift
-        else:
-            self.requires("boost/[>=1.71]", transitive_headers=True, transitive_libs=True)
+            self.requires("proxyfmu/0.3.3@osp/testing",
+                          transitive_headers=True,
+                          transitive_libs=True)
         self.requires("yaml-cpp/[~0.8]")
         self.requires("xerces-c/[~3.2]")
 
     # Exports
     exports = "version.txt"
-    exports_sources = "*"
+    exports_sources = ("src/*", "include/*", "cmake/*", "data/*", "docs/*", "tests/*", "CHANGELOG.md", "CMakeLists.txt",
+                       "CONTRIBUTING.md", "LICENSE", "README.md", "version.txt")
 
     # Build steps
 

include/cosim/algorithm/algorithm.hpp

@@ -15,6 +15,7 @@
 #include <cosim/function/function.hpp>
 #include <cosim/model_description.hpp>
 #include <cosim/observer/observer.hpp>
+#include <cosim/serialization.hpp>
 #include <cosim/time.hpp>
 
 #include <functional>
@@ -157,7 +158,9 @@ class algorithm
      *  values for some of them.
      *
      *  This function is guaranteed to be called after `setup()` and before
-     *  the first `do_step()` call.
+     *  the first `do_step()` call. Furthermore, no more subsimulators and
+     *  functions will be added or removed after `initialize()` has been called;
+     *  that is, `{add,remove}_{simulator,function}()` will not be called again.
      */
     virtual void initialize() = 0;
 
@@ -180,6 +183,36 @@ class algorithm
      */
     virtual std::pair<duration, std::unordered_set<simulator_index>> do_step(time_point currentT) = 0;
 
+    /**
+     *  Exports the current state of the algorithm.
+     *
+     *  Note that system-structural information should not be included in the
+     *  data exported by this function, only internal, algorithm-specific data.
+     *  This is because it will be assumed that the system structure is
+     *  unchanged or has already been restored when the state is imported
+     *  again, as explained in the `import_state()` function documentation.
+     */
+    virtual serialization::node export_current_state() const = 0;
+
+    /**
+     *  Imports a previously-exported algorithm state.
+     *
+     *  When this function is called, it should be assumed that the system
+     *  structure is the same as when the state was exported. That is, either
+     *
+     *  1. this is the algorithm instance from which the state was exported,
+     *     and the system structure actually hasn't changed
+     *  2. this is a new instance, but the original system structure has been
+     *     restored prior to calling this function
+     *
+     *  By "system structure", we here mean the subsimulator indexes, the
+     *  function indexes, and the variable connections.
+     *
+     *  It is guaranteed that this function is never called before
+     *  `initialize()`.
+     */
+    virtual void import_state(const serialization::node& exportedState) = 0;
+
     virtual ~algorithm() noexcept = default;
 };
 

include/cosim/algorithm/fixed_step_algorithm.hpp

@@ -56,6 +56,8 @@ class fixed_step_algorithm : public algorithm
     void setup(time_point startTime, std::optional<time_point> stopTime) override;
     void initialize() override;
     std::pair<duration, std::unordered_set<simulator_index>> do_step(time_point currentT) override;
+    serialization::node export_current_state() const override;
+    void import_state(const serialization::node& exportedState) override;
 
     /**
      * Sets step size decimation factor for a simulator.

include/cosim/algorithm/simulator.hpp

@@ -12,6 +12,7 @@
 
 #include <cosim/manipulator/manipulator.hpp>
 #include <cosim/model_description.hpp>
+#include <cosim/serialization.hpp>
 #include <cosim/time.hpp>
 
 #include <functional>
@@ -134,6 +135,72 @@ class simulator : public manipulable
     virtual step_result do_step(
         time_point currentT,
         duration deltaT) = 0;
+
+    /// A type used for references to saved states (see `save_state()`).
+    using state_index = int;
+
+    /**
+     *  Saves the current state.
+     *
+     *  This will create and store a copy of the simulator's current internal
+     *  state, so that it can be restored at a later time.  The copy is stored
+     *  internally in the simulator, and must be referred to by the returned
+     *  `state_index`. The index is only valid for this particular simulator.
+     *
+     *  The function may be called at any point after `setup()` has been called.
+     */
+    virtual state_index save_state() = 0;
+
+    /**
+     *  Saves the current state, overwriting a previously-saved state.
+     *
+     *  This function does the same as `save_state()`, except that it
+     *  overwrites a state which has previously been stored by that function.
+     *  The old index thereafter refers to the newly-saved state.
+     */
+    virtual void save_state(state_index stateIndex) = 0;
+
+    /**
+     *  Restores a previously-saved state.
+     *
+     *  This restores the simulator to a state which has previously been saved
+     *  using `save_state()`.
+     *
+     *  Note that the saved state is supposed to be the *complete and exact*
+     *  state of the simulator at the moment `save_state()` was called. For example,
+     *  if the state was saved while the simulator was in initialisation mode
+     *  (between `setup()` and `start_simulation()`), then it will be restored
+     *  in that mode, and `start_simulation()` must be called before the
+     *  simulation can start.  Similarly, if it is saved at logical time `t`,
+     *  then the first `do_step()` call after restoration must start at `t`.
+     */
+    virtual void restore_state(state_index stateIndex) = 0;
+
+    /**
+     *  Frees all resources (e.g. memory) associated with a saved state.
+     *
+     *  After this, the state may no longer be restored with `restore_state()`,
+     *  nor may it be overwritten with `save_state(state_index)`.  The
+     *  implementation is free to reuse the same `state_index` at a later point.
+     */
+    virtual void release_state(state_index stateIndex) = 0;
+
+    /**
+     *  Exports a saved state.
+     *
+     *  This returns a previously-saved state in a generic format so it can be
+     *  serialized, e.g. to write it to disk and use it in a later simulation.
+     */
+    virtual serialization::node export_state(state_index stateIndex) const = 0;
+
+    /**
+     *  Imports an exported state.
+     *
+     *  The imported state is added to the simulator's internal list of saved
+     *  states. Use `restore_state()` to restore it again. The state must have
+     *  been saved by a simulator of the same or a compatible type.
+     */
+    virtual state_index import_state(const serialization::node& exportedState) = 0;
 };
 
 } // namespace cosim

include/cosim/execution.hpp

@@ -19,6 +19,7 @@
 
 #include <boost/functional/hash.hpp>
 
+#include <cstdint>
 #include <future>
 #include <memory>
 #include <optional>
@@ -36,7 +37,7 @@ using simulator_index = int;
 using function_index = int;
 
 /// An number which identifies a specific time step in an execution.
-using step_number = long long;
+using step_number = std::int64_t;
 
 /// An object which uniquely identifies a simulator variable in a simulation.
 struct variable_id
@@ -145,6 +146,13 @@ class simulator;
  *  The `execution` class manages all the entities involved in an execution
  *  and provides a high-level API for driving the co-simulation algorithm
  *  forward.
+ *
+ *  \warning
+ *      In general, the member functions of this class are not exception safe.
+ *      This means that if any of them throw an exception, one must assume that
+ *      the `execution` object is in an invalid state and can no longer be used.
+ *      The same holds for its associated algorithm and any simulators or
+ *      functions that are part of the execution.
  */
 class execution
 {
@@ -179,13 +187,19 @@ class execution
      *      The recommended co-simulation step size for this slave.
      *      Whether and how this is taken into account is algorithm dependent.
      *      If zero, the algorithm will attempt to choose a sensible default.
+     *
+     *  \pre `initialize()` has not been called.
      */
     simulator_index add_slave(
         std::shared_ptr<slave> slave,
         std::string_view name,
         duration stepSizeHint = duration::zero());
 
-    /// Adds a function to the execution.
+    /**
+     *  Adds a function to the execution.
+     *
+     *  \pre `initialize()` has not been called.
+     */
     function_index add_function(std::shared_ptr<function> fun);
 
     /// Adds an observer to the execution.
@@ -242,6 +256,14 @@ class execution
     /// Returns the current logical time.
     time_point current_time() const noexcept;
 
+    /**
+     *  Initialize the co-simulation (in an algorithm-dependent manner).
+     *
+     *  After this function is called, it is no longer possible to add more
+     *  subsimulators or functions.
+     */
+    void initialize();
+
     /**
      *  Advance the co-simulation forward to the given logical time (blocks the current thread).
      *
@@ -255,6 +277,12 @@ class execution
      *      `true` if the co-simulation was advanced to the given time,
      *      or `false` if it was stopped before this. In the latter case,
      *      `current_time()` may be called to determine the actual end time.
+     *
+     *  \note
+     *      For backwards compatibility, this function automatically calls
+     *      `initialize()` if this hasn't already been done. However, new code
+     *      should always call `initialize()` before any of the
+     *      simulation/stepping functions.
      */
     bool simulate_until(std::optional<time_point> targetTime);
 
@@ -271,6 +299,12 @@ class execution
      *      `true` if the co-simulation was advanced to the given time,
      *      or `false` if it was stopped before this. In the latter case,
      *      `current_time()` may be called to determine the actual end time.
+     *
+     *  \note
+     *      For backwards compatibility, this function automatically calls
+     *      `initialize()` if this hasn't already been done. However, new code
+     *      should always call `initialize()` before any of the
+     *      simulation/stepping functions.
      */
     std::future<bool> simulate_until_async(std::optional<time_point> targetTime);
 
@@ -281,6 +315,12 @@ class execution
      *      The actual duration of the step.
      *      `current_time()` may be called to determine the actual time after
      *      the step completed.
+     *
+     *  \note
+     *      For backwards compatibility, this function automatically calls
+     *      `initialize()` if this hasn't already been done. However, new code
+     *      should always call `initialize()` before any of the
+     *      simulation/stepping functions.
      */
     duration step();
 
@@ -314,6 +354,28 @@ class execution
     /// Set initial value for a variable of type string. Must be called before simulation is started.
     void set_string_initial_value(simulator_index sim, value_reference var, const std::string& value);
 
+    /**
+     *  Exports the current state of the co-simulation.
+     *
+     *  \pre `initialize()` has been called.
+     *  \pre `!is_running()`
+     */
+    serialization::node export_current_state() const;
+
+    /**
+     *  Imports a previously-exported co-simulation state.
+     *
+     *  Note that the data returned by `export_current_state()` only describe
+     *  the *state* of the system, not its structure.  This means that it's the
+     *  caller's responsibility to either
+     *
+     *  1. not modify the system structure between state export and state import
+     *  2. restore the pre-export system structure prior to state import
+     *
+     *  \pre `initialize()` has been called.
+     *  \pre `!is_running()`
+     */
+    void import_state(const serialization::node& exportedState);
 
 private:
     class impl;

include/cosim/fmi/v1/fmu.hpp

@@ -165,6 +165,13 @@ class slave_instance : public fmi::slave_instance
         gsl::span<const value_reference> variables,
         gsl::span<const std::string> values) override;
 
+    state_index save_state() override;
+    void save_state(state_index overwriteState) override;
+    void restore_state(state_index state) override;
+    void release_state(state_index state) override;
+    serialization::node export_state(state_index stateIndex) const override;
+    state_index import_state(const serialization::node& exportedState) override;
+
     // fmi::slave_instance methods
     std::shared_ptr<fmi::fmu> fmu() const override
     {

include/cosim/fmi/v2/fmu.hpp

@@ -21,9 +21,11 @@
 #include <string>
 #include <string_view>
 #include <vector>
+#include <queue>
 
 
 struct fmi2_import_t;
+using fmi2_FMU_state_t = void*;
 
 
 namespace cosim
@@ -165,6 +167,13 @@ class slave_instance : public fmi::slave_instance
         gsl::span<const value_reference> variables,
         gsl::span<const std::string> values) override;
 
+    state_index save_state() override;
+    void save_state(state_index stateIndex) override;
+    void restore_state(state_index stateIndex) override;
+    void release_state(state_index stateIndex) override;
+    serialization::node export_state(state_index stateIndex) const override;
+    state_index import_state(const serialization::node& exportedState) override;
+
     // fmi::slave_instance methods
     std::shared_ptr<fmi::fmu> fmu() const override
     {
@@ -178,13 +187,25 @@ class slave_instance : public fmi::slave_instance
     fmi2_import_t* fmilib_handle() const;
 
 private:
+    struct saved_state
+    {
+        fmi2_FMU_state_t fmuState = nullptr;
+        bool setupComplete = false;
+        bool simStarted = false;
+    };
+    void copy_current_state(saved_state& state);
+    state_index store_new_state(saved_state state);
+
     std::shared_ptr<v2::fmu> fmu_;
     fmi2_import_t* handle_;
 
     bool setupComplete_ = false;
     bool simStarted_ = false;
 
     std::string instanceName_;
+
+    std::vector<saved_state> savedStates_;
+    std::queue<state_index> savedStatesFreelist_;
 };