How to record custom data in experiments

You can configure experiments and experimental runs to record different data such as time, collisions, deadlocks, poses, velocities, targets and commands of all agents, safety margin violations, efficacy, and data logged from tasks.

You can record additional data through custom probes. Probes are callbacks called during experimental runs. Depending on the use case, you can subclass one of RecordProbe or GroupRecordProbe to record data in a single dataset or in a map of datasets. In case you need more flexibility, subclass instead the base class Probe and create/write any number of datasets.

A Single Dataset

When the data you want to record fits a single dataset, you can use RecordProbe.

This is the case for instance, when you want to record the same number of items per step and per agent, and all have the same type.

1. Add a subclass of RecordProbe

   C++Python

   Specialize navground::sim::RecordProbe.

   #include "navground/sim/experimental_run.h"
   #include "navground/sim/probe.h"
   
   namespace sim = navground::sim;
   
   class MyProbe : public sim::RecordProbe {
   

2. Specify the type

   C++Python

   using Type = uint8_t;
   

3. Overwrite the method that define the shape of the dataset

   C++Python

   sim::Dataset::Shape get_shape(const sim::World &world) const override {
       return ...;
   }
   

4. Overwrite the method that fill the dataset

   C++Python

   void update(sim::ExperimentalRun *run) override {
     // call get_data()->push(...) or get_data()->append(...)
   }
   

5. Add the probe to the experiment or the experimental run

   C++Python

   experiment.add_record_probe<MyProbe>("my_key");
   

6. Run the experiment (or the experimental run)

A Group of Datasets

When the data you want to save does not fit a single dataset, like, for example, when data associated with different agents may have different shapes, you can split it up in multiple datasets indexed by string-valued keys, which may be the UID or the index of the agents.

In this case, you can use GroupRecordProbe to setup and manage a dictionary of datasets.

1. Add a subclass of GroupRecordProbe

   C++Python

   Specialize navground::sim::GroupRecordProbe.

   #include "navground/sim/experimental_run.h"
   #include "navground/sim/probe.h"
   
   namespace sim = navground::sim;
   
   class MyProbe : public sim::GroupRecordProbe {
   

2. Specify the type

   C++Python

   using Type = uint8_t;
   

3. Overwrite the method that define the shape of the datasets

   C++Python

   sim::Dataset::Shape get_shapes(const World &world, bool use_uid) const override {
       // return ...;
   }
   

4. Overwrite the method that fill the datasets

   C++Python

   void update(sim::ExperimentalRun *run) override {
       // get the datasets using
       // data = get_data(key)
       // call get_data()->push(...) or get_data()->append(...)
   }
   

5. Add the probe to the experiment or the experimental run

   C++Python

   experiment.add_group_record_probe<MyProbe>("my_key");
   

6. Run the experiment (or the experimental run)

A Custom Probe

When a group of datasets does not fit your needs, you can subclass the base class Probe to implement custom callbacks.

This is the case for instance when the datasets should belong to a hierarchy of groups. In this case, you need to explicitly add the datasets to the ExperimentalRun.

1. Add a subclass of Probe

   C++Python

   Specialize navground::sim::Probe.

   #include "navground/sim/experimental_run.h"
   #include "navground/sim/probe.h"
   
   class MyProbe : public navground::sim::Probe {
   

2. Overwrite the callbacks to create and update the datasets

   C++Python

   void prepare(sim::ExperimentalRun *run) override {
       // create one or more datasets
       auto data = run->add_record("group/subgroup/key");
       // set their type and shape
       get_data()->set_type<float>();
       get_data()->set_item_shape({3, 4});
   }
   
   void update(sim::ExperimentalRun *run) override {
      auto data = run->get_record("group/subgroup/key");
      // call get_data()->push(...) or get_data()->append(...);
   }
   
   void finalize(sim::ExperimentalRun *run) override {
      // last chance to create/modify datasets
   }
   

3. Add the probe to the experiment or the experimental run

   C++Python

   experiment.add_probe([](){return std::make_shared<MyProbe>();});
   

4. Run the experiment (or the experimental run)

Complete example

Have at look at the C++ and Python examples to see complete examples of performing experiments using custom probes.