Proposal to add new endpoint for generic XY

[siwei-zhang-work]

Overview

There has been a need to support a density view in the viewer. Progress on this was previously paused due to the lack of support for views with a non-timestamp X-axis. This proposal aims to see the feasibility for later contributions.

The proposed view

Composition of the view

1. Right-hand Side: This part of the view is similar to the current density function available in Eclipse, where each bar represents a bin interval. The X-axis in this view is flexible and can represent various metrics, such as duration or frequency, depending on the server's implementation. For example, a category could be a specific range of execution duration, like 20ms–30ms.

2. Left-hand Side: This section will resemble the XY tree view. An extra panel specifying the description of X-axis and Y-axis can also be here.

Relations with "Time Range Selectors":

The bar chart view would show 2 parts similar to what we already have for segment store statistics with Total and Selected. One part is Total, which refers to the entire trace. And the other part is selected, which refers to [Selection Range Start, Selection Range End] and is affected by the window selected by the user.

Since the bar chart does not have time as the X-axis, zooming in the bar chart view cannot affect the other views.

Example view with selections:

The proposed API

The required API is akin to the existing XY API.

1. The viewer requests a tree of series
2. The viewer requests the model of any selected series, the series are selected in the same way as the series in XY view
3. The viewer visualizes the models as a bar chart

Proposed endpoints

Proposal 1 : a new endpoint for bar chart

- GET (POST) bar category tree
  - viewer request
    - time range
  - server response
    - list of "Y-series"  – Same as `"model"` for XY.
      - id
      - parent id
      - name
      - hasData
      - style
      - labels
      - isDefault
- GET (POST) bar model
  - viewer request
    - time range (no sampling)
    - list of selected entries to provide
  - server response
    - list of series
      - series id
      - series name
      - style
      - Y-Values
      - Y-Axis description
      - X-Values (list of categories)
      - X-Axis description
    - title (this is available in the XY view endpoint, for instance `"title": "Latency vs Time"`)

Other considerations

This view could be extremely slow as it queries all the segments to calculate the counts for each category instead of sampling, so performance needs to be considered when implementing the backend.

[bhufmann]

Thanks for the proposal. We will review soon.

[bhufmann]

@siwei-zhang-work see below inline some questions for this proposal.

Overview

There has been a need to support a density view in the viewer. Progress on this was previously paused due to the lack of support for views with a non-timestamp X-axis. This proposal aims to see the feasibility for later contributions.

The proposed view

Composition of the view

1. Right-hand Side: This part of the view is similar to the current density function available in Eclipse, where each bar represents a bin interval. However, the X-axis in this view would be customizable to represent various metrics like duration, frequency, etc.

Customizable here means, customized by the back-end, right? So, the back-end data provider implementation will determine what the x-axis represents.

2. Left-hand Side: This section will resemble the XY tree view. The Y-values will represent the count for each category, while the X-values will represent different categories. For example, a category could be a specific range of execution duration, like 20ms–30ms. The categories don't have to be uniformly distributed and can be customized by the user.

category are buckets/bins, right? They might no have a range and just a value, e.g. color of car.

Y-series:
- my-experiment
  - thread-groupA
    - threadA
      - threadA-type1
      - threadA-type2
    - threadB
      - threadB-type1
      - threadB-type2
  - thread-groupB
    - ...

3. Panel for Legends: There will be a panel in the view displaying the X-legend and Y-legend.

What do you mean here with panel? Where is the panel supposed to be visualized. Please clarify.

Note: How and where the legend is displayed is FE implementation specific as long as the information is passed over the TSP. But it's good to show here the vision for such a view.

Relations with "Time Range Selectors":

The bar chart view would show 2 parts similar to what we already have for segment store statistics with Total and Selected. One part is windowed, which refers to the [Time Range Start, Time Range End] and is affected by the zooms in time-related views. And the other part is selected, which refers to [Selection Range Start, Selection Range End] is affected by the window selected by the user.

How are the 2 parts supposed to be visualized? 2 view below each other or side by side or in the same view? Please clarify.
Note: How and where the 2 parts are going to be displayed is FE implementation specific as long as the TSP is transporting the required data. However it's good (like above) to explain here the vision for this view.

Since the bar chart does not have time as the X-axis, zooming in the bar chart view cannot affect the other views.

The proposed API

The required API is akin to the existing XY API.

1. The viewer requests a tree of series

2. The viewer requests the model of any selected series

Same comment here, that the way the selected series are visualized is FE specific, but it's good to explain how it could be done.

3. The viewer visualizes the models as a bar chart

Proposed endpoints:

- GET (POST) bar category tree (similar to XY tree)
  - viewer request
    - time range
  - server response
    - list of "Y-series" (same as `"model"` for XY)
      - id
      - parent id
      - name
      - hasData
      - style
      - labels
      - isDefault
    - y-legend provided by this view (enum type)
    - x-legend provided by this view (enum type)

I'm not fully clear how the legend will look like, for x-legend nor for y-legend. For example, x-axis are duration what would be the legend? or y-axis is count what would be the legend. For me, the legend determines e.g. color of a series.

Please clarify and give an example.

- range for the categories (e.g., `[min execution time, max execution time]`)

If category is a "real" category, e.g. color of cars, then there is no range. How would it look like this field? start and end would be the same?

• GET (POST) bar model
  • viewer request
    • time range (no sampling)
    • list of entries (categories) to provide, the provided categories mustn't overlap
  • server response
    • list of series
      • series id
      • series name
      • style
      • Y-Values (count per category)
      • X-Values (list of categories)
    • title (this is available in the XY view endpoint, for instance "title": "string", not sure how it is used)

Should we define the data type of x axis and y axis? The tree part has DataType of the whole tree and each column. But that is not sufficient for the axis. It would be good to be able to indicate that the x-axis are duration.

Other considerations

This view could be extremely slow as it queries all the segments to calculate the counts for each category instead of sampling. But O(n) for iterating the intervals (what has been done in eclipse for AbstractSegmentStoreDensityViewer) we can't escape easily.

What you describe here is specific for the Trace Compass server implementation when using segment stores. This is not relevant for the TSP specification. However, it's a good reminder that the back-end implementation needs to be performant.

[siwei-zhang-work]

Thanks for the review. The proposal is now updated according to comments.

Customizable here means, customized by the back-end, right? So, the back-end data provider implementation will determine what the x-axis represents.

Yes, that's correct. The back-end implementation determines the meaning of the X-axis (e.g., duration, frequency), and the front-end will use that to label and visualize accordingly.

category are buckets/bins, right? They might not have a range and just a value, e.g. color of car.

Yes they could be, and I see that the current design doesn't support it. Discussed in another comment below.

What do you mean here with panel? Where is the panel supposed to be visualized. Please clarify.

I made a POC to clarify:

Sorry I see that how I put the description of x/y values in the description of left-hand side view can be confusing. They are now moved up in the description on left-hand side view.

How are the 2 parts supposed to be visualized? 2 view below each other or side by side or in the same view? Please clarify.

Same comment here, that the way the selected series are visualized is FE specific, but it's good to explain how it could be done.

The plan is to put them side-by-side in the same view with no aggregation, which is consistent with how the XY view is shown. And example view is added in the proposal.

I've also realized that to be consistent with how Data Tree is presented, the windowed (View Range Start and View Range End in Time Range Data) shouldn't be considered. So I updated the proposal accordingly.

I'm not fully clear how the legend will look like, for x-legend nor for y-legend. For example, x-axis are duration what would be the legend? or y-axis is count what would be the legend. For me, the legend determines e.g. color of a series.

It would’ve been more accurate to call it “legend type (unit),” and the design wasn’t well thought out. The enum type I had in mind was something like:

 Y-axis: {Count, Percentage}
 X-axis: {Duration, Frequency, Memory Usage}

But they should've been combined with range for the categories. Updated the api design.

If category is a "real" category, e.g. color of cars, then there is no range. How would it look like this field? start and end would be the same?

If the category is a "real" category, then I don't think we have any choice apart from sending all the "real" values here. Combining the previous comments, I updated:

 - y-legend provided by this view (enum type)
 - x-legend provided by this view (enum type)
 - range for the categories (e.g., `[min execution time, max execution time]`)

with:

   - Y-axis
      - label (string) – The display name for the Y-axis.
      - unit (string) – The unit of measurement (e.g., "" for count, "%" for percentage).
   - X-axis
      - label (string) – The display name for the X-axis.
      - unit (string) – The unit of measurement (e.g., "" for categorical data, "ns", "byte").
      - categoryType (enum: "ranged" | "categorical") – Indicates whether the X-axis uses a continuous range or discrete categories.
      - available values – Either a numeric range (for ranged types) or a list of categories (for categorical types).

Should we define the data type of x axis and y axis? The tree part has DataType of the whole tree and each column. But that is not sufficient for the axis. It would be good to be able to indicate that the x-axis are duration.

Do you mean the units for x axis and y axis? I've now added them.

What you describe here is specific for the Trace Compass server implementation when using segment stores. This is not relevant for the TSP specification. However, it's a good reminder that the back-end implementation needs to be performant.

Rephrased.

[bhufmann]

• Y-axis
  - label (string) – The display name for the Y-axis.
  - unit (string) – The unit of measurement (e.g., "" for count, "%" for percentage).
  • X-axis
    • label (string) – The display name for the X-axis.
    • unit (string) – The unit of measurement (e.g., "" for categorical data, "ns", "byte").
    • categoryType (enum: "ranged" | "categorical") – Indicates whether the X-axis uses a continuous range or discrete categories.
    • available values – Either a numeric range (for ranged types) or a list of categories (for categorical ty

I think the axis need a field for the DataType. The ISeriesModel of Trace Compass for XY charts has methods to get the x-axis and y-axis description getXAxisDescription() and getYAxisDescription() (see class TmfXYAxisDescription). It has similar content than what you propose here. This is not serialized yet in the Trace Server (see here) , but it was meant for the future. When implementing the back-end those class should be re-used and, if needed, augmented (e.g. for categorical). I wonder if the same naming as it was used for SegmentType which are (CATEGORICAL, CONTINUOUS) (Note we can't use SegmentType because we don't want to create a coupling to the segment package).

Note that the ISeriesModel is for common X-axis in time. It's not probably not sufficient for other types of xy charts and your proposal and in the BE there are probable other structures are needed.

[siwei-zhang-work]

I think the axis need a field for the DataType.

I wonder if the same naming as it was used for SegmentType which are (CATEGORICAL, CONTINUOUS) (Note we can't use SegmentType because we don't want to create a coupling to the segment package).

Added DataType. But I think the categoryType in X-axis should be part of the DataType so I removed it.

The ISeriesModel of Trace Compass for XY charts has methods to get the x-axis and y-axis description getXAxisDescription() and getYAxisDescription() (see class TmfXYAxisDescription). It has similar content than what you propose here. This is not serialized yet in the Trace Server (see here) , but it was meant for the future. When implementing the back-end those class should be re-used and, if needed, augmented (e.g. for categorical).

Note that the ISeriesModel is for common X-axis in time. It's not probably not sufficient for other types of xy charts and your proposal and in the BE there are probable other structures are needed.

Thanks I see that a lot of methods for XY endpoint can be made more generic and reused. But I'm starting to think that this proposal should've have been proposing a new endpoint, it should also be trying to make XY endpoint more generic? In theory the existing XY endpoint allows the bar chart as long as the "xValues" can be string. I modified the proposal 2 trying to make it a backward-compatible change.

[bhufmann]

Thanks for the updates.

Thanks I see that a lot of methods for XY endpoint can be made more generic and reused. But I'm starting to think that this proposal should've have been proposing a new endpoint, it should also be trying to make XY endpoint more generic? In theory the existing XY endpoint allows the bar chart as long as the "xValues" can be string. I modified the proposal 2 trying to make it a backward-compatible change.

The existing endpoint for XY charts are for charts with a common time axis as XY axis. It implies certain behaviours in the viewer. I think a new endpoint here is ok and maybe easier to realize instead of putting everything into one endpoint.

Besides, the proposal looks good from what I can see at the moment. Having said that, I think during implementation we might face a situation that we might have to adjust it. This is ok during until we have a full stack implementation of one (or more) use case.

Once you have code to share (drafts) I will try it out.

[siwei-zhang-work]

A modification was made to the second endpoint to let server request the list of categories:

- GET (POST) bar model
  - viewer request
    - time range (no sampling)
    **- [Added] list of sampled categories**
    - list of entries (categories) to provide, the provided categories mustn't overlap
  - server response
    - list of series
      - series id
      - series name
      - style
      - Y-Values
      - X-Values (list of categories)
    - title (this is available in the XY view endpoint, for instance `"title": "Latency vs Time"`)

[siwei-zhang-work]

Updated according to discussions during server implementation:
eclipse-tracecompass/org.eclipse.tracecompass#291