Need help useing both Eye Gaze Pointer and ShellHandRayPointer

[mintos002]

Hello MRTK Community,

I'm currently working on an application for HoloLens 2 using Unity and the Mixed Reality Toolkit (MRTK). In my application, I need to control the scene using both the GazePointer and HandRayPointer in the same scene. However, I'm facing an issue with the pinch motion not working when the GazePointer is active and the HandRayPointer is inactive.

Current Implementation:
I have a toggle button to switch between enabling and disabling the HandRayPointer.
When the HandRayPointer is enabled, the GazePointer is disabled, and vice versa.
I'm using the PointerUtils class from MRTK to control the behavior of the HandRayPointer and GazePointer.

Desired Behavior:
I want to be able to use the pinch motion when the GazePointer is active and the HandRayPointer is inactive.
When the HandRayPointer is disabled, I should still be able to perform the pinch motion to interact with buttons or other UI elements.

Current Code:
Notice that the toggleButton variable is the button to press in order to switch between modes, and eyePointer is the sphere guide for the eye tracking. The button callback executes the HandRayOnOff() function.

using Microsoft.MixedReality.Toolkit;
using Microsoft.MixedReality.Toolkit.Input;
using Microsoft.MixedReality.Toolkit.Utilities;
using System.Collections.Generic;
using UnityEngine;

public class cEyeHandController : MonoBehaviour
{
    // We will track the hand in this case
    public GameObject toggleButton;
    public GameObject eyePointer;

    private void Start()
    {
        // Disable HandRay
        PointerUtils.SetHandRayPointerBehavior(PointerBehavior.AlwaysOff, Handedness.Any);

        // Enable Eye Gaize
        PointerUtils.SetGazePointerBehavior(PointerBehavior.AlwaysOn);
    }

    public void HandRayOnOff()
    {
        // If is not toggeled, show hand ray
        if (!toggleButton.GetComponent<Microsoft.MixedReality.Toolkit.UI.Interactable>().IsToggled)
        {
            // Disable HandRay
            PointerUtils.SetHandRayPointerBehavior(PointerBehavior.AlwaysOff, Handedness.Any);

            // Enable Eye Gaize
            PointerUtils.SetGazePointerBehavior(PointerBehavior.AlwaysOn);
            // Show eye cursor
            if(!eyePointer.activeSelf) eyePointer.SetActive(true);

        }
        else
        {
            // Disable HandRay
            PointerUtils.SetHandRayPointerBehavior(PointerBehavior.AlwaysOn, Handedness.Any);

            // Enable Eye Gaize
            PointerUtils.SetGazePointerBehavior(PointerBehavior.AlwaysOff);
            // Hide eye cursor
            if (eyePointer.activeSelf) eyePointer.SetActive(false);
        }
    }
}

Steps to Reproduce:
Enable the GazePointer and disable the HandRayPointer.
Try to perform the pinch motion on UI buttons or interactive objects.

Expected Result:
The pinch motion should be recognized and trigger the interaction with UI elements.

Actual Result:
The pinch motion doesn't work when the GazePointer is active and the HandRayPointer is inactive.

Additional Information:
Unity version: 2019.4.35f1
MRTK version: 2.7.2.0
Hardware: HoloLens 2

Requested Help:
I'm seeking assistance in resolving this issue so that I can enable the GazePointer and still be able to use the pinch motion for interacting with UI elements. I'm open to any suggestions, code modifications, or best practices that can help me achieve the desired behavior.

Thank you in advance for your help and support!

[Radixtrator]

Were you ever able to figure this out? I have been trying to do the same, use gaze for selecting a button and then pinching to activate it. But as soon as I disable the HandPointer it also disables pinch

[mintos002]

Nah, It seems that, with MRTK2.7, having both gaze selection and hand ray activation simultaneously isn't possible, even with the activate/deactivate option. I couldn't find a workaround, so I opted for eye gaze alone. Perhaps using MRTK 3 version may offer a solution. Let me know if you find a solution!