[Bug Report] CloudXR conflicts with Cameras

[KevinGeng3]

Describe the bug

For example, if I remove the camera sensors in my Isaac-Lift-Cube-Aloha-IK-Abs-v0 env and run the following command:

python scripts/environments/teleoperation/teleop_vp_agent.py     --task Isaac-Lift-Cube-Aloha-IK-Abs-v0     --teleop_device handtracking_abs

everything goes well.
But if I add a camera sensor to Isaac-Lift-Cube-Aloha-IK-Abs-v0 env and run the same command with 'enable_cameras':

./isaaclab.sh -p scripts/environments/teleoperation/teleop_vp_agent.py     --task Isaac-Lift-Cube-Aloha-IK-Abs-v0     --teleop_device handtracking_abs --enable_cameras

an error happened:

Traceback (most recent call last):
  File "xxx", line xxx, in <module>
    from isaacsim.xr.openxr import OpenXRSpec
ModuleNotFoundError: No module named 'isaacsim.xr

System Info

• Isaac Sim Version: 4.5
• OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-59-generic
• GPU: NVIDIA GeForce RTX 4090
• CUDA: 12.4
• GPU Driver: 550.144.03

If you know how to solve this bug, please tell me, thank you!

[KevinGeng3]

I think this bug is caused by the experience file. If you use --enable_cameras, AppLauncher would load "IsaacLab/apps/isaaclab.python.rendering.kit".Otherwise, AppLauncher would load "IsaacLab/apps/isaaclab.python.xr.openxr.kit". And the "IsaacLab/apps/isaaclab.python.rendering.kit" does not contain anything about XR.

Could you please support both cameras and XR in one experience file in the future? Thank you!

[KevinGeng3]

I tried to merge the key contents of the two experience files:

##
# 合并自：
# - camera: https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs/blob/main/apps/omni.isaac.sim.python.gym.camera.kit
# - xr:    _isaac_sim/apps/isaacsim.exp.xr.openxr.kit
##

[package]
title       = "Isaac Lab Python Camera + OpenXR"
description = "An app for running Isaac Lab with rendering and OpenXR enabled"
version     = "2.1.0"

# 浏览时可筛选到 "experience"
keywords = ["experience", "app", "isaaclab", "python", "camera", "minimal", "xr", "openxr"]

[dependencies]
# Camera 部分依赖
"isaaclab.python"                = {}
"omni.kit.property.physx"        = {}
"omni.kit.material.library"      = {}
# XR 部分新增
"isaacsim.xr.openxr"             = {}
"omni.kit.xr.system.openxr"      = {}
"omni.kit.xr.profile.ar"         = {}

[settings.isaaclab]
# 启用 camera 渲染
cameras_enabled = true

[settings]
# app 基本信息
app.versionFile                             = "${exe-path}/VERSION"
app.folder                                  = "${exe-path}/"
app.name                                    = "Isaac-Sim"
app.version                                 = "4.5.0"

# 保持 camera 文件中的输出和性能相关设置
app.enableStdoutOutput                      = false
isaac.startup.ros_bridge_extension          = ""

# Renderer 性能优化（camera 优先）
rtx.translucency.enabled                    = false
rtx.reflections.enabled                     = false
rtx.indirectDiffuse.enabled                 = false
rtx-transient.dlssg.enabled                 = false
rtx.directLighting.sampledLighting.enabled  = true
rtx.directLighting.sampledLighting.samplesPerPixel = 1
rtx.sceneDb.ambientLightIntensity           = 1.0
# rtx.shadows.enabled                       = false
rtx.pathtracing.maxSamplesPerLaunch         = 1000000
rtx.viewTile.limit                          = 1000000
exts."omni.renderer.core".present.enabled   = false
rtx.raytracing.cached.enabled               = false
rtx.ambientOcclusion.enabled                = false
rtx.post.dlss.execMode                      = 0
renderer.multiGpu.maxGpuCount               = 1
renderer.gpuEnumeration.glInterop.enabled = true

# 异步渲染 & XR 冲突：camera 强制 sync，故保留 false
omni.replicator.asyncRendering              = false
app.asyncRendering                          = false   # 来自 XR，camera 无此项，故保留
app.asyncRenderingLowLatency                = false   # 来自 XR
app.renderer.waitIdle                       = true
app.hydraEngine.waitIdle                    = true
app.updateOrder.checkForHydraRenderComplete = 1000

# 其他
app.audio.enabled                           = false
exts."omni.replicator.core".Orchestrator.enabled = false

# XR 专有设置
app.xr.enabled                              = true
xr.ui.enabled                               = false
xr.depth.aov                                = "GBufferDepth"
defaults.xr.profile.ar.renderQuality        = "off"
defaults.xr.profile.ar.anchorMode           = "custom anchor"
persistent.xr.profile.ar.render.nearPlane   = 0.15

[settings.physics]
updateToUsd                = false
updateParticlesToUsd       = false
updateVelocitiesToUsd      = false
updateForceSensorsToUsd    = false
outputVelocitiesLocalSpace = false
useFastCache               = false
visualizationDisplayJoints = false
fabricUpdateTransformations       = false
fabricUpdateVelocities             = false
fabricUpdateForceSensors           = false
fabricUpdateJointStates            = false

[settings.exts."omni.kit.registry.nucleus"]
registries = [
    { name = "kit/default", url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/106/shared" },
    { name = "kit/sdk",     url = "https://ovextensionsprod.blob.core.windows.net/exts/kit/prod/sdk/${kit_version_short}/${kit_git_hash}" },
    { name = "kit/community", url = "https://dw290v42wisod.cloudfront.net/exts/kit/community" },
]

[settings.app.python]
interceptSysStdOutput = false
logSysStdOutput      = false

[settings.app.renderer]
skipWhileMinimized = false
sleepMsOnFocus     = 0
sleepMsOutOfFocus  = 0

[settings.app.exts]
folders = [
    "${exe-path}/exts",         # kit extensions
    "${exe-path}/extscore",     # kit core extensions
    "${exe-path}/../exts",      # isaac extensions
    "${exe-path}/../extsDeprecated",
    "${exe-path}/../extscache",
    "${exe-path}/../extsPhysics",
    "${exe-path}/../isaacsim/exts",
    "${exe-path}/../isaacsim/extsDeprecated",
    "${exe-path}/../isaacsim/extscache",
    "${exe-path}/../isaacsim/extsPhysics",
    "${app}",                   # 寻找其他 app 文件
    "${app}/../source",         # Isaac Lab 源码扩展
]

# 资源根路径（与两个文件一致）
persistent.isaac.asset_root.default = "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.5"
persistent.isaac.asset_root.cloud   = "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.5"
persistent.isaac.asset_root.nvidia  = "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.5"

And now the AppLauncher can successfully launch Isaacsim. But as soon as I click the StartAR button, and connect my vision pro, the whole system will get stuck. If anyone knows how to handle this problem, please tell me! Thank you!