Skip to content

Implement staging buffer and use it for the material system #1675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

VReaperV
Copy link
Contributor

@VReaperV VReaperV commented Jun 8, 2025

Adds GLStagingBuffer and GLBufferCopy(). GLStagingBuffer allows setting buffer storage and map flags to 0 on the destination buffers, which pretty much guarantees that they will go into VRAM. The data is uploaded through mapping stagingBuffer and doing a copy of the data on the GPU.

The stagingBuffer itself should (still depends on the driver) go into the BAR memory (on Nvidia this shows up as DMA_CACHED when using r_glDebugProfile on; r_glDebugMode 7).

Slightly improves material system performance. Currently this is used for all material system buffers except the portal one (since it requires reading from the GPU), and for the 2 geometry cache buffers that are currently used.

Also fixed GLBuffer.mapped state being retained on map change/load.

@VReaperV VReaperV added T-Bug T-Improvement Improvement for an existing feature A-Renderer T-Performance labels Jun 8, 2025
@VReaperV VReaperV moved this to In Progress in Material system Jun 8, 2025
@VReaperV VReaperV force-pushed the staging-buffer branch 3 times, most recently from 96d8c0d to 88c7d08 Compare June 8, 2025 09:03
@VReaperV
Copy link
Contributor Author

VReaperV commented Jun 8, 2025

The CI error is caused by some unrelated bullshit.

@slipher
Copy link
Member

slipher commented Jun 8, 2025

FAILED: daemon 
: && /usr/bin/g++ -std=gnu++14 -fvisibility=hidden -fno-strict-aliasing -fno-gnu-unique -pthread -fPIC -march=x86-64 -mtune=generic -mcx16 -O3 -DNDEBUG -D_FORTIFY_SOURCE=2 -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now CMakeFiles/client-objects.dir/GeneratedSource/DaemonBuildInfo/Engine.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/cmd.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/common.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/crypto.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/cvar.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/files.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/huffman.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/msg.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/net_chan.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/net_ip.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/translation.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_bot.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_ccmds.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_client.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_init.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_main.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_net_chan.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_sgame.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_snapshot.cpp.o CMakeFiles/client-objects.dir/src/engine/server/CryptoChallenge.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_avi.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_cgame.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_console.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_download.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_input.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_main.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_parse.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_scrn.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_serverlist.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_serverstatus.cpp.o CMakeFiles/client-objects.dir/src/engine/client/dl_main.cpp.o CMakeFiles/client-objects.dir/src/engine/client/hunk_allocator.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/ALObjects.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Audio.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Emitter.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/OggCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/OpusCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Sample.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Sound.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/SoundCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/WavCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_keys.cpp.o CMakeFiles/client-objects.dir/src/engine/client/key_binding.cpp.o CMakeFiles/client-objects.dir/src/engine/client/key_identification.cpp.o CMakeFiles/client-objects.dir/src/engine/sys/sdl_input.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/DetectGLVendors.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/gl_shader.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/shaders.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_animation.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_backend.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_bsp.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_cmds.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_curve.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_fbo.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_font.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GeometryCache.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GeometryOptimiser.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GLMemory.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/InternalImage.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/Material.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/TextureManager.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_crn.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_dds.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_jpg.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_ktx.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_png.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_tga.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_webp.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_init.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_light.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_main.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_marks.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_mesh.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_iqm.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_md3.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_md5.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_skel.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_noise.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_scene.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shade.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shader.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shade_calc.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_skin.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_sky.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_surface.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_vbo.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_video.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_world.cpp.o CMakeFiles/client-objects.dir/src/engine/sys/sdl_glimp.cpp.o CMakeFiles/client.dir/src/engine/client/ClientApplication.cpp.o -o daemon  libengine-lib.a  -lncursesw  /usr/lib/x86_64-linux-gnu/libformw.so  /usr/lib/x86_64-linux-gnu/libGL.so  /usr/lib/x86_64-linux-gnu/libSDL2main.a  /usr/lib/x86_64-linux-gnu/libSDL2.so  /usr/lib/x86_64-linux-gnu/libogg.so  /usr/lib/x86_64-linux-gnu/libvorbisfile.so  /usr/lib/x86_64-linux-gnu/libvorbis.so  /usr/lib/libopusfile.so  /usr/lib/x86_64-linux-gnu/libopus.so  /usr/lib/x86_64-linux-gnu/libogg.so  /usr/lib/x86_64-linux-gnu/libvorbisfile.so  /usr/lib/x86_64-linux-gnu/libvorbis.so  /usr/lib/libopusfile.so  /usr/lib/x86_64-linux-gnu/libopus.so  /usr/lib/x86_64-linux-gnu/libwebp.so  /usr/lib/x86_64-linux-gnu/libjpeg.so  /usr/lib/x86_64-linux-gnu/libpng.so  /usr/lib/x86_64-linux-gnu/libfreetype.so  /usr/lib/x86_64-linux-gnu/libGLEW.so  /usr/lib/x86_64-linux-gnu/libopenal.so  /usr/lib/x86_64-linux-gnu/libcurl.so  libsrclibs-mumblelink.a  libsrclibs-findlocale.a  /usr/lib/x86_64-linux-gnu/libhogweed.so  /usr/lib/x86_64-linux-gnu/libnettle.so  /usr/lib/x86_64-linux-gnu/libgmp.so  libsrclibs-nacl-native.a  -lm  /usr/lib/x86_64-linux-gnu/librt.a  -ldl  /usr/lib/x86_64-linux-gnu/libz.so  libsrclibs-minizip.a && :
/usr/bin/ld: CMakeFiles/client-objects.dir/src/engine/renderer/GLMemory.cpp.o: in function `GLStagingBuffer::MapBuffer(long)':
GLMemory.cpp:(.text+0x555): undefined reference to `GLStagingBuffer::SIZE'

That sounds related

@VReaperV
Copy link
Contributor Author

VReaperV commented Jun 8, 2025

FAILED: daemon 
: && /usr/bin/g++ -std=gnu++14 -fvisibility=hidden -fno-strict-aliasing -fno-gnu-unique -pthread -fPIC -march=x86-64 -mtune=generic -mcx16 -O3 -DNDEBUG -D_FORTIFY_SOURCE=2 -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now CMakeFiles/client-objects.dir/GeneratedSource/DaemonBuildInfo/Engine.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/cmd.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/common.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/crypto.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/cvar.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/files.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/huffman.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/msg.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/net_chan.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/net_ip.cpp.o CMakeFiles/client-objects.dir/src/engine/qcommon/translation.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_bot.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_ccmds.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_client.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_init.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_main.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_net_chan.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_sgame.cpp.o CMakeFiles/client-objects.dir/src/engine/server/sv_snapshot.cpp.o CMakeFiles/client-objects.dir/src/engine/server/CryptoChallenge.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_avi.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_cgame.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_console.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_download.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_input.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_main.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_parse.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_scrn.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_serverlist.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_serverstatus.cpp.o CMakeFiles/client-objects.dir/src/engine/client/dl_main.cpp.o CMakeFiles/client-objects.dir/src/engine/client/hunk_allocator.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/ALObjects.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Audio.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Emitter.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/OggCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/OpusCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Sample.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/Sound.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/SoundCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/audio/WavCodec.cpp.o CMakeFiles/client-objects.dir/src/engine/client/cl_keys.cpp.o CMakeFiles/client-objects.dir/src/engine/client/key_binding.cpp.o CMakeFiles/client-objects.dir/src/engine/client/key_identification.cpp.o CMakeFiles/client-objects.dir/src/engine/sys/sdl_input.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/DetectGLVendors.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/gl_shader.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/shaders.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_animation.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_backend.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_bsp.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_cmds.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_curve.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_fbo.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_font.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GeometryCache.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GeometryOptimiser.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/GLMemory.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/InternalImage.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/Material.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/TextureManager.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_crn.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_dds.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_jpg.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_ktx.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_png.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_tga.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_image_webp.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_init.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_light.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_main.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_marks.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_mesh.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_iqm.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_md3.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_md5.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_model_skel.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_noise.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_scene.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shade.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shader.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_shade_calc.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_skin.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_sky.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_surface.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_vbo.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_video.cpp.o CMakeFiles/client-objects.dir/src/engine/renderer/tr_world.cpp.o CMakeFiles/client-objects.dir/src/engine/sys/sdl_glimp.cpp.o CMakeFiles/client.dir/src/engine/client/ClientApplication.cpp.o -o daemon  libengine-lib.a  -lncursesw  /usr/lib/x86_64-linux-gnu/libformw.so  /usr/lib/x86_64-linux-gnu/libGL.so  /usr/lib/x86_64-linux-gnu/libSDL2main.a  /usr/lib/x86_64-linux-gnu/libSDL2.so  /usr/lib/x86_64-linux-gnu/libogg.so  /usr/lib/x86_64-linux-gnu/libvorbisfile.so  /usr/lib/x86_64-linux-gnu/libvorbis.so  /usr/lib/libopusfile.so  /usr/lib/x86_64-linux-gnu/libopus.so  /usr/lib/x86_64-linux-gnu/libogg.so  /usr/lib/x86_64-linux-gnu/libvorbisfile.so  /usr/lib/x86_64-linux-gnu/libvorbis.so  /usr/lib/libopusfile.so  /usr/lib/x86_64-linux-gnu/libopus.so  /usr/lib/x86_64-linux-gnu/libwebp.so  /usr/lib/x86_64-linux-gnu/libjpeg.so  /usr/lib/x86_64-linux-gnu/libpng.so  /usr/lib/x86_64-linux-gnu/libfreetype.so  /usr/lib/x86_64-linux-gnu/libGLEW.so  /usr/lib/x86_64-linux-gnu/libopenal.so  /usr/lib/x86_64-linux-gnu/libcurl.so  libsrclibs-mumblelink.a  libsrclibs-findlocale.a  /usr/lib/x86_64-linux-gnu/libhogweed.so  /usr/lib/x86_64-linux-gnu/libnettle.so  /usr/lib/x86_64-linux-gnu/libgmp.so  libsrclibs-nacl-native.a  -lm  /usr/lib/x86_64-linux-gnu/librt.a  -ldl  /usr/lib/x86_64-linux-gnu/libz.so  libsrclibs-minizip.a && :
/usr/bin/ld: CMakeFiles/client-objects.dir/src/engine/renderer/GLMemory.cpp.o: in function `GLStagingBuffer::MapBuffer(long)':
GLMemory.cpp:(.text+0x555): undefined reference to `GLStagingBuffer::SIZE'

That sounds related

Oh right, it's because of the moronic behaviour with static const in class declarations.

@VReaperV
Copy link
Contributor Author

VReaperV commented Jun 8, 2025

Well no, it's gotta be caused by something else too, because even with constexpr it still fails.

VReaperV added 3 commits June 8, 2025 13:19
Adds `GLStagingBuffer` and `GLBufferCopy()`. `GLStagingBuffer` allows setting buffer storage and map flags to 0 on the destination buffers, which pretty much guarantees that they will go into VRAM. The data is uploaded through mapping `stagingBuffer` and doing a copy of the data on the GPU.
@slipher
Copy link
Member

slipher commented Jun 8, 2025

Yeah I don't think constexpr fixed anything for that case; it just behaves the same as const. In general you need C++17 inline constexpr for constants that truly behave as they ought to

@VReaperV
Copy link
Contributor Author

VReaperV commented Jun 8, 2025

Yeah I don't think constexpr fixed anything for that case; it just behaves the same as const. In general you need C++17 inline constexpr for constants that truly behave as they ought to

Yeah, it's defaulting to C++14 apparently.

@VReaperV
Copy link
Contributor Author

VReaperV commented Jun 8, 2025

It'll just have to do with the old redundant way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Renderer T-Bug T-Improvement Improvement for an existing feature T-Performance
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

2 participants