xrdp occasionally generates SIGSEGV when resizing with RFX encoder

[roffery]

xrdp version

0.10.4-2

Detailed xrdp version, build options

xrdp 0.10.4
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2025 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=x86_64-redhat-linux-gnu
      --host=x86_64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-fuse
      --enable-pixman
      --enable-painter
      --enable-vsock
      --enable-ipv6
      --enable-utmp
      --with-socketdir=/run/xrdp
      --with-imlib2
      build_alias=x86_64-redhat-linux-gnu
      host_alias=x86_64-redhat-linux-gnu
      CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  
      LDFLAGS=-Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
      CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 1.1.1k  FIPS 25 Mar 2021

Operating system & version

RHEL 8 (EPEL RPM)

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xorgrdp

What desktop environment do you use?

Xfce

Environment xrdp running on

VM

What's your client?

Guacamole (via Firefox)

Area(s) with issue?

Crashes such as segfault

Steps to reproduce

1. Open Firefox
2. Set browser.fullscreen.autohide to true
3. Set Firefox window to full screen
4. Login to xrdp via Guacamole
5. Close tab (hover at top of window to show toolbar)
6. Login to xrdp via Guacamole (xrdp usually crashes at this point)

Repeat steps 5-6 if needed - usually crashes first time.

I can reproduce this if browser.fullscreen.autohide is set to true in Firefox. I suspect this is related to the window resizing that happens when switching tabs.

I was not able to reproduce this in xrdp 0.9.25 which I tested downgrading to. I did get disconnected on 0.9.25 after some resize events however.

✔️ Expected Behavior

No xrdp segfault

❌ Actual Behavior

xrdp segfaults

Anything else?

messages.txt

xrdp.log

[matt335672]

Hi @roffery

Thanks for raising this.

I can try to reproduce this on AlmaLinux 8. It will take me a bit of time to get to it, as we're trying to get something basic working on Wayland, which is a bit of a priority at the moment.

I've had a look at the logs you've provided to see if there's a possible short-term mitigation. Guacamole is using RFX which is not a common choice these days, and might be why we're not seeing this elsewhere. You could try disabling RFX. There might be a Guacamole setting, or try setting the max_bpp to 24 in /etc/xrdp/xrdp.ini.

A couple of questions for you:-

• Can you confirm you're using Guacamole on the same machine as xrdp?
• Are you using the EPEL version? If so, let me know what it is.
• Do you have the capability to build RPMs in-house?

Thanks in advance for the info.

[roffery]

Hi @matt335672

Thanks for getting back to me so soon. Completely understand regarding priorities.

Just some extra information:

• We already have max_bpp set to 24
• We're using the EPEL 8 xrdp, xrdp-selinux and xorgxrdp RPMs
• Guacamole is on a separate server to xrdp
• We do have the capability to build RPMs
• Also noting we are using Guacamole 1.6.0 which has recent changes to GFX/RFX: https://issues.apache.org/jira/browse/GUACAMOLE-377

After chatting with my team today we tested out setting the Guacamole disable-gfx option (info from upstream: https://guacamole.apache.org/doc/gug/configuring-guacamole.html#performance-flags). This looks to have disabled RemoteFX which I can see in the xrdp logs attached. After disabling GFX/RFX in Guacamole I can't reproduce the bug anymore (the performance isn't as good but this is OK as a workaround for our purposes).

xrdp.1.log

xrdp.2.log

[matt335672]

Thanks for getting back to me @roffery

That should give me enough information to get something sorted out here when I get round to it.

Are there any other changes to xrdp.ini I need to be aware of?

[roffery]

Hey @matt335672, I don't believe there's anything overly special in our xrdp.ini, but I've attached it for reference in case you need it.

xrdp.ini.txt

[matt335672]

@roffery - I'm currently unable to reproduce this on my rig.

I've installed Guacamole 1.6.0 on a Debian 12 machine, and I'm using it to connect to v0.10.4 on a Ubuntu 24.04 machine. I'm using fork=false and max_bpp=24 in my xrdp.ini

I'll have another go on Alma 8 to see if that behaves any differently. If I still can't get it I'll need to ask you to try some debugging for me.

I noticed a couple of things about your config which are worthy of note:-

1. I don't think fork=false is doing what you think it is. In this mode, xrdp simply uses a thread for each connection to the machine rather than a separate process. You can still have multiple connections to the machine.
2. You're not using runtime_user and runtime_group which are highly recommended for this version of xrdp for security purposes.

[matt335672]

Another reason I'm not seeing this is my Guacamole installation doesn't seem to be generating resize events - it's just scaling the display.

I suspect I need to set resize-method for the Guacamole client.

@roffery - can you provide me with a list of your client settings for Guacamole please?

[roffery]

Hey @matt335672, we are using resize-method=display-update.

Here are our other settings (N.B. we're using rdp_disable_gfx=true at the moment as workaround):

- scicore.guacamole.guacamole_connection:
    # <snip>
    max_connections_per_user: 2
    protocol: rdp
    port: 3389
    rdp_color_depth: '24'
    rdp_enable_full_window_drag: false
    rdp_server_layout: en-us-qwerty
    rdp_security: tls
    rdp_ignore_server_certs: true
    rdp_disable_gfx: false
    resize_method: display-update

[roffery]

Also thanks a lot for your suggestions - we'll look into changing those settings for fork= and runtime_user/runtime_group

[matt335672]

This took a while to reproduce on my rig.

I took the above config and placed it in Ubuntu 24.04 running v0.10.x (commit 559f948). The only change I made to the config was to change the logging info level from TRACE to INFO as this was really slowing things down.

I then used this command line with xfreerdp to connect in the same way as Guacamole which made things a lot easier:

xfreerdp  /ipv4:force  +rfx /gfx:rfx +dynamic-resolution /network:lan /u:testuser /d: /v:<host>

The /ipv4:force was needed as the posted config is IPv4 only. In order to use the RFX codec, the /network:lan switch was also required.

I then attached a debugger to xrdp (single process) and slowly increased the window height to generate lots of resize requests. SEGV received with the following stack trace:-

Thread 3 "xrdp" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x77dfe41ff6c0 (LWP 276247)]
0x00006102311f6971 in xrdp_mm_frame_ack (self=0x77dfdc09c2e0, frame_id=frame_id@entry=93) at xrdp_mm.c:3891
3891	    LOG_DEVEL(LOG_LEVEL_DEBUG, "xrdp_mm_frame_ack: "
(gdb) where
#0  0x00006102311f6971 in xrdp_mm_frame_ack (self=0x77dfdc09c2e0, frame_id=frame_id@entry=93) at xrdp_mm.c:3891
#1  0x00006102312089f8 in callback (id=106662182624368, msg=21847, param1=93, param2=0, param3=0, param4=0) at xrdp_wm.c:2062
#2  0x000077dfe56e4848 in xrdp_rdp_process_frame_ack (self=self@entry=0x77dfdc008c00, s=s@entry=0x610239819b90) at xrdp_rdp.c:1423
#3  0x000077dfe56e5180 in xrdp_rdp_process_data (self=self@entry=0x77dfdc008c00, s=s@entry=0x610239819b90) at xrdp_rdp.c:1584
#4  0x000077dfe5699216 in libxrdp_process_data (session=0x77dfdc008b80, s=s@entry=0x610239819b90) at libxrdp.c:243
#5  0x00006102311fc984 in xrdp_process_loop (self=self@entry=0x610239819c70, s=s@entry=0x610239819b90) at xrdp_process.c:76
#6  0x00006102311fcdab in xrdp_process_data_in (self=0x610239819ad0) at xrdp_process.c:205
#7  0x000077dfe572fee8 in trans_check_wait_objs (self=0x610239819ad0) at trans.c:444
#8  0x00006102311fd346 in xrdp_process_main_loop (self=self@entry=0x610239819c70) at xrdp_process.c:292
#9  0x00006102311e549f in xrdp_process_run (in_val=0x0) at xrdp_listen.c:153
#10 0x000077dfe4e9caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#11 0x000077dfe4f29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x

It seems what is happening is the RFX encoder is being deleted as part of the resize, then a frame ACK is received from the client which cannot be processed.

I was unable to create the same issue with H.264.

The following patch prevents the crash for me. It's also attached as patch.txt:-

--- a/xrdp/xrdp_mm.c
+++ b/xrdp/xrdp_mm.c
@@ -1459,22 +1459,33 @@ xrdp_mm_egfx_caps_advertise(void *user, int caps_count,
 static int
 xrdp_mm_update_module_frame_ack(struct xrdp_mm *self)
 {
-    int fif;
     struct xrdp_encoder *encoder;
 
     encoder = self->encoder;
-    fif = encoder->frames_in_flight;
-    if (encoder->frame_id_client + fif > encoder->frame_id_server)
+    if (encoder == NULL)
+    {
+        // Can't pass the ack to the encoder. Tell the module all
+        // frames are ACK'd
+        if (self->mod != NULL)
+        {
+            self->mod->mod_frame_ack(self->mod, 0, INT_MAX);
+        }
+    }
+    else
     {
-        if (encoder->frame_id_server > encoder->frame_id_server_sent)
+        int fif = encoder->frames_in_flight;
+        if (encoder->frame_id_client + fif > encoder->frame_id_server)
         {
-            LOG_DEVEL(LOG_LEVEL_DEBUG, "xrdp_mm_update_module_ack: "
-                      "frame_id_server %d", encoder->frame_id_server);
-            encoder->frame_id_server_sent = encoder->frame_id_server;
-            if (self->mod != NULL)
+            if (encoder->frame_id_server > encoder->frame_id_server_sent)
             {
-                self->mod->mod_frame_ack(self->mod, 0,
-                                         encoder->frame_id_server);
+                LOG_DEVEL(LOG_LEVEL_DEBUG, "xrdp_mm_update_module_ack: "
+                          "frame_id_server %d", encoder->frame_id_server);
+                encoder->frame_id_server_sent = encoder->frame_id_server;
+                if (self->mod != NULL)
+                {
+                    self->mod->mod_frame_ack(self->mod, 0,
+                                             encoder->frame_id_server);
+                }
             }
         }
     }
@@ -3887,22 +3898,30 @@ xrdp_mm_frame_ack(struct xrdp_mm *self, int frame_id)
     {
         return 1;
     }
-    encoder = self->encoder;
-    LOG_DEVEL(LOG_LEVEL_DEBUG, "xrdp_mm_frame_ack: "
-              "incoming %d, client %d, server %d", frame_id,
-              encoder->frame_id_client, encoder->frame_id_server);
-    if ((frame_id < 0) || (frame_id > encoder->frame_id_server))
+    if ((encoder = self->encoder) == NULL)
     {
-        /* if frame_id is negative or bigger then what server last sent
-           just ack all sent frames */
-        /* some clients can send big number just to clear all
-           pending frames */
-        encoder->frame_id_client = encoder->frame_id_server;
+        /* No encoder - Possibly a late frame ack with a resize in progress */
+        LOG_DEVEL(LOG_LEVEL_INFO, "xrdp_mm_frame_ack: "
+                  "Frame ack incoming %d with no encoder!", frame_id);
     }
     else
     {
-        /* frame acks can come out of order so ignore older one */
-        encoder->frame_id_client = MAX(frame_id, encoder->frame_id_client);
+        LOG_DEVEL(LOG_LEVEL_DEBUG, "xrdp_mm_frame_ack: "
+                  "incoming %d, client %d, server %d", frame_id,
+                  encoder->frame_id_client, encoder->frame_id_server);
+        if ((frame_id < 0) || (frame_id > encoder->frame_id_server))
+        {
+            /* if frame_id is negative or bigger then what server last sent
+               just ack all sent frames */
+            /* some clients can send big number just to clear all
+               pending frames */
+            encoder->frame_id_client = encoder->frame_id_server;
+        }
+        else
+        {
+            /* frame acks can come out of order so ignore older one */
+            encoder->frame_id_client = MAX(frame_id, encoder->frame_id_client);
+        }
     }
     xrdp_mm_update_module_frame_ack(self);
     return 0;

@roffery - are you able to test this patch as-is? I can give you a hand with an RPM rebuild if necessary.

patch.txt

[roffery]

Hi @matt335672, thanks a lot!

I've built a local RPM as xrdp-0.10.4-3 (+ selinux etc.) with your patch and installed that, and re-enabled GFX on my Guacamole connection. I can't reproduce the issue with resize events now working. I was immediately able to reproduce the issue on downgrading to the EPEL xrdp-0.10.4-2.

I'll continue to use the local RPM over coming weeks.

Cheers