fix(drm): turn a captured frame only when the plane did not rotate it (#16345)

* fix(drm): turn a captured frame only when the plane did not rotate it

A compositor rotates an output either in hardware, setting the primary
plane's `rotation` property, or in software, drawing the scanout already
turned. `wl_output` cannot tell the two apart, and `frame_transform`
guessed: 90/270 turned, 180 left alone, which is right for i915 + mutter
(rotate-180 in hardware, scanout upright) and wrong wherever the
compositor drew the scanout turned, where 180 comes out upside down.

Measured before changing it. virtio-gpu (no rotation property, mutter 46):
the scanout dump at 180 is the desktop upside down, at 90 and 270 the
logical desktop turned inside the native mode framebuffer. i915 + mutter
(GNOME 50) at 180: the plane reports rotate-180 and the dump is upright,
identical to the unrotated one. amdgpu + KWin 6 at 180: the plane stays at
rotate-0 and the dump is upside down.

The producer reads `drmtap_plane_rotation()` (libdrmtap 0.5.8, optional
symbol) right after each grab and stamps it on the frame, in the dma-buf
descriptor and in the CPU frame header, both `serde(default)` so an older
producer still parses. A plane that rotated scans out the logical desktop
whatever the output transform says: the compositor programs it from the
CRTC transform, which also folds in the connector's panel orientation that
`wl_output` never carries, so the consumer leaves such a frame alone. A
plane at rotate-0 scanned out what the compositor drew, and the frame is
turned by the whole output transform. `None` (a library before 0.5.8, or
nothing it could read) keeps the previous rule, so nothing changes until
the library says otherwise. The session is still sized from the output
transform: a plane that rotated 90 in hardware hands over the portrait
scanout those dims already name.

* build: pin libdrmtap 0.5.8 (rustdesk-org/libdrmtap 95d4d7454)

The rotation fix reads drmtap_plane_rotation(), added in libdrmtap 0.5.8. The mirror carries that commit since 2026-09-25, so the pin moves there.

* drm: pin the decoding of a frame from a producer older than the plane rotation

The root service and the --server are upgraded separately, so a frame
header or a dma-buf descriptor without `plane_rotation` must still
decode, as None, which keeps the previous rule. The tests built both
messages with the field set to None; this one decodes payloads that omit
it, and one that carries it (greptile on #16345). Serde already reads a
missing Option as None, so what the test pins is the wire name and the
None on absence, for both messages.

Mutation: renaming the field on the wire, in either message, turns it red.

* drm: take the plane rotation right after the grab, not after the copies

`drmtap_plane_rotation()` reads the rotation property of the plane the
last grab came from at the moment it is called. The cpu path asked only
after `DrmReader::grab()` had copied the frame into its buffer and the
producer had copied it again into the message, so on a large frame the
compositor had time to change the plane and the frame went out with the
rotation of the next one; `frame_transform` turns that frame by it with
nothing to absorb the skew (zhou, review of 26-sep).

`DrmReader` now reads the rotation right after a successful
`drmtap_grab_mapped` or `drmtap_grab_desc`, before any copy or release,
and `plane_rotation()` returns that snapshot, so both paths stamp a frame
with the rotation its own grab saw.

Test: a_frame_keeps_the_rotation_its_plane_had_when_it_was_grabbed, in
`plane_rotation_tests` (which CI already runs), drives the reader through
a fake library whose frame release turns the plane: the rotation read at
the grab survives on the cpu grab and on the dma-buf export. Mutations:
no snapshot on the cpu grab, none on the export, and the cpu snapshot
taken after the release, each turn it red.

libs/scrap/src/common/drm_reader.rs

@@ -377,6 +377,9 @@ pub struct DrmReader {
     /// alternation and log on every sample, which is the per-frame logging this was written to
     /// avoid. -1 is "nothing reported yet".
     last_provenance: i8,
+    /// The plane rotation read right after the last successful grab, before any copy: libdrmtap
+    /// reads the property when asked, so a read after the copies could describe the next frame.
+    last_plane_rotation: Option<u32>,
 }
 
 impl DrmReader {
@@ -414,6 +417,7 @@ impl DrmReader {
             lib,
             ctx,
             buf: Vec::new(),
+            last_plane_rotation: None,
         })
     }
 
@@ -437,6 +441,7 @@ impl DrmReader {
                     format!("drmtap_grab_mapped failed: errno {errno}"),
                 ));
             }
+            self.last_plane_rotation = self.read_plane_rotation();
             if frame.data.is_null() || frame.width == 0 || frame.height == 0 {
                 (self.lib.frame_release)(self.ctx, &mut frame);
                 return Err(io::ErrorKind::WouldBlock.into());
@@ -531,6 +536,23 @@ impl DrmReader {
             .map(|s| s.to_owned())
     }
 
+    /// The DRM `rotation` bitmask of the plane the last successful grab read from, as it was right
+    /// after that grab. A plane without the property answers rotate-0: it cannot have turned
+    /// anything, so the compositor drew the scanout already turned and the whole output transform
+    /// is still to be undone. `None` only when the library cannot say: it predates the call
+    /// (0.5.8), no plane is bound, or the property set could not be read.
+    pub fn plane_rotation(&self) -> Option<u32> {
+        self.last_plane_rotation
+    }
+
+    fn read_plane_rotation(&mut self) -> Option<u32> {
+        let f = self.lib.plane_rotation?;
+        let mut rotation: u32 = 0;
+        // SAFETY: self.ctx is a live context and `rotation` outlives the call.
+        let rc = unsafe { f(self.ctx, &mut rotation) };
+        plane_rotation_answer(rc, rotation)
+    }
+
     /// Zero-copy EXPORT grab: fills a `drmtap_dmabuf_desc` (dma-buf fd, plane layout, HDR metadata) WITHOUT mapping, detiling or copying pixels, so on this
     /// path the root process never loads libEGL/libGLESv2. The exported fd is READ-ONLY (libdrmtap drops `DRM_RDWR` and `dup` shares that open file
     /// description), so the `--server` that receives it can map the scanout but never write the live framebuffer. Validation here is METADATA ONLY.
@@ -562,6 +584,7 @@ impl DrmReader {
                     format!("drmtap_grab_desc failed: errno {errno}"),
                 ));
             }
+            self.last_plane_rotation = self.read_plane_rotation();
             // `desc.dma_buf_fd` is the canonical fd (what split_capture.c sends); `frame` owns it too and `frame_release` closes the library's copy.
             let raw_fd = if desc.dma_buf_fd >= 0 {
                 desc.dma_buf_fd
@@ -985,3 +1008,97 @@ mod hotspot_guess_tests {
         assert_eq!(infer_hotspot(&px, w, h), (0, 0));
     }
 }
+
+/// DRM_MODE_ROTATE_0: the `rotation` bitmask of a plane that turned nothing.
+const DRM_MODE_ROTATE_0: u32 = 1 << 0;
+
+/// What a `drmtap_plane_rotation` return means to the consumer. Success is the mask. `-ENOTSUP`
+/// (the plane has no `rotation` property) is rotate-0, not "unknown": such a plane cannot have
+/// turned the scanout, so the frame arrives turned by the whole output transform, which is the
+/// virtio-gpu and vmwgfx case this exists for. Anything else (`-ENOENT` with no plane bound, a
+/// failed property read) is `None`, and the consumer keeps its pre-0.5.8 rule.
+fn plane_rotation_answer(rc: c_int, rotation: u32) -> Option<u32> {
+    if rc == 0 {
+        Some(rotation)
+    } else if rc == -hbb_common::libc::ENOTSUP {
+        Some(DRM_MODE_ROTATE_0)
+    } else {
+        None
+    }
+}
+
+#[cfg(test)]
+mod plane_rotation_tests {
+    use super::*;
+    use std::os::fd::AsRawFd;
+    use std::sync::atomic::{AtomicI32, AtomicU32, Ordering};
+
+    #[test]
+    fn a_plane_without_the_property_turned_nothing() {
+        assert_eq!(plane_rotation_answer(0, 0x4), Some(0x4));
+        assert_eq!(plane_rotation_answer(0, 0x1), Some(0x1));
+        assert_eq!(plane_rotation_answer(-hbb_common::libc::ENOTSUP, 0), Some(0x1));
+        assert_eq!(plane_rotation_answer(-hbb_common::libc::ENOENT, 0), None);
+        assert_eq!(plane_rotation_answer(-hbb_common::libc::EINVAL, 0), None);
+        assert_eq!(plane_rotation_answer(-hbb_common::libc::EIO, 7), None);
+    }
+
+    // What the plane answers whenever it is asked. The fake release turns it, the way a compositor
+    // may once the frame is out, so a read taken after the release or outside the grab sees 0x1.
+    static PLANE: AtomicU32 = AtomicU32::new(0x1);
+    static FD: AtomicI32 = AtomicI32::new(-1);
+    static PIXELS: [u8; 16] = [0; 16];
+
+    unsafe extern "C" fn plane_rotation(_: *mut drmtap_ctx, rotation: *mut u32) -> c_int {
+        *rotation = PLANE.load(Ordering::SeqCst);
+        0
+    }
+    unsafe extern "C" fn frame_release(_: *mut drmtap_ctx, _: *mut drmtap_frame_info) {
+        PLANE.store(0x1, Ordering::SeqCst);
+    }
+    unsafe extern "C" fn grab_mapped(_: *mut drmtap_ctx, f: *mut drmtap_frame_info) -> c_int {
+        (*f).data = PIXELS.as_ptr() as *mut _;
+        (*f).width = 2;
+        (*f).height = 2;
+        (*f).stride = 8;
+        (*f).format = DRM_FORMAT_XRGB8888;
+        0
+    }
+    unsafe extern "C" fn grab_desc(
+        _: *mut drmtap_ctx,
+        d: *mut drmtap_dmabuf_desc,
+        _: *mut drmtap_frame_info,
+    ) -> c_int {
+        (*d).dma_buf_fd = FD.load(Ordering::SeqCst);
+        (*d).width = 2;
+        (*d).height = 2;
+        (*d).num_planes = 1;
+        (*d).pitches[0] = 8;
+        0
+    }
+
+    #[test]
+    fn a_frame_keeps_the_rotation_its_plane_had_when_it_was_grabbed() {
+        let lib: &'static DrmtapLib = Box::leak(Box::new(DrmtapLib::fake(
+            grab_mapped,
+            frame_release,
+            grab_desc,
+            plane_rotation,
+        )));
+        let mut r = DrmReader {
+            lib,
+            ctx: std::ptr::null_mut(),
+            buf: Vec::new(),
+            last_provenance: -1,
+            last_plane_rotation: None,
+        };
+        PLANE.store(0x4, Ordering::SeqCst);
+        r.grab().expect("the fake grab succeeds");
+        assert_eq!(r.plane_rotation(), Some(0x4), "cpu grab");
+        let (fd, _peer) = std::os::unix::net::UnixStream::pair().unwrap();
+        FD.store(fd.as_raw_fd(), Ordering::SeqCst);
+        PLANE.store(0x8, Ordering::SeqCst);
+        r.grab_desc().expect("the fake export succeeds");
+        assert_eq!(r.plane_rotation(), Some(0x8), "dma-buf export");
+    }
+}

libs/scrap/src/common/drmtap_dl.rs

@@ -140,6 +140,12 @@ type FnGrabDesc =
 type FnOpenRender = unsafe extern "C" fn(*const c_char) -> *mut drmtap_ctx;
 // libdrmtap >= 0.4.15; returns a ctx-owned string, or NULL if it has none.
 type FnRenderNode = unsafe extern "C" fn(*mut drmtap_ctx) -> *const c_char;
+/// `drmtap_plane_rotation`, added in libdrmtap 0.5.8: the DRM `rotation` bitmask the primary
+/// plane scans out with, read now (0x1 = 0, 0x2 = 90, 0x4 = 180, 0x8 = 270, plus 0x10/0x20 for
+/// a reflection). 0 with `*rotation` set; `-ENOTSUP` when the plane has no such property, which
+/// means the compositor can only have rotated in software; `-ENOENT` with no plane bound;
+/// `-EINVAL` on a null argument.
+type FnPlaneRotation = unsafe extern "C" fn(*mut drmtap_ctx, *mut u32) -> c_int;
 type FnConvertDmabuf =
     unsafe extern "C" fn(*mut drmtap_ctx, *const drmtap_dmabuf_desc, *mut drmtap_frame_info) -> c_int;
 
@@ -162,9 +168,64 @@ pub struct DrmtapLib {
     pub open_render: FnOpenRender,
     pub convert_dmabuf: FnConvertDmabuf,
     pub render_node: Option<FnRenderNode>,
+    /// Optional: it only exists from libdrmtap 0.5.8. Absent means the library cannot say whether
+    /// the plane rotated the scanout, and the consumer keeps the pre-0.5.8 rule for that case.
+    pub plane_rotation: Option<FnPlaneRotation>,
     pub version: (c_int, c_int, c_int),
 }
 
+/// A library whose capture entry points are the given fakes and whose others do nothing, for the
+/// tests of the reader.
+#[cfg(test)]
+impl DrmtapLib {
+    pub(crate) fn fake(
+        grab_mapped: FnGrabMapped,
+        frame_release: FnFrameRelease,
+        grab_desc: FnGrabDesc,
+        plane_rotation: FnPlaneRotation,
+    ) -> Self {
+        unsafe extern "C" fn open(_: *const drmtap_config) -> *mut drmtap_ctx {
+            std::ptr::null_mut()
+        }
+        unsafe extern "C" fn close(_: *mut drmtap_ctx) {}
+        unsafe extern "C" fn list_displays(_: *mut drmtap_ctx, _: *mut drmtap_display, _: c_int) -> c_int {
+            0
+        }
+        unsafe extern "C" fn get_cursor(_: *mut drmtap_ctx, _: *mut drmtap_cursor_info) -> c_int {
+            -1
+        }
+        unsafe extern "C" fn cursor_release(_: *mut drmtap_ctx, _: *mut drmtap_cursor_info) {}
+        unsafe extern "C" fn open_render(_: *const c_char) -> *mut drmtap_ctx {
+            std::ptr::null_mut()
+        }
+        unsafe extern "C" fn convert_dmabuf(
+            _: *mut drmtap_ctx,
+            _: *const drmtap_dmabuf_desc,
+            _: *mut drmtap_frame_info,
+        ) -> c_int {
+            -1
+        }
+        DrmtapLib {
+            _lib: hbb_common::libloading::os::unix::Library::this().into(),
+            open,
+            close,
+            list_displays,
+            list_devices: None,
+            grab_mapped,
+            frame_release,
+            get_cursor,
+            cursor_release,
+            cursor_hotspot_valid: None,
+            grab_desc,
+            open_render,
+            convert_dmabuf,
+            render_node: None,
+            plane_rotation: Some(plane_rotation),
+            version: (0, 5, 8),
+        }
+    }
+}
+
 // SAFETY: the resolved fn pointers are plain C entry points with no interior mutability;
 // libdrmtap contexts are used single-threaded by the caller. The Library handle is never moved out.
 unsafe impl Send for DrmtapLib {}
@@ -290,6 +351,8 @@ impl DrmtapLib {
                 lib.get(b"drmtap_render_node").ok().map(|s| *s);
             let cursor_hotspot_valid: Option<FnCursorHotspotValid> =
                 lib.get(b"drmtap_cursor_hotspot_valid").ok().map(|s| *s);
+            let plane_rotation: Option<FnPlaneRotation> =
+                lib.get(b"drmtap_plane_rotation").ok().map(|s| *s);
             // Log the load only now that every required symbol resolved: this fn still returns None on a missing one.
             let loaded_from = real
                 .as_ref()
@@ -333,6 +396,14 @@ impl DrmtapLib {
                      a driver that really puts the hotspot at (0, 0)."
                 );
             }
+            if (minor, patch) >= (5, 8) && plane_rotation.is_none() {
+                log::warn!(
+                    "libdrmtap at {loaded_from} reports v{major}.{minor}.{patch} but is missing \
+                     drmtap_plane_rotation: it is a stale or pre-release build. Check what the \
+                     soname symlink points at. A 180-degree output rotated by the compositor in \
+                     software will be captured upside down."
+                );
+            }
             Some(DrmtapLib {
                 _lib: lib,
                 open,
@@ -348,6 +419,7 @@ impl DrmtapLib {
                 open_render,
                 convert_dmabuf,
                 render_node,
+                plane_rotation,
                 version: (major, minor, patch),
             })
         }