It has been fixed by MGLRU patchset since kernel 6.1. Do:

    cat > /etc/tmpfiles.d/mglru-min-ttl.conf <<EOF
    w-      /sys/kernel/mm/lru_gen/enabled          -       -       -       -       y
    w-      /sys/kernel/mm/lru_gen/min_ttl_ms       -       -       -       -       1000
    EOF

and reboot. I've been struggling with this issue, as many others, for years. Now I can run two VMs with 8 GB physical RAM and 5+ GB swapped, and it barely noticeable.

More information, although a bit outdated (pre-MGLRU): https://notes.valdikss.org.ru/linux-for-old-pc-from-2007/en/... Linux issue%3A poor performance under RAM shortage conditions