Running x86/x86-64 applications on Fedora Asahi Remix

There are lots of lots of legacy x86/x86-64 applications that users want to run on arm64 platforms, including Windows applications and games. To support this in Fedora Asahi Remix, we have integrated a stack of existing and bespoke components to make it possible to transparently run x86/x86-64 apps directly on arm64 Linux.

Since Apple platforms use a 16K page size natively and x86/x86-64 processors use a 4K page size, this is especially tricky, as x86/x86-64 applications generally do not work when presented with a host kernel that requires 16K page alignment. To bridge this gap, we are using a microVM to run an entirely separate guest Linux kernel in 4K page size mode. To keep it as seamless as possible, the guest environment is designed to be as close as possible to the host environment, and we use native context GPU passthrough to have high-performance graphics inside the guest.

The stack consists of these components:

  • muvm (package: muvm), our bespoke microVM runner based on libkrun. This also includes components for X11 forwarding and HID input device proxying.

  • FEX-emu (package: fex-emu), a fast userspace x86/x86-64 emulator focused on correctness.

  • The Fedora FEX RootFS (package: fex-emu-rootfs-fedora), which provides common x86/x86-64 library dependencies to be used by emulated applications.

  • mesa (packages: mesa-fex-emu-overlay-i386 and mesa-fex-emu-overlay-x86-64), built for the x86/x86-64 architectures and packaged as a FEX RootFS overlay. This provides the OpenGL/OpenCL/Vulkan support for Apple GPUs.

We also have our own Steam wrapper that automates the process of installing and launching Steam inside the microVM stack. When running Windows games using Steam, these open-source components are used behind the scenes:

  • Proton, a Wine distribution aimed at gaming.

  • dxvk, a translation layer that converts the Windows DirectX 8 - DirectX 11 APIs to Vulkan.

  • vkd3d-proton, a translation layer that converts the Windows DirectX 12 API to Vulkan.

Scope

This technology stack is primarily aimed at running x86 and x86-64 games, but it can also be used to run non-game productivity applications. When possible, you should prefer native alternatives over emulation. Please read this FAQ entry for more information.

The scope of this solution is limited to portable x86 and x86-64 applications that are intended to be run from your home directory (or, at most, manually unpacked into /opt by the user), including AppImages. It is not intended to run x86-64 applications that must be installed as system packages, which are built for a specific Linux distribution or require installation of complex system dependencies, or which require running dedicated installers as root.

In particular, the x86-64 environment is not a self-contained root filesystem, but rather a minimal, immutable overlay on top of your existing arm64 root filesystem. That means that you cannot make changes to it, install additional packages, etc. There is no root access available at all within the x86-64 environment.

Usage

Steam

Just use dnf install steam to install our Steam wrapper, and then run Steam from your desktop’s launcher (or the steam command) to download and install Steam. This will install all necessary dependencies automatically.

Other applications

To install the emulation stack by itself, use dnf install fex-emu. This will pull in the required dependencies automatically.

You cannot run x86-64 applications directly from the host (yet), as they must be launched from the microVM. To do so, run muvm -- /path/to/executable. You must use an absolute path, as muvm does not currently preserve the current working directory. In this environment, the kernel’s binfmt support is already configured to use FEX to run x86/x86-64 applications, so you should be able to just run them.

If your application uses a launcher shell script instead of directly running its main binary, you should run it through FEXBash. For example, use muvm -- FEXBash /path/to/launcher.sh. Doing so ensures that the shell runs in the emulated environment and a few critical shell commands behave as they would for x86-64 applications, which makes it more likely that the shell script will work as intended.

You can also use muvm -- bash to launch an arm64 shell within the 4K MicroVM, or muvm -- FEXBash to launch an x86-64 shell. The x86-64 shell will behave similarly to the arm64 shell and most commands will run as arm64 binaries, but a few (such as ls) will run under emulation, which lets you "see" the world as x86-64 apps do.

How it works

muvm creates a virtual machine that shares as much with the host OS as possible. Within the VM, the root filesystem is the same as the host root filesystem, with the following exceptions:

  • /dev, /sys, and /proc are guest-private, except for /dev/shm which is shared with the host, allowing host apps and guest apps to coherently share memory.

  • /run is also private to the guest

  • The FEX-emu rootfs and overlay images are mounted under /run/fex-emu/, with the combined overlay rootfs available at /run/fex-emu/rootfs.

  • /usr/share/fex-emu and /usr/local/share/fex-emu are overmounted with a tmpfs to inject a FEX Config.json suitable for use within the VM

  • A tmpfs is also mounted on /tmp/.X11-unix, so X11 server sockets are private to the VM

  • The entire host filesystem view is available at /run/muvm-host, including any overlaid mounts. For example, you can access the host’s /run at /run/muvm-host/run. (Note: /run/muvm-host/dev exists but will not do what you might hope it does. Host devices are not available in the guest.)

This means that /usr, /home, /etc, /opt, /var, /tmp, and any other directories in your filesystem root are shared between the guest and the host. The aarch64 guest OS does not run its own root filesystem, but rather runs exactly the same binaries as your host OS does.

Additionally, FEX itself uses the filesystem mounted at /run/fex-emu/rootfs as its virtual RootFS. This means that x86/x86-64 applications (and only those) will see the contents of that directory overlaid on top of the root filesystem. This is how we make x86/x86-64 libraries available to those applications, while still sharing most of the filesystem contents.

When muvm starts, it registers FEX as a binfmt provider, so x86/x86-64 applications will be transparently run through it. On startup, FEX will detect that TSO support is available on the Apple Silicon platform (even within the VM), and automatically enable it for faster accurate emulation.

Mountpoints in the host are propagated to the guest when they are first accessed, automagically. This allows guest apps to distinguish different filesystems, which keeps device/inode semantics correct. If you have a partition mounted on the host at /mnt/steam and you run mount within the guest, you won’t see it at first. If you run ls /mnt/steam and then run mount again, the mount will have magically been added to the mount list. This is normal and working as intended!

Known issues

Performance isn’t very good

As this project is still in its early stages, we aimed for correctness for the initial release. Performance optimization will happen over time. We’re aware of several changes that should bring significant performance improvements, and we’re actively working on it!

For Windows DX8-DX11 games under Proton in particular, you might want to try WineD3D instead of DXVK. WineD3D uses OpenGL instead of Vulkan as its backend, and it may have better performance thanks to optimizations in our OpenGL driver that are not available on Vulkan. To enable it, change the Steam launch options to PROTON_USE_WINED3D=1 %command%. Note that DXVK tends to have better compatibility, so this is a trade-off. Let us know what games work better using either backend!

Older 32-bit games may run very slowly if they make heavy use of the 80-bit x87 floating-point unit, since these operations have to be emulated in software for full compatibility (the same issue exists in Rosetta on macOS). You can run these games with hardware-based 64-bit floating-point emulation, which is less accurate but much faster. To do so, change the Steam launch options to FEX_X87REDUCEDPRECISION=1 %command%. This mode can cause subtle issues in some games due to the reduced accuracy, but most games should run fine (and much faster).

The VM uses a lot of RAM

To allow guest apps to use a large amount of RAM (as some modern games require), by default muvm allows the guest to use up to 80% of the system RAM. This also means that some of that will be taken up by guest page cache, appearing to the host as if the VM is taking up most of system RAM. muvm has the ability to reduce guest page cache usage as host memory pressure increases, so if you increase host memory usage, the VM should reduce its usage accordingly (as long as it is able to discard unused cache RAM).

On lower RAM size machines (16GB or lower), we recommend not running any heavy host applications while the VM is in use. We don’t recommend running complex games on 8GB machines.

To inspect VM memory usage while it is running, use muvm -ti -- free. You can also run muvm -ti -- htop (if you have htop installed) to get more detailed information, or substitute your system information tool of choice.

If you wish to limit the maximum memory usage of the MicroVM, you can configure the guest RAM allocation with the muvm --mem=SIZE parameter.

I can’t access media mounted under /run/media within the VM

This does not work (not even via /run/muvm-host/run/media) due to missing POSIX ACL support in libkrun at this time. You must manually mount any disks that you wish to use within the VM, e.g. under /mnt.

FAQ

Is this like Rosetta on macOS?

This is as close to Rosetta as we can get! The main difference is that Rosetta side-steps the page size issue by instead relying on the XNU kernel’s multiple page size support for user processes, so it doesn’t need a VM. While making Linux support mixed page sizes would not be completely impossible in theory, it would be an enormous project that would likely take years to complete, and it isn’t at all clear whether such a change would be accepted upstream (Linux doesn’t even have boot-time page size selection within a single kernel yet!).

Other than the page size issue, FEX and Rosetta are comparable technologies (both are emulators, despite what Apple marketing might have you believe). Both FEX and Rosetta use the unique Apple Silicon CPU feature that is most important for x86/x86-64 emulation performance: TSO mode. Thanks to this feature, FEX can offer fast and accurate x86/x86-64 emulation on Apple Silicon systems.

Why not just use a 4K host kernel?

While Apple Silicon systems support 4K CPU pages, the rest of the hardware (IOMMUs, GPU) runs with 16K pages only. The Linux kernel does not play nicely in this environment, as it generally assumes that the CPU page size is at least as large or larger than the IOMMU page size. In the past we had some kernel patches to make this partially work, but they were buggy and incomplete, so we abandoned the approach. Even if it did work well, running the whole system using 4K pages has a measurable performance impact, so we would never ship 4K kernels by default. Therefore, running x86/x86-64 apps would require that users manually change their kernel and reboot, which is quite cumbersome.

Why not box64?

box64 and FEX-Emu have different approaches to emulation, with FEX-Emu aiming for better correctness by default (but requiring a more complex setup) while box64 aims to cover more "out of the box" use cases (like running a subset of applications directly on a 16K kernel without a VM using some tricks). We have chosen FEX-Emu for our stack because we believe it will have higher compatibility with its approach, but both have their uses. box64 is packaged in Fedora, so we encourage users to try it (both natively and within muvm) and let us know how it compares!

My x86-64 or x86 application is missing some system libraries, what do I do?

Our immutable RootFS contains a large set of common x86-64 and x86 libraries that are commonly used as dependencies, but we cannot ship every possible library. You can view the package list here.

If the missing library is a relatively simple, common library with no or very simple dependencies, and which would not add much size to our RootFS images, please submit a PR to the repo linked above so we can include it in future releases of the RootFS. Make sure to note what application requires the library, and why you think we should include it.

If your app requires a complicated framework (such as Qt) or an uncommon, niche library, then the application is not built as a "portable" application and not expected to work out-of-the-box on most systems. To work around the issue, you can manually download the missing libraries, extract them into your home directory, and use LD_LIBRARY_PATH to make your application find them. You can use the following command to download x86-64 RPMs from your arm64 Fedora installation:

dnf download --repo=fedora --repo=updates --forcearch=x86_64 --best [package1] [package2]...

You can then use rpmdev-extract to extract the contents of the RPM, and then configure LD_LIBRARY_PATH as appropriate.

It is also possible to overlay RPMs into the existing RootFS, though this should be considered advanced functionality. Once you have an RPM, you can convert it to an erofs image using these commands:

rpm2archive -n mypackage.rpm
mkfs.erofs --tar=f mypackage.rpm.erofs mypackage.rpm.tar

Then, you can manually launch muvm with the base erofs images and your custom erofs image on top, like this:

muvm \
    -f /usr/share/fex-emu/RootFS/default.erofs \
    -f /usr/share/fex-emu/overlays/mesa-x86_64.erofs \
    -f /usr/share/fex-emu/overlays/mesa-i386.erofs \
    -f mypackage.rpm.erofs \
    <your muvm arguments here>

This will overlay the add-on package onto the RootFS used for FEX. Please keep in mind that this may or may not work as intended, and it should not be considered a supported solution.

Steam says steamwebhelper crashed, what do I do?

Just let it restart, and it should work on the second try. Steam has a timeout for steamwebhelper, and when running under emulation, startup is slow enough that the timeout expires. This usually only happens on a cold startup.

I am unable to download/run certain games in Steam

Ensure Steam Play is enabled for all games. It should be under Menu > Settings > Compatibility > Enable Steam Play for all other titles. Restart Steam once enabled.

Pressing keys makes the touchpad stop responding

This is caused by the "Disable while typing" touchpad feature. You can turn it off in the touchpad/input settings for your desktop environment.

Can I run Windows applications outside of Steam?

At this point, we do not support running Windows apps outside of Steam as non-proton Wine not yet work on Fedora. We are working on resolving the underlying FEX issue, so we expect to support this relatively soon.

In the meantime, you can use Steam’s Proton to run non-Steam Windows applications directly from Steam.

Can I run x86-64/x86 Linux applications?

Native Linux games should generally work under muvm, as long as they are self-contained and do not depend on complex host system libraries (we ship a large selection of common dependencies, but not everything under the sun).

Is Wayland supported?

Wayland is not supported inside the VM at this time. As most of the legacy x86/x86-64 applications people want to run are X11 applications, we are focusing on X11 support first. This means that you cannot run native Wayland apps inside the VM at this time. Of course, the host desktop is still a Wayland desktop, and X11 support is provided by XWayland.

Can I access hardware from applications running within the microVM?

As the VM does not pass through any host hardware other than the GPU and the virtual filesystem, you will not be able to use applications that require direct hardware access. We use software passthrough for the following interfaces:

  • X11 protocol (display, keyboard, mouse)

  • Gamepads via hid/uinput passthrough

  • Sound I/O via the PulseAudio socket protocol [1]

We are researching the possibility of passing through PipeWire, which will allow webcams to be used within the VM.

Can I use input methods (IMEs) in applications within the microVM?

You can use the classic xim input method system used in X11. muvm should already configure the environment variables appropriately to make this work for Qt and GTK applications (loading the "xim" plugin), as long as the input method framework you are using on your host system supports it. We have tested this with fcitx5 and Steam running on KDE Plasma.

In the future, once Wayland passthrough is supported, the native Wayland input protocol mechanism should work with any host input method framework (through a plugin usually called "wayland"). There are no plans to support non-window-system-based input methods (such as the direct "ibus" and "fcitx" plugins), since they would require us to ship x86-64 shared libraries for all possible input methods in the immutable virtual x86-64 system, and would also require proxying of their bespoke protocols, which is infeasible.

Is this like a Qemu/libvirt/UTM/Parallels/VMWare/VirtualBox/etc. VM?

No, muvm does not work like a traditional whole-system VM. While it does also use KVM as a backend for efficient virtualization, the concept is very different to traditional VMs running entirely separate guest operating systems. The guest kernel is a special kernel optimized to start up in a fraction of a second, and the VM monitor passes through the host filesystem mostly as-is. There is no low-level hardware passthrough (USB, etc.) and instead we focus on higher-level software protocol passthrough, like X11/Wayland. The VM does not run its own standalone init system, only some minimal startup code. This means that the environment within the VM should "feel" the same as the host OS from the point of view of applications, just with a 4K page size instead of a 16K page size.

Do browsers work within the VM guest?

Yes, but they will run in X11 mode. However, there is one caveat: browser instances within the guest cannot communicate with browser instances outside the guest, and it is dangerous to run the same browser profile in both the guest and the host.

To avoid these problems, muvm configures an environment variable to force Firefox to use a dedicated profile when launched within the VM. This means that applications that launch a browser (such as for login or documentation purposes) will work as intended, but Firefox will launch using a dedicated profile without access to your cookies, history, etc.

If you launch the same browser profile in the guest and the host simultaneously, your profile data may become corrupted. If your default browser is not Firefox, and you are using emulated apps that might launch your default browser inadvertently, we strongly recommend closing all browser windows before using muvm or manually configuring separate profiles.

Can I sudo inside the VM?

Since the VM monitor runs as your own user identity, it cannot gain root privileges. "root" inside the VM still only has the privileges of your own user, so sudo doesn’t make much sense (and in fact doesn’t work). We recommend installing software that you want to use with muvm+FEX under your home directory. For software that is designed to be installed under /opt or similar, we recommend performing the installation steps manually on the host OS, and then just running the app under muvm.

If you need access to a root shell within the VM for debugging purposes, you can run muvm -tip 3335 -- bash. Keep in mind that, despite being "root", you will not be able to modify most system files owned by root, and any files you create will actually be owned by your non-root user identity. A root shell is mainly useful to do things like strace other processes or change guest kernel or network configuration settings (but these changes will not persist across a VM restart).

Can applications within the VM communicate with applications outside the VM?

Communication is mostly limited to the host filesystem. The VM shares your home directory (and in fact most of the filesystem) with the host, so any files you create on one side will be visible on the other.

Thanks to virtiofs-DAX, shared memory communication (/dev/shm) is also available between guest apps and host apps. This is used, for example, by the X11 forwarding code.

It is also possible to share audio between host and guest apps by using the PulseAudio forwarding support. For example, you can record guest audio by using a recording app on the host and recording from the system "Monitor" device. You can also configure virtual sinks/sources in the host using the normal PipeWire mechanisms, and direct guest apps to use those for audio I/O to have custom audio routing and processing. Note that the native PipeWire protocol is not passed through, only the PulseAudio protocol which is more limited (but more commonly used by applications). ALSA applications are supported via the pulse plug-in.

If you are using a host compositor that supports XWayland video bridging (such as KDE Plasma / KWin), you will be able to screen share / screen capture from the VM, including full host screens and Wayland windows. Make sure the app you are running supports "classic" XComposite window/screen capture. When you initiate screen sharing, you will be able to directly select X11 application windows, or choose the virtual "Xwayland Video Bridge" window. When you do so, KDE will automatically prompt you for the actual window or screen you wish to share.

Why do I have fewer CPU cores inside the VM?

By default, muvm passes through as many CPUs as there are performance cores on your host machine, and pins those vCPUs to the physical performance cores. Since the host CPU scheduler has no visibility into the guest CPU scheduler, this ensures that performance is consistent. You can modify this behavior with the muvm --cpu-list=CPU_LIST option.

How do I use an external drive as a Steam library folder?

Follow these steps to get an external drive set up for Steam:

  1. Format your external drive using a Linux-native filesystem such as ext4.

    FAT32, exFAT, and other filesystems without support for Linux permissions will not work.
  2. Mount your drive manually under a directory accessible to muvm, such as /mnt/steam.

    We recommend mounting the drive manually (e.g. sudo mount /dev/sdX1 /mnt/steam, where sdX1 is your drive’s device file). If you configure the drive to mount automatically using /etc/fstab, then your system will not boot if that drive is not connected.
    The default mountpoint for drives mounted via the desktop environment (udisks) will not work at this time.
  3. Ensure that the filesystem root is accessible to your regular user:

    sudo chown ${USER}: /mnt/steam

  4. Create an empty folder named steamapps at the root of the mount:

    mkdir /mnt/steam/steamapps

  5. Start up Steam normally

  6. Click on the Library tab, then click the gear (settings) icon

  7. Select Storage on the left menu, click on the combo box at the top of the panel, and select Add Drive.

  8. Browse to your drive mountmount (/mnt/steam), such that the only folder visible in the file picker list window is the empty steamapps folder within, then (without making any further selections) click on the Select button.

You should now be able to select the new download location, make it the default, and download games to it.


1. This works with PipeWire running on the host with pipewire-pulse, as installed by default. You do not need to and should not install PulseAudio proper, as that will break your speaker support!