QEMU CXL Host

Similar to running CXL Simple Host, QEMU-based CXL Host can be used for the memory expander demo.

cxl-util can be run to create a CXL switch and a group of CXL SLDs to allow QEMU to interact with them.

Step 1. Core

In the root directory of opencxl-core:

./cxl-util start \
  -c switch \
  -c sld-group \
  --config-file configs/1vcs_4sld.yaml

The above command sets up a CXL-Switch (with one VCS) and connects 4 Single Logical Devices (e.g. memory devices) to a CXL-Switch.

Step 2. QEMU

note

KVM is a prerequisite to running QEMU. Be sure to have the CPU virtualization support (e.g. VT-x, AMD-V) enabled in the BIOS prior to continuing.

From the root directory of opencxl-qemu, go to cxl_scripts:

cd cxl_scripts

Now, if you are running native Ubuntu, run:

./start-vm.sh

Instead, if you went through the Docker Setup, run:

./start-vm.sh opencxl-core

Be sure to have cxl-util running (from Step 1) in a separate terminal prior to executing the above command.

Once the OS boots, log in with:

Username: eeum
Password: eeum

Step 3. Setting Up Guest OS

Create CXL Memory Regions

After booting, the CXL devices attached to the guest OS are not enabled. We need to create a CXL region to access these CXL devices.

Create a CXL region:

sudo cxl create-region -m -d decoder0.0 mem0 mem1 mem2 mem3

Expected ouput:

[
  {
    "memdevs":[
      {
        "memdev":"mem2",
        "ram_size":268435456,
        "serial":0,
        "host":"0000:0f:00.0"
      },
      ...
    ]
  },
  {
    "regions":[
      {
        "region":"region0",
        "resource":11005853696,
        "size":1073741824,
        "type":"ram",
        "interleave_ways":4,
        "interleave_granularity":256,
        "decode_state":"commit"
      }
    ]
  }
]

Run sudo cat /proc/iomem to confirm that region and dax device are created successfully:

...
290000000-828fffffff : CXL Window 0
  290000000-2cfffffff : region0
    290000000-2cfffffff : dax0.0
      290000000-2cfffffff : System RAM (kmem)

Load CXL region using devdax driver

The created region can be accessed using a dax device when dax device is loaded as a character device. We need to reconfigure the dax device so that it will not be a System RAM.

sudo daxctl reconfigure-device --human --mode=devdax --force dax0.0

Expected ouput:

{
  "chardev":"dax0.0",
  "size":"1024.00 MiB (1073.74 MB)",
  "target_node":0,
  "align":2097152,
  "mode":"devdax"
}

Step 4. Generating IO Traffic

Get into the superuser mode: Password is eeum

sudo su

Go to the iogen tool directory:

cd /root/iogen

Run iogen to generate writes to the CXL memory devices.

./iogen

(Optional) QEMU with Custom Kernel

Building Custom Kernel For Developers

You can customize the OS image by getting the Linux kernel source and building a custom kernel binary and kernel drivers. The instructions are as follow:

1. Upgrade packages and reboot

   sudo dnf upgrade -y && sudo reboot

2. Install dependencies.

   sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign grubby -y

3. Get the kernel source.

   fedpkg clone -a kernel

4. Get source for a specific version. The example will checkout f39.

   cd kernel && git switch f39

5. Get additional dependencies.

   sudo dnf builddep kernel.spec -y

6. Get sources.

   fedpkg sources

7. Extract sources. In this example, the downloaded kernel version is 6.7.4.

   tar -xf linux-6.7.4.tar.xz

8. Move to the source directory.

   cd linux-6.7.4

9. Copy installed kernel config.

   cp /boot/config-$(uname -r) .config

10. Open .config. Find and update CONFIG_CXL_REGION_INVALIDATION_TEST.

    Set CONFIG_CXL_REGION_INVALIDATION_TEST=y

11. Build kernel.

    make -j$(nproc)

12. Install kernel modules.

    sudo make modules_install

13. Install kernel.

    sudo make install

14. Reboot.