From 357d1e3bc7d2d80e5271bc4f3ac8537e30dc8046 Mon Sep 17 00:00:00 2001 From: David Gibson Date: Sun, 16 Oct 2016 12:04:15 +1100 Subject: spapr: Improved placement of PCI host bridges in guest memory map Currently, the MMIO space for accessing PCI on pseries guests begins at 1 TiB in guest address space. Each PCI host bridge (PHB) has a 64 GiB chunk of address space in which it places its outbound PIO and 32-bit and 64-bit MMIO windows. This scheme as several problems: - It limits guest RAM to 1 TiB (though we have a limited fix for this now) - It limits the total MMIO window to 64 GiB. This is not always enough for some of the large nVidia GPGPU cards - Putting all the windows into a single 64 GiB area means that naturally aligning things within there will waste more address space. In addition there was a miscalculation in some of the defaults, which meant that the MMIO windows for each PHB actually slightly overran the 64 GiB region for that PHB. We got away without nasty consequences because the overrun fit within an unused area at the beginning of the next PHB's region, but it's not pretty. This patch implements a new scheme which addresses those problems, and is also closer to what bare metal hardware and pHyp guests generally use. Because some guest versions (including most current distro kernels) can't access PCI MMIO above 64 TiB, we put all the PCI windows between 32 TiB and 64 TiB. This is broken into 1 TiB chunks. The first 1 TiB contains the PIO (64 kiB) and 32-bit MMIO (2 GiB) windows for all of the PHBs. Each subsequent TiB chunk contains a naturally aligned 64-bit MMIO window for one PHB each. This reduces the number of allowed PHBs (without full manual configuration of all the windows) from 256 to 31, but this should still be plenty in practice. We also change some of the default window sizes for manually configured PHBs to saner values. Finally we adjust some tests and libqos so that it correctly uses the new default locations. Ideally it would parse the device tree given to the guest, but that's a more complex problem for another time. Signed-off-by: David Gibson Reviewed-by: Laurent Vivier --- include/hw/pci-host/spapr.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) (limited to 'include') diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h index 23dfb09d36..b92c1b59f1 100644 --- a/include/hw/pci-host/spapr.h +++ b/include/hw/pci-host/spapr.h @@ -84,8 +84,14 @@ struct sPAPRPHBState { #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL #define SPAPR_PCI_MEM32_WIN_SIZE \ ((1ULL << 32) - SPAPR_PCI_MEM_WIN_BUS_OFFSET) +#define SPAPR_PCI_MEM64_WIN_SIZE 0x10000000000ULL /* 1 TiB */ -#define SPAPR_PCI_MMIO_WIN_SIZE 0xf80000000 +/* Without manual configuration, all PCI outbound windows will be + * within this range */ +#define SPAPR_PCI_BASE (1ULL << 45) /* 32 TiB */ +#define SPAPR_PCI_LIMIT (1ULL << 46) /* 64 TiB */ + +#define SPAPR_PCI_2_7_MMIO_WIN_SIZE 0xf80000000 #define SPAPR_PCI_IO_WIN_SIZE 0x10000 #define SPAPR_PCI_MSI_WINDOW 0x40000000000ULL -- cgit v1.2.3