Introduction into the Genode porting for Xilinx MicroBlaze

This file gives an overview to the Genode porting for MicroBlaze-based platforms. To get a quick introduction in how to build and run Genode on such platforms, please refer to:

 <GENODE_DIR/base-mb/doc/getting_started.txt>

Xilinx MicroBlaze is a so-called softcore CPU, which is commonly used as part of FPGA-based System-on-Chip designs. At Genode Labs, we are regularly using this IP core, in particular for our Genode FPGA Graphics Project, which is a GUI software stack and a set of IP cores for implementing fully-fledged windowed GUIs on FPGAs:

Ever since we first released the Genode FPGA project, we envisioned to combine it with the Genode OS Framework. In Spring 2010, Martin Stein joined our team at Genode Labs and accepted the challenge to bring the Genode OS Framework to the realms of FPGA-based SoCs. Technically, this implies porting the framework to the MicroBlaze CPU architecture. In contrast to most softcore CPUs such as the popular Lattice Mico32, the MicroBlaze features a MMU, which is a fundamental requirement for implementing a microkernel-based system. Architecturally-wise MicroBlaze is a RISC CPU similar to MIPS. Many system parameters of the CPU (caches, certain arithmetic and shift instructions) can be parametrized at synthesizing time of the SoC. We found that the relatively simple architecture of this CPU provides a perfect playground for pursuing some of our ideas about kernel design that go beyond the scope of current microkernels. So instead of adding MicroBlaze support into one of the existing microkernels already supported by Genode, we went for a new kernel design. Deviating from the typical microkernel, which is a self-sufficient program running in kernel mode that executes user-level processes on top, our design regards the kernel as a part of Genode's core. It is not a separate program but a library that implements the glue between user-level core and the raw CPU. Specifically, it provides the entrypoint for hardware exceptions, a thread scheduler, an IPC mechanism, and functions to manipulate virtual address spaces (loading and flushing entries from the CPU's software-loaded TLB). It does not manage any physical memory resources or the relationship between processes. This is the job of core. From the kernel-developer's point of view, the kernel part can be summarized as follows:

• The kernel provides user-level threads that are scheduled in a round-robin fashion.

• Threads can communicate via synchronous IPC.

• There is a mechanism for blocking and waking up threads. This mechanism can be used by Genode to implement locking as well as asynchronous inter-process communication.

• There is a single kernel thread, which never blocks in the kernel code paths. So the kernel acts as a state machine. Naturally, there is no concurrency in the execution paths traversed in kernel mode, vastly simplifying these code parts. However, all code paths are extremely short and bounded with regard to execution time. Hence, we expect the interference with interrupt latencies to be low.

• The IPC operation transfers payload between UTCBs only. Each thread has a so-called user-level thread control block which is mapped transparently by the kernel. Because of this mapping, user-level page faults cannot occur during IPC transfers.

• There is no mapping database. Virtual address spaces are manipulated by loading and flushing physical TLB entries. There is no caching of mappings done in the kernel. All higher-level information about the interrelationship of memory and processes is managed by the user-level core.

• Core runs in user mode, mapped 1-to-1 from the physical address space except for its virtual thread-context area.

• The kernel paths are executed in physical address space (MicroBlaze). Because both kernel code and user-level core code are observing the same address-space layout, both worlds appear to run within a single address space.

• User processes can use the entire virtual address space (4G) except for a helper page for invoking syscalls and a page containing atomic operations. There is no reservation used for the kernel.

• The MicroBlaze architecture lacks an atomic compare-and-swap instruction. On user-level, this functionality is emulated via delayed preemption. A kernel- provided page holds the sequence of operations to be executed atomically and prevents (actually delays) the preemption of a thread that is currently executing instructions at that page.

• The MicroBlaze MMU supports several different page sizes (1K up to 16MB). Genode fully supports this feature for page sizes >= 4K. This way, the TLB footprint can be minimized by choosing sensible alignments of memory objects.

Current state

The MicroBlaze platform support resides in the base-mb repository. At the current stage, core is able to successfully start multiple nested instances of the init process. Most of the critical kernel functionality is working. This includes inter-process communication, address-space creation, multi-threading, thread synchronization, page-fault handling, and TLB eviction.

The nested init scenario runs on Qemu, emulating the Petalogix Spartan 3A DSP1800 design, as well as on real hardware, tested with the Xilinx Spartan 3A Starter Kit configured with an appropriate Microblaze SoC.

This simple scenario already illustrates the vast advantage of using different page sizes supported by the MicroBlaze CPU. If using 4KB pages only, a scenario with three nested init processes produces more than 300.000 page faults. There is an extremely high pressure on the TLB, which only contains 64 entries. Those entries are constantly evicted so that threshing effects are likely to occur. By making use of flexible page sizes (4K, 16K, 64K, 256K, 1M, 4M, 16M), the number of page faults gets slashed to only 1.800, speeding up the boot time by factor 10.

On hardware the capability remains to increase execution speed significantly by turning on instruction- and data-caches. However this feature has not been tested for now.

The kernel provides, beyond the requirements of the nested init scenario, allocation, handling and deallocation of IRQs to the userland to enable core to offer IRQ and IO Memory session services. This allows custom device-driver implementations within the userland.

Currently, there is no restriction of IPC communication rights. Threads are addressed using their global thread IDs (in fact, using their respective indices in the KTCB array). For the future, we are planning to add capabilty-based delegation of communication rights.