Genode Porting Guide: Porting devices drivers

Even though Genode encourages writing native device drivers, this task sometimes becomes infeasible. Especially if there is no documentation available for a certain device or if there are not enough programming resources at hand to implement a fully fledged driver. Examples of ported drivers can be found in the dde_linux, dde_bsd, and dde_ipxe repositories.

In this chapter we will exemplary discuss how to port a Linux driver for an ARM based SoC to Genode. The goal is to execute driver code in user land directly on Genode while making the driver believe it is running within the Linux kernel. Traditionally there have been two approaches to reach this goal in Genode. In the past, Genode provided a Linux environment, called dde_linux26, with the purpose to offer just enough infrastructure to easily port drivers. However, after adding more drivers it became clear that this repository grew extensively, making it hard to maintain. Also updating the environment to support newer Linux-kernel versions became a huge effort which let the repository to be neglected over time.

Therefore we choose the path to write a customized environment for each driver, which provides a specially tailored infrastructure. We found that the support code usually is not larger than a couple of thousand lines of code, while upgrading to newer driver versions, as we did with the USB drivers, is feasible.

Basic driver structure

The first step in porting a driver is to identify the driver code that has to be ported. Once the code is located, we usually create a new Genode repository and write a port file to download and extract the code. It is good practice to name the port and the hash file like the new repository, e.g. dde_linux.port if the repository directory is called <genode-dir>/repos/dde_linux. Having the source code ready, there are three main tasks the environment must implement. The first is the driver back end, which is responsible for raw device access using Genode primitives, the actual environment that emulates Linux function calls the driver code is using, and the front end, which exposes for example some Genode-session interface (like NIC or block session) that client applications can connect to.

Further preparations

Having the code ready, the next step is to create an *.mk file that actually compiles the code. For a driver library lib/mk/<driver name>.mk has to be created and for a stand-alone program src/<driver name>/target.mk is created within the repository. With the *.mk file in place, we can now start the actual compilation. Of course this will cause a whole lot of errors and warnings. Most of the messages will deal with implicit declarations of functions and unknown data types. What we have to do now is to go through each warning and error message and either add the header file containing the desired function or data type to the list of files that will be extracted to the contrib directory or create our own prototype or data definition.

When creating our own prototypes, we put them in a file called lx_emul.h. To actually get this file included in all driver files we use the following code in the *.mk file:

 CC_C_OPT  += -include $(INC_DIR)/lx_emul.h

where INC_DIR points to the include path of lx_emul.h.

The hard part is to decide which of the two ways to go for a specific function or data type, since adding header files also adds more dependencies and often more errors and warnings. As a rule of thumb, try adding as few headers as possible.

The compiler will also complain about a lot of missing header files. Since we do not want to create all these header files, we use a trick in our *.mk file that extracts all header file includes from the driver code and creates symbolic links that correspond to the file name and links to lx_emul.h. You can put the following code snippet in your *.mk file which does the trick:

#
# Determine the header files included by the contrib code. For each
# of these header files we create a symlink to _lx_emul.h_.
#
GEN_INCLUDES := $(shell grep -rh "^\#include .*\/" $(DRIVER_CONTRIB_DIR) |\
                        sed "s/^\#include [^<\"]*[<\"]\([^>\"]*\)[>\"].*/\1/" | \
                        sort | uniq)

#
# Filter out original Linux headers that exist in the contrib directory
#
NO_GEN_INCLUDES := $(shell cd $(DRIVER_CONTRIB_DIR); find -name "*.h" | sed "s/.\///" | \
                           sed "s/.*include\///")
GEN_INCLUDES    := $(filter-out $(NO_GEN_INCLUDES),$(GEN_INCLUDES))

#
# Put Linux headers in 'GEN_INC' dir, since some include use "../../" paths use
# three level include hierarchy
#
GEN_INC         := $(shell pwd)/include/include/include

$(shell mkdir -p $(GEN_INC))

GEN_INCLUDES    := $(addprefix $(GEN_INC)/,$(GEN_INCLUDES))
INC_DIR         += $(GEN_INC)

#
# Make sure to create the header symlinks prior building
#
$(SRC_C:.c=.o) $(SRC_CC:.cc=.o): $(GEN_INCLUDES)

$(GEN_INCLUDES):
  $(VERBOSE)mkdir -p $(dir $@)
  $(VERBOSE)ln -s $(LX_INC_DIR)/lx_emul.h $@

Make sure LX_INC_DIR is the directory containing the lx_emul.h file. Note that GEN_INC is added to your INC_DIR variable.

The DRIVER_CONTRIB_DIR variable is defined by calling the select_from_port function at the beginning of a Makefile or a include file, which is used by all other Makefiles:

 DRIVER_CONTRIB_DIR := $(call select_from_ports,driver_repo)/src/lib/driver_repo

The process of function definition and type declaration continues until the code compiles. This process can be quite tiresome. When the driver code finally compiles, the next stage is linking. This will of course lead to another whole set of errors that complain about undefined references. To actually obtain a linked binary we create a dummies.cc file. To ease things up we suggest to create a macro called DUMMY and implement functions as in the example below:

 /*
  * Do not include 'lx_emul.h', since the implementation will most likely clash
  * with the prototype
  */

#define DUMMY(retval, name) \
  DUMMY name(void) { \
    PDBG( #name " called (from %p) not implemented", __builtin_return_address(0)); \
  return retval; \
}

 DUMMY(-1, kmalloc)
 DUMMY(-1, memcpy)
 ...

Create a DUMMY for each undefined reference until the binary links. We now have a linked binary with a dummy environment.

Debugging

From here on, we will actually start executing code, but before we do that, let us have a look at the debugging options for device drivers. Since drivers have to be tested on the target platform, there are not as many debugging options available as for higher level applications, like running applications on the Linux version of Genode while using GDB for debugging. Having these restrictions, debugging is almost completely performed over the serial line and on rare occasions with an hardware debugger using JTAG.

For basic Linux driver debugging it is useful to implement the printk function (use dde_kit_printf or something similar) first. This way, the driver code can output something and additions for debugging can be made. The __builtin_return_address function is also useful in order to determine where a specific function was called from. printk may become a problem with devices that require certain time constrains because serial line output is very slow. This is why we port most drivers by running them on top of the Fiasco.OC version of Genode. There you can take advantage of Fiasco's debugger (JDB) and trace buffer facility.

The trace buffer can be used to log data and is much faster than printk over serial line. Please inspect the ktrace.h file (at base-foc/contrib/l4/pkg/l4sys/include/ARCH-*/ktrace.h) that describes the complete interface. A very handy function there is

fiasco_tbuf_log_3val("My message", variable1, variable2, variable3);

which stores a message and three variables in the trace buffer. The trace buffer can be inspected from within JDB by pressing T.

JDB can be accessed at any time by pressing the ESC key. It can be used to inspect the state of all running threads and address spaces on the system. There is no recent JDB documentation available, but

should be a good starting point. It is also possible to enter the debugger at any time from your program calling the enter_kdebug("My breakpoint") function from within your code. The complete JDB interface can be found in base-foc/contrib/l4/pkg/l4sys/include/ARCH-*/kdebug.h.

Note that the backtrace (bt) command does not work out of the box on ARM platforms. We have a small patch for that in our Fiasco.OC development branch located at GitHub: http://github.com/ssumpf/foc/tree/dev

The back end

To ease up the porting of drivers and interfacing Genode from C code, Genode offers a library called DDE kit. DDE kit provides access to common functions required by drivers like device memory, virtual memory with physical-address lookup, interrupt handling, timers, etc. Please inspect os/include/dde_kit to see the complete interface description. You can also use grep -r dde_kit_ * to see usage of the interface in Genode.

As an example for using DDE kit we implement the kmalloc call:

void *kmalloc(size_t size, gfp_t flags)
{
  return dde_kit_simple_malloc(size);
}

It is also possible to directly use Genode primitives from C++ files, the functions only have to be declared as extern "C" so they can be called from C code.

The environment

Having a dummy environment we may now begin to actually execute driver code.

Driver initialization

 #define module_init(fn) void module_##fn(void) { fn(); }

 module_init(ehci_hcd_init);

 module_ehci_hcd_init();

Device probing

The front end

An important design question is how the front end is attached to the driver. In some cases the front end may not use the driver directly, but other Linux subsystems that are ported or emulated by the environment. For example, the USB storage driver implements parts of the SCSI subsystem, which in turn is used by the front end. The whole decision depends on the kind of driver that is ported and on how much additional infrastructure is needed to actually make use of the data. Again an USB example: For USB HID, we needed to port the USB controller driver, the hub driver, the USB HID driver, and the generic HID driver in order to retrieve keyboard and mouse events from the HID driver.

The last step in porting a device driver is to make it accessible to other Genode applications. Typically this is achieved by implementing one of Genode's session interfaces, like a NIC session for network adapters or a block session for block devices. You may also define your own session interfaces. The implementation of the session interface will most likely trigger driver calls, so you have to have to keep an eye on the dummy functions. Also make sure that calls to the driver actually do what they are supposed to, for example, some wrong return value of a dummy function may cause a function to return without performing any work.

Notes on synchronization

After some experiences with Linux drivers and multi-threading, we lately choose to have all Linux driver code executed by a single thread only. This way no Linux synchronization primitives have to be implemented and we simply don't have to worry about subtle pre- and postconditions of many functions (like "this function has to be called with lock x being held").

Unfortunately we cannot get rid of all threads within a device-driver server, there is at least one waiting for interrupts and one for the entry point that waits for client session requests. In order to synchronize these threads, we use Genode's signalling framework. So when, for example, the IRQ thread receives an interrupt it will send a signal. The Linux driver thread will at certain points wait for these signals (e.g., functions like schedule_timeout or wait_for_completion) and execute the right code depending on the kind of signal delivered or firmly speaking the signal context. For this to work, we use a class called Signal_dispatcher (base/include/base/signal.h) which inherits from Signal_context. More than one dispatcher can be bound to a signal receiver, while each dispatcher might do different work, like calling the Linux interrupt handler in the IRQ example.