Release notes for the Genode OS Framework 16.05

After ten years of developing and refining the Genode OS Framework and conducting countless experiments, it is time to condense the gathered body of experience into a fundamentally revised API. Version 16.05 marks the most profound API change in the project's history. The underlying motivation is to reduce complexity while preserving the flexibility of the framework. With a narrow and orthogonal API, components become easier to develop, to evaluate, and to maintain. The second motivation is our aspiration to ultimately attain a stable binary interface that works across different kernels. The new API is a significant step in this direction. Section The great API renovation presents the rationale behind these changes, and outlines the steps for migrating to the new API. It is complemented by technical details given in Section Base framework and a comprehensive documentation update (Section New revision of the Genode Foundations book).

The second focus of the current release is the update of Genode's arsenal of device drivers, in particular the driver stacks ported from Linux. Those drivers comprise the Intel wireless stack, the Intel graphics driver, the Linux TCP/IP stack, and the Linux USB stack. Genode scenarios can now take advantage of the same drivers as modern Linux distributions based on the Linux kernel version 4.4.3. Furthermore, the audio driver ported from OpenBSD received an update. Section Device drivers describes this line of work in detail. The device-driver topic is nicely complemented with a greatly improved ACPI support presented in Section Enhanced ACPI support.

With respect to the framework's feature set, version 16.05 introduces the ability to use the Rust programming language for Genode components (Section New support for the Rust programming language), and allows the GNU debugger to be used on top of 64-bit NOVA (Section Enhanced GDB support on NOVA).

According to the road map, we originally planned a few more functional additions that did not make it into the release. Even through the work on the package-management topic progresses, we are still at an experimental stage. We decided to defer the work on the NAT component and instead focused on improvements of the base-hw kernel (Section Execution on bare hardware (base-hw)) that are highly anticipated by the developers of the Muen separation kernel. Speaking of Muen, the support for running Genode on this kernel has reached the state where Virtualbox can be executed inside a Muen partition. However, we decided to not rush the integration of this feature into Genode's mainline. Right now, it still resides on a topic branch. If you are eager to try it out right now, please get in touch via the mailing list.

The great API renovation   High-level overview of the changes   New revision of the Genode Foundations book   API migration guide Base framework   New component API   Consolidation of core's SIGNAL, CAP, RM, and PD services   New log-output facilities   XML processing   Dataspace helpers   Thread API and CPU-session interface   Child management   Stylistic changes   Removed and to-be-removed APIs Low-level OS infrastructure   Enhanced GDB support on NOVA   New support for the Rust programming language   Dynamic linker   New component for writing ROM modules to files   C runtime     Sysctl     libc_pipe plugin Device drivers   HDA audio driver update   Linux kit   Wifi driver update   USB driver update   Intel graphics driver update   Enhanced ACPI support   Generalized SDHCI driver Libraries and applications   LxIP update   Qemu USB Platforms   Generalization of platform-specific headers   NOVA microhypervisor   Execution on bare hardware (base-hw)   Linux Tools and build system   Usability improvements of the ports tools   Updated tool chain Removal of stale features

The great API renovation

Genode's base API evolved over the years. When we started in 2006, our mindset was very much influenced by L4, which regarded synchronous IPC as the only mechanism required. We used to implement components ("servers") in a procedural programming style with a fully synchronous control flow within and across components (IPC).

We eventually realized that the restriction to synchronous IPC was misguided (see Section 3.6.2. "Asynchronous notifications" in the Genode Foundations book for a detailed discussion of the problems). When we started to embrace the use of asynchronous notifications and to disregard blocking RPC between components, we found ourselves designing components as state machines rather than procedural programs. To promote this programming style, we introduced the so-called server API (os/server.h). See NIC loopback server for a canonical example of such a component.

We found that this new style greatly increased the robustness and flexibility of the components. In particular, it completely alleviates race conditions, which constantly troubled us in the past. Now, following the new paradigm, we may end up in a deadlock, but such a situation is easy to debug compared to sporadic race conditions. Over the past two years, we have redesigned all Genode session interfaces to avoid blocking RPC. The only remains are the parent and root interfaces, which we still need to address.

Still, the Genode API retained compatibility to the old (= wrong) style of developing components. We take the current release as an opportunity to finally clean the API from our past mistakes.

High-level overview of the changes

Removal of side effects

Up to now, most components rely on the globally available Genode::env() singleton object for interacting with its environment. When calling a function or method, one never knows for sure if the called code interacts with the env. E.g., we can simply create an instance of Timer::Connection without any arguments. Under the hood, the Connection object accesses the env to open a timer session at the parent. In the spirit of capability-based security, we should shun the reliance on side effects like the global env. Instead, we should pass all materials that are needed by the called code explicitly to the called code (e.g., by passing a reference to a Parent & as argument). This way, someone inspecting the calling code can immediately see the possible reach of the called code. Another prominent example is the latent use of env()->heap(), which allows any code to arbitrary consume memory. Instead, we should better pass an Allocator & to the called code. This way, we re-enforce that library code stays clean from the policy where memory is allocated from. If no Allocator & is passed, we know that no dynamic memory allocation can be performed. If an Allocator & is required, the called code needs to explain (in the form of its documentation) why the allocator is actually needed. Moreover, by passing an Allocator_guard, the calling code can impose a limit of the memory consumption on the called code.

Component API

Traditionally, components started their execution at a main function because, well, this is how it's supposed to be, right? The program exits as soon as main returns. With the introduction of the server API mentioned above, we explored a different approach: The execution of a component starts at a construct function that takes a form of the Genode environment as argument. The function is expected to initialize the component. Upon completion of the initialization, the function returns. At this point, the component becomes ready to respond to incoming RPC requests or signals. Each time such a request/signal comes in, a handler is executed. The handler applies the component-internal state changes and returns immediately. No blocking call is performed. We have essentially a state machine.

Over the past two years, we have applied this new approach to all new components - to a great success. So it is time to promote this API to become Genode's base API. The major benefits are:

• Servers become easier to develop as the API is much simpler. Previously, a server had to manually create a CAP connection and an RPC entrypoint. Now, each component has a ready-to-use entrypoint.

• Signals and RPC requests are handled in the context of the same thread, which alleviates the need for locking as long as the component is single threaded, which is actually the case for most components.

• The notions of Thread, Signal_receiver, Rpc_entrypoint are no longer needed by the developer of a component. There is simply an Entrypoint, which is able to handle both RPC requests and signals.

Shunning pointers

The Genode API has still a lot of places where pointers are taken as arguments or exposed from objects. We didn't know better when we started with Genode. Now we do. With the current release, we streamline the base API in this respect. Just as one simple example, the Heap used to take an RM-session pointer and an RAM-session pointer as arguments. These should be references. Pointers should be used only in situations where a nullptr is a reasonable argument, or when dealing with string literals as char const *.

We are not there yet

Whereas we find designing an API that successfully strikes the balance between the lowest possible API complexity and highest possible flexibility extremely challenging, we find ourselves even more challenged with the execution of the transition from one API to another. The order of steps must be planned carefully and interim solutions must be designed as migration paths. We hope that the current release does not induce too much pain for the users of the framework. Wherever we found a way to smoothen the migration path, we took it. But there are a few disruptive changes that are explained in detail in Section [Base framework].

There are still a few loose ends that we will address in the subsequent release, in particular the redesign of the parent interface to become completely asynchronous.

New revision of the Genode Foundations book

The profound changes of the Genode API prompted us to update the Genode Foundations book accordingly. Whereas the principle architecture remained unchanged, many details needed an adjustment. The changes between the last year's edition and the current revision are:

• Consolidation and interface changes of core services (CAP, SIGNAL, RM, CPU, PD),

• Architectural changes related to the management of virtual-memory regions (region maps),

• Updated functional specification matching the new API (e.g., Entrypoint, Env, Component, Thread, log API, IPC),

• Updated NOVA modifications and limitations (kernel-memory quotas, remote unmap, 64-bit guests, write-combining),

• Label-dependent session routing / policy selection

To see the changes in detail, please refer to the book's revision history.

As another useful resource to get acquainted with the new API, please also consider the updated client-server tutorial.

API migration guide

In the current release we promote the former "server API" to the general "component API", which should solely be used for new components. The old API is now considered as deprecated. This short guide provides the list of steps one has to take to adapt an existing server-API-style component to the new API.

The steps are as follows:

1. Replace include directives

   The Server namespace is superseded by the Component namespace and the Entrypoint class has moved to the Genode namespace. Therefore, instead of including the os/server.h header, the base/component.h header must be included.

2. Replace server definition

   Where the old API used the Server hook functions

   char const *Server::name()                    { return "server_ep";         }
   size_t      Server::stack_size()              { return 8*1024*sizeof(long); }
   void        Server::construct(Entrypoint &ep) { static Main main(ep);       }
   

   the new API uses the Component hook functions

   size_t Component::stack_size()                { return 8*1024*sizeof(long); }
   void   Component::construct(Genode::Env &env) { static Main main(env);      }
   

   Note that the name() function ceased to exist.

3. Use Env reference

   Whereas the old API passes a reference to an Entrypoint, the new API passes a reference of the Genode::Env to the component's construct function. The initial entrypoint of the component can be accessed through the Env reference by calling the ep() method. As intermediate step while doing the migration work, it is sufficient to call this method in the construct() method, e.g.:

   void Component::construct(Genode::Env &env) { static Main main(env.ep()); }
   

4. Replace Signal_rpc_member

   The Signal_rpc_member class is considered deprecated and superseded by the Signal_handler class (see base/signal.h). Note that the actual signal-handling method as passed to a Signal_handler has no argument, as opposed to the old API where the handler was called with a counter value.

5. Replace global Genode::env() accessor

   The most cutting change is the discontinuation of the global Genode::env() accessor function. Typically, this accessor was used throughout a component to access its environment, e.g. use Genode::env()->heap() as allocator or Genode::env()->rm_session()->attach() to attach a dataspace.

   Since a component starts its life now at the execution of the construct function, the component has to pass a reference to its Env interface or rather a reference to the needed part of the Env interface on. This explicit way of handling access to the component's environment will require a restructuring of the component. It is good practice to focus on the need at hand rather than to always pass the Env reference on. On that account, it is important to note that the Rm_session is no longer accessible and its place is taken by the Region_map. Where one would have called env()->rm_session()->attach to attach a given dataspace, env.rm().attach has now to be used.

   Furthermore, the global heap allocator object was removed. Components that want to use a heap like allocator have to create such an allocator manually:

   Genode::Heap heap { &env.ram(), &env.rm() };
   

6. Use new log facilities

   Instead of relying on the old print macros (PDBG, PERR, PINF, PLOG, PWRN) and thereby Genode::printf(), a component should use the new Log class to produce diagnostic LOG output. There are a few convenient template functions, namely Genode::error(), Genode::log() and Genode::warning() that can be used. Note that these functions do not use a format string to print various different data types but just a list of arguments, e.g.:

   int const      i = 42;
   char const * str = "world";
   Genode::log("Hello ", str, "! ", i);
   

   produces Hello world! 42.

   Note that certain data types, e.g. enum and char, might have to be casted, e.g.:

   enum { FOO, BAR };
   Genode::log("enum: ", (int)FOO, " ", (int)BAR);
   

Base framework

New component API

Each component is a composition of a protection domain (PD session), a memory budget (RAM session), and a CPU session, from which the initial thread is created. These sessions form the environment of the component, which is represented by the Env interface class (base/env.h). The environment is provided to the component as argument to the Component::construct (base/component.h) function.

Each component is equipped with an initial Entrypoint (base/entrypoint.h) that is accessible via Env::ep(). Each entrypoint is able to respond to both RPC requests and signals. The Component::construct function is executed in the context of the entrypoint. None of the entrypoint's RPC objects or signal handlers will be called before returning from the Component::construct function.

Consolidation of core's SIGNAL, CAP, RM, and PD services

With the new API, each component implicitly requires a session to core's CAP service (to be able to handle RPC requests by the entrypoint) and a session to the SIGNAL service (to dispatch signals by the entrypoint). Therefore, the separation of these services serves no purpose any longer. To simplify the API, the former SIGNAL, CAP, RM, and PD services are now integrated into the PD service. This change has several benefits:

• It reduces the API complexity,

• It reduces the component-startup costs because only one session must be created instead of four.

• It reduces policy with respect to the dimensioning of the various session quotas.

In order to unify the API across all different base platforms, the PD session interface contains no kernel-specific parts any longer. There is now a dedicated sub interface called Native_pd that accommodates such needs. A capability to this kernel-specific interface can by requested via the Pd_session::native_pd accessor method. The kernel-specific interfaces are named Nova_native_pd, Foc_native_pd, and Linux_native_pd.

New log-output facilities

Throughout Genode, we used to rely on C-style format strings for the output of diagnostic information and for the assembly of strings. There are many shortcomings of this approach such as the limitation of the printable types to the built-in format-string specifiers, the lack of type safety, and the mismatch between the implementation and the intuitive expectations by the API users (who usually expect POSIX compliance).

With the current release, we introduce the new log-output facility base/log.h that will ultimately replace the original set of convenience macros PDBG, PWRN, PERR, PINF by C++ function templates using variadic template arguments. The function templates are plainly called log, error, and warning residing in the Genode:: namespace. The replacement for PDBG must be a macro in order to encode the calling function name into the string. As of now, this functionality is not yet covered by base/log.h.

The header base/output.h contains the mechanics for the extraction of a textual representation from values and object instances. It provides the abstract Output interface to be implemented by a consumer of text. Functions for generating output for different types are named print and take an Output & as first argument. The second argument is a const & to the value to print. Overloads of the print function for commonly used basic types are readily provided. The following example illustrates how print functions for custom types may be used.

 enum Test_state { STATE_0, STATE_1 };
 static void print(Genode::Output &output, Test_state const &state)
 {
   switch (state) {
   case STATE_0: output.out_string("STATE_0"); break;
   case STATE_1: output.out_string("STATE_1"); break;
   }
 }
 ...
 Test_state state;
 state = STATE_0; Genode::log("test: ", state, " = ", (int)state);
 state = STATE_1; Genode::log("test: ", state, " = ", (int)state);

Furthermore, there is a function template that is used if none of the type-specific overloads match. This function template expects the argument to be an object with a print method. In contrast to a plain print function overload, such a method is able to incorporate private object state into the output.

The component's execution environment provides an implementation of the Output interface that targets a LOG session. This output back end is offered to the component in the form of the log, warning, and error functions that accept an arbitrary number of arguments that are printed in a concatenated fashion. Each messages is implicitly finalized with a newline character.

XML processing

Since the first version, Genode's init component relied on configurations in an XML-like syntax. For a long time, however, we remained hesitant to make the base system (core and the base API) inherently dependent on XML. For the most part, this hesitance was founded on our observation that XML tends to be disliked in systems-programmers circles. However, over the years, XML organically became the predominant syntax for component configuration as well as for the propagation of state between components. Whereas the syntactical merits of XML are highly subjective and perhaps debatable, the beauty of using XML within Genode lies in its consistent application. For example, it allows us to naturally embed a component configuration in another component's configuration without even thinking about it. Much of Genode's flexibility that we enjoy today can be attributed to this coherency. Granted, this argument would apply just as well to alternatives such as JSON or s-expressions. But we have to take one choice and stick to it. Whereas we found that XML satisfies our needs and actually never stands in the way, the impact on the code complexity is negligibly. Our XML parsing and generating utilities are in the order of 700 lines of code.

With this perspective, we take now the deliberate decision to make XML mandatory for even the lowest-level parts of the framework. For example, even the dynamic linker obtains the policy for diagnostic output from an XML-formatted configuration. Consequently, we moved the XML utilities to base/include/util/.

To ease the use of the XML parsing utilities, we also added the accessors Xml_node::type and Xml_attribute::name that return Genode::String objects.

Dataspace helpers

The use of dataspaces, e.g., for setting up shared memory between components, involves typical sequences of operations, e.g., for attaching the dataspace to the local address space. The Attached_*_dataspace utilities located at os/include/os/ take care of these technicalities. With the current release, we promote them to become part of the base API. Thereby, we can leverage those utilities even for the lowest-level components and internally within the framework. On that account, the corresponding header files were moved to base/include/base/.

The most common use case of Attached_rom_dataspace is the consumption of XML-formatted data. To accommodate this common pattern, we equipped the Attached_rom_dataspace with an accessor plainly named xml. It always returns a valid Xml_node even in the event where the dataspace is invalid or contains no XML. In such cases, the returned XML node is <empty/>. This way, we spare the caller the handling of exceptions that may occur during XML parsing. With this change in place, a configuration attribute can be obtained as follows:

Genode::Attached_rom_dataspace config(env, "config");
...
bool const verbose = config.xml().attribute_value("verbose", false);

Thread API and CPU-session interface

The thread API and the CPU-session interface underwent a major revision.

CPU session interface

Thread API

Most regular components no longer need to use the thread API directly. Instead, the Entrypoint (which is a thread) should be used whenever possible.

Child management

The Child class has been adapted to the changed core services and partially redesigned to enable the implementation of single-threaded runtime environments that virtualize the CPU, PD, and RAM services. It thereby has become free from side effects. I.e., instead of implicitly using Genode::env()->rm_session(), it takes the reference to the local region map as argument. Also, the handling of the dynamic linker via global variables is gone. Now, the linker binary must be provided as constructor argument.

Stylistic changes

Removed and to-be-removed APIs

Low-level OS infrastructure

Enhanced GDB support on NOVA

During the practical work with the GDB monitor (our Genode port of the GNU gdbserver application) and our nightly automated execution of the gdb_monitor.run test, it turned out that there are still situations that the GDB monitor cannot handle correctly. Since the error symptoms often occured sporadically and it was not obvious whether the cause of the error was located in the Genode-specific adaptations of the gdbserver code or in the platform- specific functionality of the Genode base system. As a first step to improve the stability of the debugger we tried to remove existing Genode-specific code from the gdbserver codebase and emulated the Linux-specific C interface instead. This involved a GDB-monitor-local implementation of the waitpid() which is documented relatively well. In the case of an error, there would be a better chance of comparing the GDB monitor- internal execution sequence with a corresponding test case on the Linux version of gdbserver.

The emulation of this interface is not trivial, though, because the behavior of the Linux kernel often differs from the behavior of the Genode base components. For example, when debugging a Linux program, a new thread created by the debug target gets stopped automatically by the Linux kernel, the waitpid() function then reports a SIGSTOP signal for the new thread and a SIGTRAP signal for the creating thread to the gdbserver. This behavior does not match at all with the design of the Genode base system. Therefore, we emulate it in the GDB monitor with the help of a software breakpoint on the first instruction of the new thread and an artificial creation of the corresponding SIGSTOP and SIGTRAP signal reports.

While implementing this mechanism on the NOVA base platform, we encountered several NOVA-specific corner cases. On NOVA, an RPC server is implemented by a so-called "local ECs" - a NOVA thread, which only responds to IPC and has no execution time of its own. When creating a local EC, the initial instruction pointer as passed to the Cpu_thread::start function is always 0. The real instruction pointer is defined by a so-called NOVA portal used for calling the local EC. To determine the correct start address of the thread - as needed by GDB monitor to set the initial breakpoint - changes if the NOVA base system were necessary.

The next show stopper was an attempt by GDB monitor to pause a thread of the debug target could block for quite some time if the particular thread was currently blocking in a NOVA syscall. We also found that GDB monitor could block for quite some time when trying to pause a debug target. This behavior was caused by the targeted thread being in a system call, and therefore within the NOVA hypervisor, by the time of pausing. In that case, the Cpu_thread::pause call returned only after the particular thread got executed in userland again, because only then the current register state of the thread can be transferred into userland for retrieval by GDB. We solved this problem by extending the NOVA kernel, which makes it possible to obtain the current register state of a to-be-paused thread immediately whenever possible.

Furthermore, the NOVA-specific changes for GDB enable the debugger to modify register values and to debug 64-bit applications.

As reference, the ports/run/gdb_monitor.run script demonstrates and tests a selection of the features supported by GDB on Genode.

New support for the Rust programming language

Rust is a systems programming language that currently gains a lot of popularity. It eliminates entire classes of bugs by enforcing memory safety. Unlike languages that rely on a garbage-collecting runtime, compiled Rust programs are able to run on bare-metal hardware. This makes Rust an attractive language for low-level system components.

The current Genode release introduces basic support for executing Rust programs as Genode components. This support includes the build-system integration, the configuration of the LLVM-based Rust compiler, and the port of the low-level language runtime. A simple example is provided via the libports/run/rust.run script and the accompanied code at libports/src/test/rust/. The example runs on the x86 (32 and 64 bit) and ARM architectures.

The port uses the nightly-built rust tool chain from 2016-03-03, so the nightly compiler from that day is guaranteed to work. It can be downloaded with via the following command:

 curl -sSf https://static.rust-lang.org/rustup.sh |\
      sh -s -- --channel=nightly --date=2016-03-03

Alternatively, it can be manually downloaded.

Thanks to Waylon Cude for bringing Rust to Genode!

Dynamic linker

The dynamic linker will now check if the binary pointer is valid before attempting to lookup a symbol. Shared objects with unresolved symbols and missing dependencies, e.g., a library that references errno but is not linked against libc, will now produce an error message when they are loaded by the dynamic linker instead of triggering an ominous page-fault.

New component for writing ROM modules to files

The ROM-to-file component at repos/os/src/app/rom_to_file requests a ROM session and writes the content of the ROM dataspace to a file of a file-system session. It is able to respond to configuration and ROM-module updates. The name of the ROM module must be specified via the rom attribute of the component's <config> node:

 <config rom="pointer"/>

See run/rom_to_file.run for an example.

Thanks to Johannes Schlatow for this contribution!

C runtime

Sysctl

libc_pipe plugin

Device drivers

In this release, we updated several device drivers ported from foreign OSes. In addition, we consolidated all drivers in the dde_linux repository by utilizing a new modular lx_kit and made sure that each driver is using the same Linux version now.

HDA audio driver update

The audio driver was synced with version 5.9 of OpenBSD. In addition to updating the contrib sources, the driver now uses the new component API and reports the internal mixer state.

Reporting of the mixer state is enabled by adding the report_mixer attribute to the drivers configuration and setting its value to yes.

The following snippet illustrates the format of the report:

<mixer_state>
  <mixer field="inputs.beep" value="108"/>
  <mixer field="outputs.hp_sense" value="plugged"/>
  <mixer field="outputs.master" value="128,128"/>
  <mixer field="outputs.mic_sense" value="unplugged"/>
  <mixer field="outputs.spkr_muters" value="hp,mic"/>
</mixer_state>

The mixer state can expose other mixer fields as well, depending on the used hardware. The naming scheme of the attributes intentionally matches the naming scheme of OpenBSD's mixerctl(1) program.

In return, mixer nodes may be used to configure the audio driver by specifying it in the configuration, e.g.:

<config report_mixer="yes">
  <mixer field="outputs.master" value="255,255"/>
</config>

will set the output volume to the highest possible value. Although it is now also possible to update the configuration at run time, this should be done with care. Updating the configuration while the driver is playing or recording may provoke audible artifacts. For now it is best to use the mixer component to regulate the volume rather than adjusting the audio driver directly.

Linux kit

Over the years, the way we handled our DDEs has changed. By now it became clear that it is easier to manage ported drivers without having to rely a generic API like dde_kit. Using Genode primitives directly enabled us to specially tailor each DDE to the driver in question. That being said, when having four different drivers (intel_fb, lxip, usb and wifi) and each one with its own specially tailored DDE, the amount of redundant code became huge. We created the lx_kit that enables us to share code across the drivers to address this issue. Thereby we reduced the amount of redundant code.

This modular lx_kit separates the required back-end functionality of the Linux emulation environment from the front end. Thereby each driver can reuse generic parts and supply more suitable implementations by itself. It is split into several layers whose structure is as follows:

The first layer in repos/dde_linux/src/include/lx_emul contains those header files that provide the structural definitions and function declarations of the Linux API, e.g. errno.h provides all error code values. The second layer in repos/dde_linux/src/include/lx_emul/impl contains the implementation of selected functions, e.g. slab.h provides the implementation of kmalloc(). The lx_kit back end API is the third layer and provides the Lx::Malloc interface (repos/dde_linux/src/include/lx_kit/malloc.h), which is used to implement kmalloc(). There are several generic implementations of the lx_kit interfaces that can be used by a driver.

A driver typically includes a lx_emul/impl/xyz.h header once directly in its lx_emul compilation unit. The lx_kit interface files are only included in those compilation units that use or implement the interface. If a driver wants to use a generic implementation, it must add the source file to its source file list. The generic implementations are located in repos/dde_linux/src/lx_kit/.

The modular lx_kit still depends on the private lx_emul.h header file that is tailored to each driver. Since the lx_kit already contains much of the declarations and definitions that were originally placed in these private header files, those files can now omit a large amount of code.

Wifi driver update

The wifi_drv was updated to Linux version 4.4.3 and thereby adds support for Intel 8xxx wireless cards. In order to ease debugging, the driver now enables its debugging messages when the verbose attribute in its <config> node is set to yes.

USB driver update

The USB driver was updated to Linux version 4.4.3 and like the other drivers incorporates the modular lx_kit. The new driver exposed problems with the EHCI controller on older systems, namely the Thinkpad X201. Using the new USB driver on this machine would freeze the system when USB legacy Support is enabled in the BIOS. The fix is to conduct a so-called USB hand-off that informs the BIOS that the OS wants to drive the USB host-controller and waits until the BIOS has acknowledged the request. Unfortunately, applying this quirk produces problems on certain xHCI host-controllers when using the IOMMU. In this case the driver tried to perform the hand-off request but got stuck while writing to the PCI config space. After about 20 seconds, we observed a DMA fault and the initialization of the USB driver went on. Presumably at this point the BIOS tries to access certain memory regions that - by now - are protected by the IOMMU. When using the IOMMU, we already take precautions, i.e., we look at the RMRR regions and instruct the kernel to configure the IOMMU for specific devices accordingly. We looked at the ACPI RMRR region of the USB on the machine in question and could confirm our suspicion: the registered USB RMRR region is indeed too small. For all we know, that sounds like a bug in BIOS or rather ACPI tables. To accommodate systems that nonetheless need the hand-off quirk and the user wants to use the IOMMU, we added the handling of a bios_handoff attribute to the USB driver configuration. When set to no the driver will not perform any hand-off request. The default setting is yes. If you experience any issues with the new USB driver, disabling the hand-off is advised.

While updating the driver, a regression on the Raspberry Pi was introduced. USB devices that use IRQ endpoints, e.g. USB HID devices, do not work reliably. This issue is still unresolved and dealing with it is postponed until after the release.

Furthermore, the USB session used for implementing native USB device drivers propagates an EP stall error to the client and clears the stall condition by resetting the EP now.

Intel graphics driver update

The Intel graphics driver introduced in Genode release 15.11 was updated to Linux version 4.4.3. The most prominent, functional improvement is support for Intel Skylake graphics cards. Internally, we slimmed the code parts used from the Linux kernel by resigning the ancient framebuffer and framebuffer console layer. The new driver only uses the more modern DRM layer of the Linux kernel. As a side effect, all formerly available heuristics that were applied at initialization time or whenever a display got connected are not part of the driver anymore. Now, the driver only reports any state changes like additional available displays via its report session. On the other hand it always updates the graphics configuration whenever its config ROM module changes.

To automatically control the graphics driver during display connection changes, an example component named intel_fb_controller is now available at repos/dde_linux/src/test/framebuffer/intel. This component reacts on report changes of the Intel graphics driver and configures it in a way that all available displays are showing one and the same framebuffer with their maximum resolution. Thereby displays with a minor resolution show the upper left corner of the whole framebuffer.

Enhanced ACPI support

Modern PCs provide an enormous number of ways to monitor and configure the system via the Advance Configuration and Power Interface (ACPI) Specification. Since version 12.02, we have already a basic ACPI driver in Genode, which is mainly used to look up low-level data via some clever pattern-matching heuristics. These information, like interrupt remapping, are sufficient to bootstrap and setup the Genode user-level part of the system. We wanted to go beyond this feature set and leverage further - dynamic - aspects of ACPI such as system state changes of batteries in notebooks or lid status.

The ACPI Component Architecture project ACPICA develops and maintains an operating-system-independent reference implementation of ACPI, which can be used by operating systems like Genode to utilize the full functionality of modern PCs. So we took the reference implementation of ACPICA and ported it to Genode. The port itself was relative straight forward and really a pleasure. The interfaces and abstractions to the operating system are well chosen by the ACPICA project, clearly documented, and a porter is well guided by extensive explanatory documentation.

The port is hosted in the libports repository as a standalone library without any additionally dependencies (beside Genode's base library). To utilize and experiment with the ported library, we started to develop a Genode application called app/acpica, which utilizes the library. We experimented and managed to enable the ACPI lid, ACPI embedded controller (e.g. Fn keys), ACPI AC adapter, ACPI smart battery subsystem, and ACPI fixed events, e.g., power button, on some modern Intel Skylake notebook and partly also on some older Lenovo machines, e.g. X201. Additionally, we added support to reset and power-off machines via ACPI.

ACPI state changes are reported by the acpica application by setting the config attribute report to "yes".

<start name="acpica">
  <config reset="yes" poweroff="yes" report="yes"/>
  ...

Whenever such a state change is detected, the application generates the appropriate report named acpi_lid, acpi_ac, acpi_battery, acpi_ec or acpi_fixed. The detailed content of the reports is documented in the README of app/acpica or can be manually experienced by trying out the acpica.run script in the libports repository.

In order to reset or to power-off machines, the config attribute reset respectively poweroff must be set to "yes", as shown before. If one of both attributes is configured, app/acpica opens a ROM session called "system" and monitors changes of that ROM. The ROM must be XML in the following form:

<system state="..."/>

If the state attribute is set to "reset" or to "poweroff", app/acpica will try to reset respectively power-off the machine immediately. The operation may fail if the hardware resources are owned by some other components in the Genode system. E.g., on some machines we tested, the reset operation fails because the required I/O ports are owned and are used by the x86 platform driver. For such cases, we extended the platform driver by an additionally config parameter "system" which must be set to "yes", e.g.:

<start name="platform_drv" >
  ...
  <config acpi="yes" system="yes">
  ...

With this configuration in place, the platform driver also opens and monitors the "system" ROM session and reacts upon a state change to "reset". If the platform driver owns the required I/O ports, it will trigger the ACPI reset.

In the current state, we still use our old simple ACPI driver to detect basic necessary information for the x86 platform driver. The ACPI driver reports all findings in form of a report, which is provided as a ROM session to the platform driver. Later on, after the platform driver has announced its service, the acpica application takes over the ACPI functionality of our old ACPI driver and takes care of all dynamic ACPI events.

It first looks a bit cumbersome, however the reasons are twofold. First, we wanted to start to experiment with the acpica library without putting our ACPI driver at danger. Second, we wanted to see where the complexity of the additionally ACPI features leads us. If one is using sloccount, or cloc, as complexity measure and applies the tool to the respectively folders, the following numbers show up:

 repos/os/src/driver/acpi/      ~1000
 contrib/acpica-<hash>/         ~120000  ACPICA library
 repos/libports/src/lib/acpica/ ~600     Genode-specific ACPICA libary support code
 repos/libports/src/app/acpica/ ~1000    application using the ACPICA Genode port

Of course, the numbers are inaccurate and the comparison is unfair since we do not take into account, which files of the acpica library are actually in use. Furthermore we can get more functionality with the acpica library, which we never could achieve with our own basic ACPI driver implementation. However, the point here to be made is, that we have to add much complexity to get a full ACPI capable system and that solely a small amount of the complexity actually is really required to drive an operating system like Genode.

Generalized SDHCI driver

The SDHCI driver was originally created for the Raspberry Pi. However, the same host controller is used also in other platforms, in particular Xilinx Zynq. Hence, the existing driver was generalized to become usable on such platforms. Thanks to Timo Wischer for this contribution.

Libraries and applications

LxIP update

LxIP is the port of the Linux TCP/IP stack as a library on Genode. Along with the work described in Section Linux kit, LxIP was updated to Linux version 4.4.3 and uses the lx_kit now.

Qemu USB

The QEMU USB library handles EP stalls now. In particular, this fix enables the use of USB storage devices in VirtualBox that do not support certain SCSI commands, e.g. READ_FORMAT_CAPACITY, and will stall if they receive such a command. Windows guests typically use the aforementioned command to check if the USB storage device in question is in fact an USB floppy drive.

Platforms

Generalization of platform-specific headers

In anticipation of the planned binary compatibility of Genode components across different kernels, we unified most parts of Genode's base API and largely removed the dependency on platform-specific types. The most profound change is the interface of the IPC library, which used to depend on platform-specific message-buffer layouts. Besides unifying the message buffer classes across all platforms, we reconsidered the roles of the IPC-library classes such as Ipc_marhsaller, Ipc_server, and Ipc_client. This led to several additional simplifications in the server-loop implementations, which makes the flow of control and information much more obvious, yet is also more flexible. I.e., on NOVA, we don't even have the notion of reply-and-wait. Now, we are no longer forced to pretend otherwise.

NOVA microhypervisor

The kernel received minor adjustments because of the ACPI work. One curious performance issue, we actually detected and hunted before our ACPICA library work. It happens that a modern Intel Skylake notebook (Core i 6th generation) running Genode/NOVA scenarios like noux tool-chain or Virtualbox performed really bad - sometimes it was only as fast as a 1th generation Intel Core CPU as used in X201 notebooks. After some mysterious hunting of possible reasons, it finally turned out that the UEFI vendor did not disable ACPI GPE (General Purpose Events) events properly when handing over control to the boot loader and kernel. As soon as the NOVA kernel enabled the ACPI interrupt (normally IRQ 9) because it utilizes the ACPI PM timer feature, the kernel got a storm of GPE interrupts, which got not handled properly. Still the system was alive and made progress, but the performance was really bad. We changed the kernel to disable all event sources enabled in the ACPI GPE0/1 registers. Beside that, using the acpica application also solves the performance issue, since the library also resets the GPE registers during initialization. However, currently the acpica application is optional and not loaded in all scenarios.

On 64 bit, NOVA supports the so called PCID feature, aka tagged TLB. The hardware actually supports up to 4096 processes at a time, which can be used with tagged TLBs. Unfortunately the original PCID allocator wrapped after 4096 PCIDs, which caused - beginning with the 4097th process - to accidentally re-use the same PCIDs and therefore the same TLB entries of long living processes, like kernel, core, init, and drivers. This leads to the interesting phenomena of bugs. We changed the allocator from a monotonic increasing number to a bit allocator, maintaining the used and free PCIDs more accurately.

Additionally, we extended the kernel to support scenarios on Genode where capabilities are forwarded from a capability sender to a capability receiver through one or more intermediary components (like servers). In such scenarios, the intermediary components don't need the forwarded capability for some reasons and want to free and re-use the used capability index. Unfortunately, or actually intentionally, the syscall revoke will not just revoke the local capability but all subsequent derived capabilities. Because of this kernel behaviour, we had several quirks in Genode/base-nova, especially in the user level capability map implementation and in the Genode entrypoint reply handling code to deal with such situations. With all the Genode base API changes and unification efforts of all Genode base platforms, the quirks became obvious obstacles. With this release, we added support to the kernel, to just locally drop the accessibility to the unneeded capability, but not the accessibility of the so far subsequent derived capabilities. With the kernel extension, we were able to remove the mentioned base-nova quirks.

Execution on bare hardware (base-hw)

For running Genode scenarios on our custom kernel (base-hw), two hardware timers are needed. One timer is used by the kernel as the basis for the preemptive scheduling. The other timer is used by the user-level timer driver as the timing source for user-level components.

On NOVA, we gathered good experiences with using the kernel's scheduling timer as the basis for the user-level timer. Eliminating the need for a real timer device driver (like for the PIT on x86) reduces the overall complexity. The interrupt load becomes lower without the userland triggering timer interrupts. And since the kernel uses CPU-core-local timers (i.e. the local APIC timer on x86) as opposed to a global timer in the userland, expensive cross-CPU-communication is avoided.

With the current release, we applied the lessons learned to our base-hw kernel. As a further motivation, the removal of the dependency on a timer device clears the way to run multiple Genode instances on top of the Muen separation kernel.

The timeout feature has the form of a new timeout system call that binds a signal context to a timeout. Hence, timeouts are delivered asynchronously, like interrupts, to the user-level timer service. The actual time can be requested via the timeout_age_us system call, which returns the time since the last timeout was installed.

Linux

The main purpose of Linux as Genode base platform are rapid prototyping and the development of components that do not depend on specific hardware properties. During early development (at least with C++) the most prevalent fatal bugs result in segmentation faults due to invalid pointers or insufficient stack size. Unfortunately, our platform code for Linux did not disclose much information about the exceptions that may occur and even remained silent about errors in some situations. With this release, we improve the exception-signal handling and use an alternate signal stack. The alternate stack ensures in almost all cases that Linux applications are able to handle exceptions including segmentation faults caused by stack overflows. We also enabled this facility for hybrid Linux applications.

Tools and build system

Usability improvements of the ports tools

The ports tool set introduced in Genode 14.05 has become an integral part of the work flow for Genode developers. With the current release, we improve the usability of the prepare_port tool in two respects. First, the tool now accept a list of ports instead of merely a single argument. This alleviates the need to manually re-execute the tool with different arguments. Second, if the build system encounters a missing port, it no longer backs out immediately but collects all the (potentially more than one) missing ports that are required for the build. It then presents the user with a ready-to-use command to install all missing ports at once, which greatly improves the experience of working with sophisticated system scenarios. For example, when attempting to execute the virtualbox.run script with a freshly cloned Genode source tree, the build system produces the following error message:

Error: Ports not prepared or outdated:
  dde_linux libc libiconv nova qemu-usb stdcxx virtualbox x86emu

You can prepare respectively update them as follows:
  .../prepare_port dde_linux libc libiconv nova qemu-usb stdcxx virtualbox x86emu

Furthermore, one may state the number of ports that shall be prepared in parallel at a max by using the -j parameter. If -j is not set by the user, the tool acts as with -j1.

Since the prepare_ports tool has completely replaced the former "make prepare" mechanism, we finally removed the last traces of the old mechanism in the form of the makefiles present in the respective source-code repositories.

Updated tool chain

Genode 16.05 requires a tool-chain update, which can be downloaded as precompiled binary archive for 32-bit and 64-bit Linux or built according to the tool-chain documentation. With the updated tool chain, we enable the __cxa_demangle() function to be able to print user-readable names of uncaught exceptions. In the course of our GDB improvements, we enhance the x86 debugging support for 64-bit and update the required GDB tools. Furthermore, we added the RISC-V relevant tools to the binary archive to ease developing Genode components for this platform.

Removal of stale features

We originally added chroot support to the Linux version of Genode to accommodate the use of Genode as middleware on Linux. We enabled the configuration of custom UIDs, GIDs, and chroot paths for components started by init. However, apart from a brief period of time when we experimented with the idea, it is no longer pursued. Now, with our aspiration to attain binary compatibility across kernels, we removed the Linux-specific chroot support.