Release notes for the Genode OS Framework 18.02
After being developed for over a decade, Genode remained a mystery for many people who looked at the project from a distance as it does not seem to fit any established category of software. In 2018 - declared as the Year of Sculpt on our roadmap - this will hopefully change. Genode 18.02 features the first revision of Sculpt, which is a Genode-based operating system for general-purpose computing. After being used as day-to-day OS by the entire team of Genode Labs for several months, we feel that the time is right to share the system with a broader audience (Section Sculpt for Early Adopters).
One fundamental feature of Sculpt is the ability to install and deploy software from within the running operating system, which is universally expected from any modern general-purpose OS. Section On-target package installation and deployment presents Genode's unique take on the topic of software installation and deployment.
Besides Sculpt, the current release has no shortage of other improvements. Genode's growing arsenal of 3rd-party software received profound updates and additions, including VirtualBox, Muen, seL4, several GNU packages, and libraries. Also the user-level networking stack - including the Linux-based LxIP stack and our custom NIC-router component - received a lot of attention. Thanks to the added network driver for i.MX-based hardware, this networking infrastructure becomes usable on embedded platforms based on this SoC. Furthermore, the current release continues the cultivation of the Nim programming language for Genode components.
Sculpt for Early Adopters
The current release features the first revision of Sculpt, which is a Genode-based operating system for general-purpose computing. This initial version is called Sculpt for Early Adopters (EA). Its target audience are enthusiasts who are already familiar with Genode and are eager to use a Genode-based operating system on their machines. As outlined on the roadmap, later versions will become increasingly approachable.
Please refer to the official Sculpt documentation to step right into the adventure.
On-target package installation and deployment
In May last year, we introduced the package-management concept for Genode to pursue two goals. First, to overcome the naturally limited scalability of composing Genode systems solely from source. This limit became evident in complex system scenarios that incorporate a huge amount of 3rd party software. Thanks to the introduced depot concept and its integration in Genode's workflow - in particular the run tool - the work of system integration became much more structured (by caring about packages instead of individual build targets), robust (by avoiding conditions in run scripts), and quick (by the accelerated test cycle when using pre-built packages).
The second goal is the ability to update and extend a running Genode system on the fly. We are happy to have reached this goal with the current release. As exemplified by the Sculpt scenario, packages cannot only be used as building blocks for system images but also as subsystems dynamically installed and deployed on target. Even though installation and deployment are closely related topics, both involve distinct challenges, which allow Genode to shine.
Installation / update
In traditional operating systems, the installation and update of system software is the job of privileged programs. For example, a package manager in a GNU/Linux system is typically executed with root privileges. This is troublesome because the functionality of such a program is extremely complex. In particular it is exposed to the network and has to parse content originating from potentially untrusted parties. Therefore, potential software vulnerabilities should be expected. However, in modern OSes, these programs are just assumed to behave correctly. If this overly optimistic assumption doesn't hold, the entire system is at risk.
Genode helps us to mitigate this problem by modelling each installation step as a distinct component composition where each component has a well-defined and extremely narrow role. The installation is an iterative sequence that is orchestrated by the so-called download-manager component (Figure 2).
Initially, the download manager receives a list of content to be installed into the local depot, which is stored on the file system. The depot may already be populated with (portions of) this content. In the first step, the download manager must determine the parts that are missing. To do that, it does not access the file system directly but instead hands over this task to a disposable helper component called depot-query that is spawned within a dynamic init instance. This indirection has two benefits. First, the download manager is not bothered with the complexity of accessing the file system. It does not even have any notion of files. Second, the download manager is effectively shielded from the file system. Should the file system misbehave, the liveliness of the download manager remains unaffected.
The depot-query component reports its findings to a report session. The report eventually reaches the download manager as an updated ROM module. Given the list of missing content, the download manager has to determine the information of where to obtain the content from and the public key of the content creator. This information is contained within the depot. So the download manager issues another request to the depot-query component in order to obtain it.
Once the depot-query component has responded, the download manager knows what content to get, where to get it, and how to verify it. To download the content, it changes the dynamic init instance as follows.
The depot-query component is now gone. Actually, the entire depot has moved out of sight. Instead, a fresh fetchurl component is spawned. This component is connected to the network as well as the writeable download directory public/. Internally, fetchurl employs a complex software stack, which includes the C runtime, curl, libssl, and libssh. Hence, we expect this component to be vulnerable. Since it is facing the network, we assume that vulnerabilities are exploitable. In the worst case where the component is completely in the hands of an attacker, it may write wrong content into the public/ location. But compared to executing curl or wget as root on a traditional Unix system, the reach of an attack is quite limited. For example, the mere existence of the download manager remains completely out of view of fetchurl. However, the content of public/ must not be trusted. To reinforce trust in the downloaded content, the content is accompanied with cryptographic signatures created by the content creator. Before we touch the content, we first check its authenticity. To perform this verification step, the download manager reshapes the dynamic init instance as follows.
Note that fetchurl exists no more and network connectivity is cut, effectively disposing any form of malware that might have infected fetchurl. Next a new verify component enters the picture. It is configured with a list of content to check, the signatures of the content, and the public key of the content's presumed creator. Since it accesses the public/ location exclusively, it is not prone to any potential time-of-check to time-of-use problems during the verification. Under the hood, the verify component employs a hugely complex implementation based on GnuPG. It would be naive to fully trust this code. However, when embedded in our scenario, the reach of a bug is limited because the verify component has no access to any mutable system state. It could merely give the wrong answer (which is of course bad but there is no way we can magically solve this).
Knowing that the downloaded content is indeed the same content as intended by the creator, it is time for extraction. For this step, the download manager - again - reshapes the dynamic init instance:
This time, both the public/ location as well as the trusted depot/ are visible and a new extract component is spawned. As the depot may host content from multiple sources, which potentially distrust each other, the content of each content provider resides in a dedicated subdirectory within the depot. Instead of handing over access to the entire depot to the extract tool, we mediate the file-system access via a chroot component that limits the view to the depot-provider's respective subdirectory. In the worst case where a misbehaving content provider delivers a forged (but correctly signed) archive to exploit a vulnerability of the extract component, the reach of the attack remains limited to the content provider's space within the depot.
After the extraction step has completed, the depot is populated with the new content, which may - in turn - include new dependency information. At this point, the download manager starts a new iteration. This iterative process terminates as soon as the depot-query component signals that no content of the software installation is missing.
The bottom line here is that we are able to use complex and useful software like curl, libarchive, liblzma, and GnuPG while largely distrusting it. In contrast to this software that sums up to hundreds of thousand lines of code, the download manager comprises less than 1000 lines of code. The software installation procedure described above is implemented by the depot_download subsystem hosted in the gems repository and illustrated by an equally named run script. It also forms the basis of the install/update mechanism of the Sculpt scenario.
Deployment
Once software has entered the system in the form of depot content, the remaining question is how to turn this content into running subsystems. The answer is given by the following illustration.
Like for the installation process described above, the scenario employs a dynamic init instance that is accompanied by an orchestrating component. The latter is called depot-deploy. The depot-deploy component queries information from the depot using the same depot-query component that was used during the installation. Based on the returned blueprint information for the to-be-deployed subsystems, it generates the configuration for the dynamic init instance. The subsystems hosted within this init instance access the depot content via mere ROM sessions as provided by the FS-ROM component. This makes the use of the depot transparent to the hosted subsystems.
The depot-deploy component is located in the gems repository and accompanied by a same-named run script. More importantly, it is featured in the deploy runtime of the Sculpt system.
Base framework and OS-level infrastructure
Increased default warning level
For building Genode components written in C++, the compiler flags -Wextra, -Weffc++, and -Werror are now enabled in addition to -Wall by default.
If this strict warning level is inapplicable for a given component or library, it is possible to explicitly disable the strictness in the respective build-description file by adding the following line:
CC_CXX_WARN_STRICT =
We adjusted almost all the code of the base, base-<kernel>, os, and demo repositories to comply with this new warning level. For most components hosted in the higher-level repositories (libports, ports, dde_*, gems), the strictness is disabled as of now and will be enabled component-wise wherever feasible.
While adjusting our code base, we identified the following patterns worth mentioning:
-
A class with virtual functions can no longer publicly inherit base classes without a vtable. The inherited object may either be moved to a member variable, or inherited privately. The latter would be used for classes that inherit List::Element or Avl_node. In order to enable the List and Avl_tree to access the meta data, the List must become a friend.
-
Instead of adding a virtual destructor to abstract base classes, we inherit the new Interface class, which contains a virtual destructor. This way, single-line abstract base classes can stay as compact as they are. The Interface utility resides in base/include/util/interface.h.
-
With the new warning level, all member variables must be explicitly initialized. Basic types may be initialized with =. All other types are initialized with braces { ... } or as class initializers. If basic types and non-basic types appear in a row, it is nice to only use the brace syntax (also for basic types) and align the braces.
-
If a class contains pointers as members, it must now also provide a copy constructor and assignment operator. In most cases, one would make them private, effectively disallowing the objects to be copied. Unfortunately, this warning cannot be fixed by inheriting our existing Noncopyable class (the compiler fails to detect that the inheriting class cannot be copied and still gives the error). For now, we have to manually add declarations for both the copy constructor and the assignment operator as private class members. Those declarations should be prepended with a comment like this:
/* * Noncopyable */ Thread(Thread const &); Thread &operator = (Thread const &);
In the future, we plan to revisit these occurrences and try to replace the pointers with references. In the presence of at least one reference member, the compiler would no longer implicitly generate a copy constructor. So we could remove the manual declaration.
The following caveats are expected, even if you disable the strictness in your component:
-
If your component has a class called Interface, it may collide with the new Genode::Interface class. You may have to disambiguate the names.
-
The Genode::Rpc_client is no longer a Genode::Capability. Hence, classes inherited from Genode::Rpc_client cannot refer to a Capability but must refer to Genode::Capability.
-
The Surface class is no longer copyable, which led to API changes of users of this class. E.g., the Nitpicker_buffer utility does no longer offer accessors for the contained surfaces but a new apply_to_surface method that takes a lambda function as argument.
Init
Init selects session routes based on the requested service and the client's label. The latter can be matched as label (exact match), label_prefix, or label_suffix (either end of the label matches). With the new version, these options are complemented with an additional label_last attribute that covers the prominent case where the last part of the label identifies a requested resource at the server. A typical example is the routing of a ROM session based on the name of the requested ROM module.
Reflecting the core log to the application level
Core records now log messages in a ring buffer and exports this memory as ROM named core_log. User applications may monitor this ring buffer and present or transfer the content as appropriate. The example component in repos/os/src/app/log_core transforms the content into normal log messages, which may be routed to graphical terminals or stored on file systems, e.g. by using the fs_log server.
NIC-router improvements
During the past three months, the NIC router has received several improvements that were mainly inspired by our daily experience with the component as part of our Sculpt based working environments.
The most notable new feature is the support for multiple NIC sessions at one domain. If multiple NIC-session clients connect to one domain, the NIC router acts as a simple hub between them. I.e., for every packet that is routed to the domain, each connected session receives a copy of the packet. The same applies for domain-local packets, meaning packets that target an IP address inside the IP subnet of the domain they came from. This domain-local forwarding applies before considering any other routing rules. So, in other words, it is not possible to route such traffic to another domain.
Furthermore, the logging features of the NIC router were improved. First, the router is now capable of periodically sending a report via Genode's report session. This can be activated by adding the new <report> node to the router configuration:
<config> <report interval_sec="5" bytes="yes" config="yes"> ... </config>
So far, the report provides per-domain information about the amount of sent and received data (bytes attribute) and the current IPv4 configuration like IP address, subnet mask, and gateway address (config attribute).
Second, there is a new verbosity option in the <config> node:
<config verbose_domain_state="yes">
When this option is set, the NIC router will output a short message to the log for each general state change of a domain. Currently, this includes the IP-configuration state (IP address, subnet mask, gateway address) and the number of connected NIC sessions. This is a useful addition because the purpose of the regular verbose option is to give a very deep insight into almost every activity of the router, which is vital for debugging sophisticated problems but normally floods the log. Therefore, the regular verbose option is not viable for complex setups like a Sculpt desktop environment. In such a context, the new domain-state verbosity is pretty discreet but already gives a good hint on why, for instance, packets get dropped despite the routing rules being correct.
Last but not least, the timeout configuration of the NIC router has been reworked and now allows for a much more precise adaption to the network environment. The former rtt_sec attribute of the <config> node has been replaced by the following new attributes (default values shown):
<config dhcp_discover_timeout_sec="10" dhcp_request_timeout_sec="10" dhcp_offer_timeout_sec="10" udp_idle_timeout_sec="30" tcp_idle_timeout_sec="600" tcp_max_segm_lifetime_sec="30">
Details about the new attributes can be found in the os/src/server/nic_router/README file. The default values should be appropriate for the common use case so that specifying them is normally not necessary.
New watch mechanism for file-system session
The file-system session already provided a way for watching files or directories for changes. However, the original mechanism was arguably hard to use. In addition to opening the to-be-watched file-system node, the client had to submit a so-called content-changed request into the session's request queue. In turn, the server delivered the change notification by acknowledging this request.
The new mechanism is much less bureaucratic. A file or directory can be watched by opening a watch handle rather than submitting a CONTENT_CHANGED packet to the server. Whenever a change happens at a node with an open watch handle, a CONTENT_CHANGED packet will be sent from the server to the client. This serializes the registration with other handle operations and separates I/O handle state from notification handle state.
C runtime
We changed libc's handling of clock_gettime to be explicitly configurable rather than relying on built-in heuristics. With the new version, the libc opens a timer session as a time source only if the rtc attribute of the <libc> configuration node is defined. If not configured, clock_gettime returns 0.
This change may require the adjustment of components that implicitly rely on the libc as time source. To enable such a component to use relative time (based on a timer session) but no wall-clock time, one can manually provide a pseudo real-time clock value as follows:
<vfs> <dir name="dev"> <log/> <null/> <inline name="rtc">2000-01-01 00:00</inline> </dir> </vfs> <libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc"/>
GUI stack and terminal improvements
Nit-FB improvements
The nit_fb component provides a framebuffer and input service while using the nitpicker GUI server as back end. The new version adds the initial_width and initial_height attributes, which accommodate the use case where nit_fb is used in a dynamic fashion like as a client of a window system. Here, the initial dimensions define the initial window size but - in contrast to the existing width and height attributes - the actual size can change afterwards.
Terminal resizing
The terminal-session interface gained the ability to propagate resize events from the server to the client. The new version of the graphical terminal uses this mechanism to support window resizing as well as dynamically changing the font size. At the client side, noux has become able to reflect terminal-size changes to noux applications. Applications based on ncurses (e.g., vim) are able to gracefully respond to such changes now.
Using chroot to enforce read-only file-system access
By placing a chroot component in-between a file-system client and server, the client's view on the file system can be limited to a specific directory. With the current release, chroot can additionally be used to restrict a writeable file-system session to become read-only. This is accomplished by the new writeable attribute of chroot's policy nodes. By default, it is set to "no".
API changes
Noncopyable AVL node/tree
Copying an AVL node generally violates the integrity of the corresponding tree. To rule out subtle bugs where AVL nodes are accidentally copied, AVL nodes are no longer copyable.
New Buffered_xml utility
The Buffered_xml utility located at os/buffered_xml.h simplifies the implementation of dynamically reconfigurable components that need to keep a verbatim copy of certain parts of their configuration during configuration updates.
New List_model utility
More and more components respond to dynamic configuration updates. For most components, such updates are quite simple: replace an old internal state by a new one. But in cases like init, menu_view, or window decorator, a differential update is in order. Until now, each of these components employed custom code for this task. As this code is not trivial, a common solution is preferable. This solution comes in the form of the new List_model utility located at base/include/util/list_model.h. It introduces a light-weight formalism to feed a component-internal data model from an externally-provided XML structure.
Dynamically expandable reporter utility
In many cases, components that generate reports don't explicitly handle the situation where the default buffer size of 4096 bytes is exceeded by the report. This problem is easy to miss because reports are often small at testing time but become larger when deployed in complex scenarios. In most cases, the best way to handle an Xml_generator::Buffer_exceeded exception is upgrading the report session. The new Expanding_reporter that accompanies the original Reporter in os/reporter.h eases the handling of this common case.
Languages and runtime environments
Nim programming language
A new Nim library for constructing Genode servers is now available in the World repository. This module provides utilities for the asynchronous session-creation procedure introduced in the 16.11 release. Some introductory code snippets are provided here for the adventurous.
An example of server creation using the genodeservers module:
import romclient, genodeservers var sessionsRom = newRomClient "session_requests" # synchronously open a ROM client to the parent romContent = sessionsRom.stream.readAll() # copy the ROM content to a heap string requestsParser = initSessionRequestsParser(romContent) # a state machine for parsing 'session_requests' XML for id, service, label in requestsParser.create: # the `create` iterator provider for the parser # hides the details of parsing the XML data discard txBufSize = requestsParser.argInt "tx_buf_size" # extract typed session arguments from the current parser state discard label.lastLabelElement() # label handling utilities are provided if service == "MyService": myCreateSessionProc(id, label)
This module streamlines the handling of session metadata, but the developer must still provide hand-crafted wrappers over the C++ methods for managing RPC objects and passing session capabilities to the parent. Most notoriously a global pointer symbol, `genodeEnv`, is used to expose the component environment object. In the future, this will be replaced by a typed object passed from runtime to an application entry procedure.
type MySessionCapability {. importcpp: "My_session::Session_capability", header: "my_session/capability.h".} # import a capability type type MyNativeSessionBase {. importcpp: "My_session::Session_rpc_object", header: "my_session/rpc_object.h".} # import C++ session RPC object type MyNativeSession = Constructible[MyNativeSessionBase] # apply the C++ Constructible template to defer calling # the object constructor proc construct(cppObj: MyNativeSession) {. importcpp: "#.construct(*genodeEnv)".} # call the C++ constructor, passing the global Genode::Env proc manage(cppObj: MyNativeSession): MySessionCapability {. importcpp: "genodeEnv->ep().manage(*#)".} # call a method from the gobal Env, dereferencing # thru the Constructible template type MyNimSessionObj = ref object cppImpl: MyNativeSession cap: MySessionCapability id: SessionId # C++ RPC objects are best kept in native # reference-counted Nim objects proc manage(obj: MyNimSessionObj) = obj.cppImpl.construct() # call our wrapped constructor GC_ref(obj) # manually increase the reference count on our session # object to prevent the component entrypoint from # referencing an RPC object that has been lost and # freed from the heap obj.cap = obj.cppImpl.manage() # store our capability proc myCreateSessionProc(id: SessionId): MyNimSessionObj = result = new MyNimSessionObj # create our object on the heap result.manage() # construct and manage our RPC object result.id = id # store the session id from our parent
Procedures for calling Nim code from an RPC object, dissolving and destructing RPC objects, and managing the session lifetime are exercises left to the reader.
Updated VirtualBox
Our VirtualBox port got updated from version 5.1.22 to version 5.1.32 in order to leverage the security updates and improved audio support. Additionally the boot time of Linux guests got improved by adjusting our custom virtualization back end.
Libraries and applications
New trace-logging component
The new trace-logger component can be used to easily gather, process, and export different types of tracing data. Furthermore, it marks the next step towards a user framework that makes access to Genode's manifold tracing abilities (13.08, 13.11, 15.08) intuitive and convenient.
The component can filter the available tracing subjects according to session label policies and thread names. The processing of the tracing data can then be configured for each selected subject individually, for groups of subjects, or for all subjects together. The resulting data is exported as log output.
This is an example configuration of the trace logger, which shows the default value for each attribute (except policy.thread and policy.label):
<config verbose="no" session_ram="10M" session_arg_buffer="4K" session_parent_levels="0" period_sec="5" activity="no" affinity="no" default_policy="null" default_buffer="4K"> <policy label="init -> timer" /> <policy label_suffix=" -> ram_fs" /> <policy label_prefix="init -> encryption -> " thread="worker" policy="null" buffer="4K" /> </config>
The most important features so far when it comes to processing the traced data are:
-
Trace CPU activity and affinity (activity and affinity attribute),
-
Install individual policies for the creation of further tracing data (policy attributes) for instance, rpc_name for a log of issued RPC calls),
-
Dimensioning the subject-local trace buffers and the frequency of Trace Logger data examination (buffer and period attributes), and
-
Configure the session to the Tracing server (session attributes).
A comprehensive documentation of the trace-logger component can be found in os/src/app/trace_logger/README. An example of how to use the component is given through the run script os/run/trace_logger.run.
New component for extracting archives
The new extract component located at libports/src/app/extract extracts the content of an arbitrary number of tar.xz archives according to its configuration. It is used by the depot-download subsystem described in Section On-target package installation and deployment. The component is accompanied by the run script libports/run/extract.run that illustrates its use.
New signature-checking tool based on GnuPG
The on-target installation of software packages requires a way to verify cryptographic signatures of downloaded content within a Genode system. The new verify component located at ports/src/app/verify facilitates the code of GnuPG to verify detached OpenPGP signatures against public keys. Since GnuPG depends on libgcrypt and libgpg-error, ports of those libraries were added to the libports repository. The component comes with the run script ports/run/verify.run that demonstrates its usage.
Fetchurl component for downloading files
Fetchurl is a component for downloading files from the network, based on the curl library. It used to reside in the genode-world repository. Since it has become a mandatory part of Genode's on-target software installation mechanism, we have moved it to the libports repository now. Besides this relocation, fetchurl received a welcome modernization. In particular, the new version uses the modern socket-fs infrastructure of the libc instead of relying on the deprecated libc_lwip plugin as a hard-wired dependency.
New interactive FLIF image viewer
A simple image viewing application for the FLIF lossless image format was written from scratch using the FLIF reference decoder library. The viewer can be used to interactively view a directory of images and supports animation of GIF-like FLIF files.
Ported 3rd-party software
With the current release, the following 3rd-party software becomes available on Genode:
- libarchive
-
is a library for uncompressing and extracting various archive formats. It nicely wraps format-specific libraries like zlib behind a unified and easy-to-use API. The port can be found in the libports repository.
- lz4 and liblzma
-
implement modern compression algorithms as supported by libarchive. Thanks to Ben Larson for contributing the port of these libraries.
- Tcl
-
is used as scripting language for various Genode tools. With the new check_abi tool described in Section Automated ABI consistency checks, the Tcl shell tclsh has become a dependency of the build system. Therefore, we made tclsh available as noux package. Note, however, that this port comprises solely the functionality needed for simple scripting.
- FLIF
-
is a library for the Free Lossless Image Format. Thanks to Emery Hemingway for making it available in the genode-world repository.
- JSON-C
-
is a library for processing JSON-formatted data. Thanks to Johannes Kliemann for contributing the port to the genode-world repository.
- Drill
-
provides a utility for DNS testing. Thanks to Emery Hemingway for adding it to the genode-world repository as a side activity of improving Genode's network stack.
Updated packages for the Noux runtime environment
The current release updates the following noux packages: less (version 487), grep (version 3.1), coreutils (version 8.29), tar (version 1.30), findutils (version 4.6), which (version 2.21), sed (version 4.4), and bash (version 4.4.18). Thanks to Hinnerk van Bruinehsen for this welcome contribution.
Device drivers
Ethernet-driver for i.MX-based Wandboard
The current release contains a port of the Linux kernel driver for the Ethernet card family originally produced by Freescale. We followed our established approach to tailor an independent device-driver environment (DDE) for the specific driver. To profit from synergies with the existing drivers of the dde_linux repository, we took the Linux kernel 4.4.3 as reference.
For now the current version is limited to support the Wandboard Quad as this is the i.MX-based board that is nightly tested by our infrastructure. The support of other boards using the same IP core is planned for future releases.
The driver can be found in dde_linux/src/drivers/nic/fec. To test the driver, no further configuration is needed and you can have a look at one of the automatic network tests, like lwip.run, as a reference.
Platforms
Execution on bare hardware (base-hw)
Thanks to Johannes Schlatow from the TU Braunschweig, the support of the Zynq-7000 boards by our base-hw kernel got extended. It is now possible to use all CPU cores instead of only the primary one.
Updated Muen separation kernel
The Muen SK port has been updated to the latest development version 0.9. The most notable features and improvements are the Crash Audit facility and support for MirageOS/Solo5 subjects which may be executed alongside Genode/base-hw.
Thanks to this feature, the Muen project has reached a milestone by self-hosting the https://muen.sk website on a Muen system. Currently, the network driver is provided by a Linux subject but with some work it should be possible to replace it with a Genode/base-hw nic_drv in the future.
Further details regarding Muen v0.9 can be found in the project's release notes https://groups.google.com/forum/#!topic/muen-dev/FPL9sc4yaBE.
Updated seL4 kernel
Our remaining patches regarding UEFI framebuffer support got integrated into the upstream codebase of the seL4 kernel. Hence, we updated our seL4 port to the upstream version containing our patches.
Build system and tools
Package management
The package-management tools introduced last year have become a vital part of Genode's workflow.
- Package management documentation
-
https://genode.org/documentation/developer-resources/package_management
Prompted by the development of the on-target installation and deployment mechanism featured in the current release, the tools received the following refinements:
- Use of tar.xz as archive format
-
This change significantly reduces the size of published depot content compared to the previously used tar.gz format.
- Subdirectories for archive versions
-
In the original version of the depot layout, archives were named as <archive-name>-<version>. Hence, the depot - in particular the download location - had directories that grew in two dimensions. First, when new archives were added. Second, when new versions of existing archives were added (usually corresponding to Genode's release cycle). In the mid-term, this would have resulted in a huge number of directory entries, e.g., in the src/ subdirectory. To avoid this problem, the new version uses the scheme <archive-name>/<version> instead. This way, at the src/ level, each archive has one subdirectory (the number of subdirectories corresponds to the number of archives). Inside the subdirectory, there is one entry per version.
- Controlled rebuild of binary archives
-
When calling the depot/create tool for a binary archive with FORCE=1, the underlying source archives are re-extracted and the binary archive is rebuilt. This is usually done after local changes in the source tree to apply version updates to depot archives as needed. However, the implicit rebuild is superfluous whenever the source-version remains the same. This is particular inconvenient when re-creating pkg archives that refer to a large number of src archives. Here, all binaries referenced by the pkg archive are rebuilt each time. The new REBUILD argument allows the user to skip superfluous rebuilds in such situations. Normally, FORCE=1 implies REBUILD=1. However, by explicitly specifying REBUILD=, existing binary archives whose versions remain unchanged are kept instead of being rebuilt.
Offline validation of XML configurations
The tool/run tool now automatically checks configurations against target-specific XML schemes. Each component may define a configuration scheme-file in its target.mk file as follows:
CONFIG_XSD = my_config.xsd
When the run tool checks the configuration of an instance of Genode's init component, it additionally iterates through all start nodes of this configuration. For each start node, it checks whether the according component provides a configuration-scheme file and, if so, applies it to the configuration inside the start node. This is done recursively. I.e., also the child configurations of a sub-init of a sub-init ... of the top-level init are covered this way.
Whenever the run tool detects an error in one of the checked configurations, it stops and points out the location of the error. By now, there exist configuration schemes for the init, the NIC router, and the trace logger components. Our intention is that every component that interprets its configuration will eventually be accompanied by such a scheme - not only to validate actual configuration input but also to serve as documentation for users of the component.
Automated ABI consistency checks
In version 17.02, we introduced a kernel-agnostic ABI, which ultimately paved the ground for Genode's package management. For the time being, the ABI is not set in stone. It is expected to evolve for some time until it hopefully approaches ABI stability in the mid term. Whenever Genode's API changes, the ABI may be affected. For example, symbol sizes may grow. Until now, side effects on the ABI had to be curated manually. In practice, however, such side effects are too easy to miss. Therefore, the current release adds a mandatory ABI checking step to the build process. A new tool/check_abi tool is invoked whenever a shared object is built. It reports flaws in the ABI definition (such as duplicated symbols) as well as inconsistencies between a shared object and its ABI.