Rationale
Microcontrollers are typically programmed with a single, large, program that contains all the required functionality. It often includes a threading library, file system support, drivers, networking and user code in the same binary.
Hardware memory protection mechanisms are rarely used. Therefore, bugs present in one subsystem may cause subtle errors in unrelated code due to memory corruption bugs. As software complexity grows it becomes harder to find the cause of such errors, and maintain the code base. Since library code is used directly and not through standard interfaces, the challenge of maintaining software modularity grows.
This OS solves the problem by leveraging a standard POISIX API and loading user code at runtime when needed. It then uses hardware memory protection mechanisms that are applied automatically.
Compared to the monotlithic microcontroller software approach, it is easy to allocate separate protected memory blocks for each running program. Access errors are then easily caught forcing the offending program to terminate while the rest remains unaffected. The stability of the system is improved, and many hard to detect bugs are avoided completely.
Software is generally loaded from files at runtime either by the user typing the name of the program into a command line interpreter, or programatically.
The added flexibility makes it possible to unload and reload software as needed. It is then possible to service a running system with ease, changing configuration files, updating executables etc. on the fly even across a network.
History
This section provides a brief history of how the code loading came to be.
Background
Being able to load executables is the feature that separates most embedded operating systems from a proper multiprocess OS. It is a feature that makes the entire system much more flexible, and features can be added or removed by simply starting the correct program. For those reasons research was done on how to implement such a feature for a microcontroller. Since the Linux kernel uses the .elf format and the same GNU compiler, it is just a custom linkerscript and a dump of the code with some special initialization that makes the difference between a regular program and a microcontroller program. So the first task was reading up on how linkerscripts work.
The first experiment was to make a copy of the linkerscript for the kernel, and change a few things. One of the more important things is to define where (at what address) the code will be and where the RAM will be. Instead of defining these as the start of MCU flash and RAM, the addresses were set to point to a special C array. The microcontroller would then load the executable code to that array and try to execute it.
Relocatable Code
This crude approach worked, but had a severe limitation: what if one wants to load more than one program, or the location of the array changes? The answer is code relocation. Thing is, the MCU does not have virtual memory. It means that each process will not see its own address space like on a regular computer, they will all have to share a global address space. When programs are loaded dynamically they would have to be moved: address pointers in the code has to be changed in order to reflect the new location.
Relocation is "enabled" in the compiler and linker, and the result are additional sections in the elf file. These sections are relocation tables that the runtime loader will use to poke around in the code. Each entry in the table describes a place in the code that has to be altered in some way in order to make it consistent with the place it is actually loaded into.
With this in place it is possible to load programs properly. The best part is of course that one may use the standard GNU tools, and code just like any other program. Elf is also the format Linux uses, the difference is mostly just CPU architecture, inclusion of relocation tables and a different interface to the operating system. These differences require some alterations and additions to the standard C library, however programs that does not depend on anything special (like a console filter program) will build correctly on both desktop Linux and the embedded device without any changes.
Kernel Implementation
API Selection
Most desktop software in the Linux world use the fork/exec combination to start processes. The fork syscall duplicates the running process, and the exec syscall replaces a process with a process image found on disk. This combination allows for any kind of bookkeeping to happen in-between the calls and set up various things for the new process. However this approach is impractical in small embedded applications because the lack of paged/virtual memory makes the fork inefficient. The posix standard addresses this by specifying an alternative to fork/exec called posix_spawn.
The posix_spawn call does the entire process duplication, bookkeeping and process image loading in one operation. Although this limits the actions possible during the bookkeeping phase to that of the specification, it is now possible to create processes on systems without virtual memory in a much more efficient manner by using this API instead. On desktop Linux, the API is implemented as a library call that uses fork/exec internally. Initially this may not make much sense, however we may now develop and test programs on a regular computer before deploying it on a machine that uses the API natively.
The embedded OS is one such system. In userspace, the posix_spawn function simply hands everything over to the kernel using a syscall. The kernel then does everything needed as quickly as possible.
Implementation
The implementation of the posix_spawn call is in the process.c code file. Additional code used for reading a process image and do relocation is in elfloader.c.
It is worth noting that the call is processed by a special worker thread instead of directly in the syscall handler. The syscall only schedules work for the worker thread, blocks the calling thread and then returns so that other processes may run. This is because of the time complexity of loading a process. Because syscalls are a type of interrupt it is not possible to "multitask" until the currently executing syscall completes. This "stall" is a problem for some real-time applications such as music playback, causing an audible pop. Using a worker for the bulk of the operation eliminates this problem by enabling context switching while the new process is loaded.