Embedded Systems/Linux

can range from consumer electronics like smartphones and smart TVs to industrial machinery, medical devices, automotive systems, communication equipment and more. Linux brings the power, flexibility, and stability to these resource-constrained devices.

Core components of an Linux on embedded systems are, custom and. Source code is compiled by. Common boot loader is – the Universal Boot Loader. Root FS usually is based on – software suite that provides many Unix utilities in a single executable file. Local debugging often is limited, so can be used.

Basic embedded Linux system can be compiled manually from modules above. There are tools for complete system generation from old and simple to huge, which creates customized embedded Linux distributions. PC distributions, ant others  have build for  processors.

, and  are examples of popular s supporting embedded Linux.

Unlike regular desktop or server distributions of Linux these versions of Linux usually have much fewer libraries and application, is smaller and typically boot from  or.

Build frameworks and environments

 * – Android for smartwatches and other wearables.
 * – old simple patch and make based build environment
 * and – a  based build framework and environment
 * – make-like build tool
 * – open wireless router
 * – for ancient 16-bit x86 machines and emulators
 * – for ancient 16-bit x86 machines and emulators

Components

 * The Linux Kernel
 * "" implementations: ,
 * GUIs:, , ,
 * – lightweight

variants

 * General purpose glibc, 7.9 MB
 * , 560 KB
 * , 527 KB
 * , 185 KB
 * – primarily for booting Linux systems
 * – originally developed by Google for the Android embedded system operating system
 * – originally developed by Google for the Android embedded system operating system

Further reading:
 * Comparison of C/POSIX standard library implementations for Linux
 * Choosing System C library

Distributions
Distributions for processors:
 * – port of  for  processors
 * – and  based
 * – port of
 * – proprietary based

Lightweight cloud oriented systems:
 * – uses, and.
 * RHCOS – Red Hat Enterprise Linux CoreOS for
 * https://MicroShift.io/ – lightweight Kubernetes distribution designed for edge and IoT computing environments
 * https://MicroShift.io/ – lightweight Kubernetes distribution designed for edge and IoT computing environments

Lightweight multi-platform distributions:

Common multi-platform distributions:

Features of Linux kernel for embedded systems

 * I/O and buses:
 * GPIO
 * SPI
 * I2C
 * ⚲ UAPI:
 * , https://i2c.wiki.kernel.org
 * – Multi-Function Devices
 * – peripheral USB device implementation
 * – Industrial I/O
 * – Generic Voltage and Current Regulator support
 * – Simple List Of Blocks memory allocator
 * – describes hardware components for the kernel
 * – Industrial I/O
 * – Generic Voltage and Current Regulator support
 * – Simple List Of Blocks memory allocator
 * – describes hardware components for the kernel
 * – Simple List Of Blocks memory allocator
 * – describes hardware components for the kernel
 * – describes hardware components for the kernel

Multimedia

 * Audio
 * ASoC – ALSA System on Chip
 * https://www.alsa-project.org/wiki/ASoC
 * TFT (LCD) –
 * https://www.alsa-project.org/wiki/ASoC
 * TFT (LCD) –

File systems and storage

 * – Unsorted Block Image FS
 * – a compressed read-only FS with low
 * – block device
 * More: ,

MMU less
Linux was originally designed on a processor with a (MMU). Most embedded systems do not have a MMU, as we discussed earlier (Embedded Systems/Memory) and are called s. Microcontrollers without a MMU are cheaper consumes less power.

Benefits of using a processor with a MMU:
 * can isolate running "untrusted" machine code from running "critical" code, so the "untrusted" code is guaranteed (in the absence of hardware failures) not to interfere with the "critical" code
 * makes it easier for the OS to present the illusion of virtual memory
 * can run "normal" Linux (could also run "μClinux", but what's the point?)

– "MicroController Linux" is a version of the Linux kernel that supports Altera NIOS, ARM, Freescale M68K (including DragonBall, ColdFire, PowerQUICC and others), Hitachi H8, MIPS, and Xilinx MicroBlaze processors.

Disabling parameter CONFIG_MMU enables nommu mode.

Real-time
People use a variety of methods to combine tasks with Linux:
 * 1) Run the real-time tasks on a dedicated microcontroller; communicate with a (non-real-time) PC that handles non-real-time tasks. This is pretty much your only choice if you need real-time response times below 1 microsecond.
 * 2) Run the real-time tasks in a "underlying" dedicated real-time operating system; run Linux as a "nested operating system" inside one low-priority task on top of the real-time operating system. Some of these systems claim real-time response times below 500 microseconds.
 * 3) use a Linux kernel designed to emphasize real-time tasks, and run the real-time tasks with a high priority (perhaps even as a kernel thread).

Examples of RT distributions:
 * Red Hat Enterprise Linux for Real Time
 * Real-time Ubuntu
 * Red Hat Enterprise Linux for Real Time
 * Real-time Ubuntu
 * Red Hat Enterprise Linux for Real Time
 * Real-time Ubuntu

Several RT features provide real time capabilities:
 * RT preemption
 * RT scheduling
 * CPU partitioning and isolation
 * Dynticks (Tickless, nohz)

RT preemption
The PREEMPT_RT patch has been partially merged into the mainline Linux kernel, starting from version 5.15. This means that real-time capabilities are now available as part of the mainline Linux kernel, making it easier to deploy and maintain real-time systems using Linux.

Lazy preemption (CONFIG_PREEMPT_LAZY) remains in the external patchset.

However, the real-time capabilities of Linux are still evolving, and there are ongoing efforts to improve real-time performance and reduce latency even further. The Linux Foundation's Real-Time Linux (RTL) collaborative project is focused on improving the real-time capabilities of Linux and advancing the adoption of real-time Linux in various industries, including aerospace, automotive, robotics, and telecommunications.

In summary, real-time Linux is now part of the mainline Linux kernel, and there are ongoing efforts to improve real-time performance and reduce latency even further.

Parameter enables real-time preemption.

RT scheduling policies
Scheduling policies for RT:
 * implemented in
 * implemented in
 * implemented in
 * implemented in

API:
 * – manipulate the real-time attributes of a process
 * – get the SCHED_RR interval for the named process
 * , sched_getscheduler – set and get scheduling policy/parameters
 * , sched_get_priority_max – get static priority range

Testing RT capabilities
The testing process for Real-Time Linux typically involves several key aspects. First and foremost, it is crucial to verify the accuracy and stability of the system's timekeeping mechanisms. Precise time management is fundamental to real-time applications, and any inaccuracies can lead to timing errors and compromise the system's real-time capabilities.

Another essential aspect of testing is evaluating the system's scheduling algorithms. Real-Time Linux employs advanced scheduling policies to prioritize critical tasks and ensure their timely execution. Testing the scheduler involves assessing its ability to allocate resources efficiently, handle task prioritization correctly, and prevent resource contention or priority inversion scenarios.

Furthermore, latency measurement is a critical part of Real-Time Linux testing. Latency refers to the time delay between the occurrence of an event and the system's response to it. In real-time applications, minimizing latency is crucial to achieving timely and predictable behavior. Testing latency involves measuring the time it takes for the system to respond to various stimuli and identifying any sources of delay or unpredictability.

Additionally, stress testing plays a significant role in assessing the system's robustness under heavy workloads. It involves subjecting the Real-Time Linux system to high levels of concurrent activities, intense computational loads, and input/output operations to evaluate its performance, responsiveness, and stability. Stress testing helps identify potential bottlenecks, resource limitations, or issues that might degrade the real-time behavior of the system.


 * RTLA – The realtime Linux analysis tool:
 * – CLI for the kernel's
 * – CLI for the kernel's . Kernel function measures time with function  in loop.
 * – CLI for the with interrupts disabled
 * Implementation: and
 * Linux scheduling latency debug and analysis


 * RT-Tests, source
 * cyclictest
 * some RT-Tests man pages:
 * cyclictest – measures or  delay


 * hackbench – scheduler benchmark/stress test
 * hwlatdetect – CLI for / . Kernel function  measures time delays with function  in loop.
 * oslat – measures delay with in busy loop
 * RT Tracing Tools with eBPF

Further reading about real-time Linux:


 * Introduction to Real-Time Linux: Unleashing Deterministic Computing
 * Power Management and Scheduling in the Linux Kernel (OSPM)
 * the Real-Time Linux wiki
 * CPU partitioning and isolation
 * Realtime@LWN
 * linux-stable-rt.git
 * linux-rt-devel.git
 * Realtime kernel patchset, Arch Linux
 * https://www.kernel.org/pub/linux/kernel/projects/rt/ - RT patches for upstream kernel
 * (HPET)
 * Demystifying the Real-Time Linux Scheduling Latency
 * Real-time kernel tuning in RHEL 9
 * Linux subsystems related to real-time
 * Interrupts
 * Deferred works
 * Non-maskable interrupt handler (NMI)
 * System management interrupt (SMI)
 * latency @ LKML
 * PREEMPT_RT @ LKML
 * latency @ LKML
 * PREEMPT_RT @ LKML

Further readings about Embedded Linux:
 * Embedded Linux Conference (ELC) @ EOSS 2023
 * https://elinux.org
 * ARM support
 * https://lwn.net/Kernel/Index/#Architectures-ARM
 * architecture-specific initialization
 * Learning the vi Editor
 * https://lwn.net/Kernel/Index/#Architectures-ARM
 * architecture-specific initialization
 * Learning the vi Editor
 * Learning the vi Editor