After 20 years, Real-Time Linux (PREEMPT_RT) is finally – finally – in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. […] The real-time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. An RTOS is a specialized operating system designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, “Real-time is the fastest worst-case scenario.” He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. […]
So, why is Real-Time Linux only now completely blessed in the kernel? “We actually would not push something up unless we thought it was ready,” Rostedt explained. “Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in.” In addition, the path to the mainline wasn’t just about technical challenges. Politics and perception also played a role. “In the beginning, we couldn’t even mention real-time,” Rostedt recalled. “Everyone said, ‘Oh, we don’t care about real-time.’” Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.
The final hurdle for full integration was reworking the kernel’s print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it’s called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: “Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that’s great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds.” That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.
This is super interesting. I’ll admit I wasn’t even aware of this effort. Even real-time usage of Windows relies on a parallel kernel.
This sounds like it’ll create a lot of cool opportunities and reduce friction.
But the RT patches have been available for 20 years… not sure why the fact that it is mainlined would suddenly expand its popularity? It might be easier to get started sure, but people doing RT were already going to such troubles anyway.
It’s like saying GPS was available for decades before, why would putting it in everyone’s phone expand its popularity.
For myself, I’m hoping the nerds and hackers that otherwise found it not worth the effort will start creating tools to manage real time better and start building them into the applications they write. That way you don’t need to pay an arm and a leg to RedHawk for the privilege of dynamically isolating CPUs or have to reboot the computer to modify kernel arguments a la RedHat MRG.
I don’t think that comparison is fair because I explicitly said that people who wanted RT are already going out of their way to get things done. The average desktop user (putting GPS in every phone) won’t benefit from it (or RT) and it could likely make their experience even worse.
People who wanted GPS were also already using it by buying TomToms or other GPS devices…
The average desktop user already benefits from the changes the RT folks have slowly been getting into the baseline.
People who want RT are not everyday people though.
How does an average desktop user benefit from any RT changes?
You seem to have the idea that there are “people who want RT” and they’ll overcome any inconvenience to get it, therefore making RT more convenient to use won’t increase use of it.
Clearly nonsense, and I think the GPS analogy is a good one. My mum isn’t “a person who wants GPS” and she would never have bought a GPS device in the 00s, but she uses one now because it’s conveniently already available in her phone.
The most obvious answer is gaming. Hard 60/144Hz deadlines is RT. But there are lots of changes that got into the kernel from the RT group, starting with getting rid of the BKL, which helped everyone.
I said it’ll reduce friction, you said it might be easier. Looks like we’re in complete agreement, right?
Music Makers will love this
I’m curious about this. I’ve started playing with Reaper and getting into music recording and production. I’m very fresh on the scene and haven’t used any DAWs on any other OS, except viewed protools on Mac. I can’t quite get latency free playback, which may just be user error and configs.
Do you know if something like this will have default benefits out of the box, or will we need to somehow configure our apps and services to utilize these changes? I’m completely ignorant but am really intrigued.
I have a relative who I helped set up RTLinux (?) on Kubuntu back in 2018. I think they were playing around with Helm, Calf, Ardour, Hydrogen, Rosegarden et all and JACK of course, but that’s all I remember on the music production side - I was helping more along the lines of PPAs, compiling and configuring and mostly on WE after I came back from somewhere, so 3AM or smth (not good for memory).
Does this mean anything to the average user, or is this a very specific use case?
Probably some use cases for “regular” users. Someone mentioned music production, though that’s probably more professional than hobby.
To my understanding, you mostly need real time performance for specialty cases where timing is absolutely critical. So I guess if you were building custom drones or custom control boards for drones, you could use real time Linux for that now since the timing could be guaranteed.
So what about 3D printing? Currently, input shaping uses an accelerometer to calculate resonances and uses that data to adjust movement and reduce flaws in the printing process. For anyone with knowledge of both fields, would this allow a built-in or add-on accelerometer to be used in real time to compensate for momentum and resonances even further?
Real time operating systems are important for CNC machines and other industrial controls. Been a while since I kept up with Linux CNC and the open source control world, but I think this may have big implications for motion controls including printers.
This is why the big name CNC controls are mostly proprietary OS like FANUC, HAAS, Siemens, Mazak. Lot of them have Mitsubishi down in the guts. Heidenhain has Linux, at least in the top layer, iirc. Citizen had windows in the top layer.
What’s preventing that from working now? If it’s indeterminate latency, then yeah, absolutely. Theoretically this will give you the ability to have a very deterministic loop around the accelerometer data, but 3d printers don’t move all that fast to begin with so having unbounded latency might not matter. The determinism we’re talking about here is on the order of tens of microseconds or less.
3D printers can move very fast. They typically don’t because it causes all kinds of deformations in the print. Mostly the issues are in acceleration, decelaration, cornering, and controlling the heat snd flow of melting filament.
I don’t know whether or not the accelerometer thing can be done in real time, or if there would be any benefit.
Check out the 2-minute Benchy for an example of how fast a 3D printer can get. This is a test print that typically should take about 45 minutes to an hour at very basic settings.
But also note the quality of the end product. It looks pretty awful. If we could print accurately at even remotely similar speeds, it would be fantastic.
“Very fast” is relative. 1200mm/s is very fast for 3D printing, no argument. But it’s 1.2mm/millisecond, and we’re talking about time scales in the microsecond range. I suspect you’re going to run into materials issues far before real time performance becomes a limiting factor in print speed and quality.
Pretty specific use case. A normal OS handleds time slicing and core assignment for processes and uses it’s judgement for that. So at any time your process can be suspended and you don’t know when you get your next time slice.
Same with when you make wait calls. You might say wait 100ms but it may be much longer before your process gets to run again.
In a real time OS if you have real time priority the OS will suspend anything else including it self to give you the time you request. It also won’t suspend you no matter how long you use the core.
So if you need to control a process with extreme precision like a chemical manufacturing process, medical device, or flying a rocket where being 10ms late means failure they are required.
However with great power comes great responsibility. You need to make sure your code calls sleep frequently enough that other tasks have time to run. Including things like file io or the gui.
My guess is not a whole lot to the average user, but it would allow for things to still respond when other things have bogged down resources. I am assuming real world applications would be industry like a machine safery stop should always have a quick turn around, and not be delayed by harddrive writes. But may like how they write special OS code for spacecraft where sending and receiving instructions on board has special states and if response isn’t given in timely manner the system can recognize, so malfunctions are prevented. There was an artivle/podcast somewhere abouy how this all had to work in realtime and not be queued waiting
No, but you might notice programs get more responsive as time goes on, ofc if people use it wrong you might see more freezes and system crashes.
Heck yeah. Now a microkernel version of Linux would be the icing on top!
RedoxOS is trying to do that with Rust
I do understand why. Due to all the heated discussions with Rust in Linux, it does makes a lot of sense to rebuild a kernel in Rust from scratch. However, it will take a long time to get on the same level of Linux (read: 30+ years).
Definitely will take time, though let’s not discount the fact that Linux came about before the internet was the internet. I don’t know if it will take 30 years, but certainly ten years or more doesn’t seem unreasonable.
There’s also free, easy access to a major open source OS and its devs! … Maybe it’s just wishful thinking that they’ll be able to kang a bunch of useful stuff off’ Linux 😅 Useful design structures or something, despite the different overall kernel design.
I don’t even like Rust, I just like new things 😅 Am excited to see where, if anywhere, it goes. Will be glad to see some success getting past C. … Which I don’t dislike; I just like new things 😁 I hope and expect that good comes of Redox regardless of its success, which definitely is a possibility.
after much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year.
So what was the compromise with print_k??
Doesn’t say, but I am curious. They said their workarounds broke other workarounds which caused a lot of implementation delay, but I’m not sure what the actual compromise was to address all that.
Answer probably lies somewhere in the kernel maintainer’s mailing list, I’d imagine. Just not equipped to search for it right at the moment.
Let’s ask ChatGPT! ahhaah just kidding… kidding…
😡
KIDDING! aaaaaaaaaaaaaah
Look what I just did:
"In a real-time kernel, the trouble with using print_k (or similar logging functions) often revolves around potential disruptions to real-time performance. Here are some key issues:
-
Blocking Behavior: print_k may block if the output buffer is full, leading to unpredictable delays in real-time tasks.
-
Interrupt Context: Using logging functions within interrupt handlers can lead to priority inversion, causing lower-priority tasks to block higher-priority ones.
-
Latency: Printing can introduce significant latency, which is detrimental in real-time systems that require deterministic timing.
-
Context Switching: Frequent logging can increase context switching overhead, impacting overall system performance.
-
Overhead: The computational overhead of formatting and outputting strings can interfere with time-sensitive operations.
For these reasons, it’s typically recommended to use alternative methods, such as circular buffers for logging, or to minimize logging in real-time contexts.
"
I got this back from ChatGPT (most likely false info!):
ChatGPT response (you have been warned)
The compromise for integrating PREEMPT_RT into the Linux mainline kernel, including the handling of printk, required several changes and concessions over time. These compromises made it possible to finally integrate real-time (RT) capabilities while maintaining the overall philosophy and structure of the Linux kernel. Key Compromises Made:
-
Soft-Real-Time Focus:
One of the biggest compromises was accepting that Linux would focus on soft real-time capabilities instead of hard real-time guarantees. This was crucial for widespread acceptance in the mainline kernel, as hard real-time guarantees were too difficult to meet without significant changes that would have disrupted the kernel’s general-purpose use cases. PREEMPT_RT was designed to offer deterministic behavior with reduced latencies, but it doesn’t provide the strict guarantees of traditional hard real-time systems like VxWorks or QNX.
-
printk Latency Handling:
Non-blocking printk: One compromise involved updating printk to avoid long blocking during logging operations. It was changed to be more asynchronous, reducing the impact on real-time scheduling. Deferred printing: Another approach was to defer the actual printing of log messages to avoid introducing large latencies in time-critical paths. The goal was to prevent printk from stalling critical tasks, especially those with real-time priority.
-
Voluntary Preemption Points:
Voluntary preemption was introduced, allowing kernel code paths to insert preemption points that allow the scheduler to preempt running tasks, improving latency. However, this does not guarantee immediate preemption, which is another compromise compared to true hard real-time systems. These preemption points had to be carefully placed to balance performance and responsiveness without destabilizing the general-purpose kernel.
-
Threaded Interrupts:
Another significant compromise was the conversion of hard IRQs (interrupts) into threaded interrupts. While this allows real-time tasks to take precedence over hardware interrupts (which would traditionally have a higher priority), it involves some overhead and performance trade-offs. Not all interrupts could be threaded easily, and this change required reworking many drivers to ensure that they were compatible with threaded interrupts.
-
Preemptible Spinlocks:
Spinlocks in the kernel traditionally prevent preemption. To enable real-time preemption, spinlocks were made preemptible, allowing real-time tasks to preempt even code holding spinlocks. This change wasn’t trivial, as it involved significant reworking of kernel locking mechanisms to ensure the system remained stable and avoided deadlocks, without degrading performance in non-real-time scenarios.
-
This is exactly the information I already had before based purely off the name print_k
And is this already where it goes wrong with people trying to use “AI”…
-
after using gentoo for a couple months, i think i know a thing or two about spending Real Time getting the linux kernel
Intriguing. The first and last time I saw anything RT kernel related was nearly a decade ago, when I stumbled on it as an alternate kernel in the old synaptic package manager.
This has got to be a good thing for emulation, forwarding api calls can happen in real time now, that would make it entirely more responsive and might be indistinguishable from running on bare metal.
Cool. I remember having to use Red Hawk Linux years ago. It’s a real time variant of Red Hat. It was such a pain to get the dependencies workin on it.
Anyone know what real time means here? Does it mean that sleeping a thread is more accurate (as in the thread is resumed at the correct time after calling sleep)? Or is there an API that implements some functionality for something that should run in real time?
I’m not an expert, but have used a real time kernel for scientific research, using rtxi. My understanding is that the real time threads allows the computations to occur in a deterministic amount of time. This is necessary if you want to quickly respond to changes in personal membrane voltage with injections of current, and don’t want it to sometimes take longer to calculate how much current to inject.
RT systems are all about responsibility.
When you’re driving a vehicle, the vehicle has a lot of CPUs inside that help the steering.
When these CPUs pause - for even 0.4 seconds - something bad can happen. Somebody can get seriously injured, or even killed. So it is very important that these systems never freeze or pause for any reason. That is what RT is all about.
