HN Theater @HNTheaterMonth

(2015)

Previously: https://news.ycombinator.com/item?id=9735358

And see [1], my note about this talk being more or less the same as the 2010 Minix talk at FOSDEM[2]. If you saw that one, or any other talk by Tanenbaum of that period, I'll spoil it for you: there's nothing substantially different in this talk.

1. https://news.ycombinator.com/item?id=9736214

2. https://www.youtube.com/watch?v=bx3KuE7UjGA

⬐ AdmiralAsshat

Apologies, I checked the date and the comments before submitting and they looked recent. I wasn't aware that the article itself was a repost. Clearly I should've checked the date on the embedded Youtube video!

⬐ smcl

I hadn't seen it before, and HN wouldn't have let you submit it if the link had been posted too much recently. So thanks for sharing! :-D

Writing OS's is hard and there are no shortcuts. You should take a look at microkernels and the work Andrew Tanenbaum did with Minix 3 [1] and his modular design as this is the sort of direction you're probably looking for with customisable OS's.

[1] https://www.youtube.com/watch?v=bx3KuE7UjGA

Here's Andy Tannenbaum's response to the performance issues of microkernels: https://www.youtube.com/watch?v=bx3KuE7UjGA#t=2832 (should start at about 47:15)

⬐ tormeh

Tl;dr: He thinks reliability trumps performance. I mean, he has a point, but then there's the reliability of the hardware, and any process can be made more reliable if you know the details of the the software instead of just trying to make everything about the system the software runs on reliable. I guess a reliable OS can be a middle ground between bespoke systems with redundant hardware and a single computer running Debian. Well, I guess a reliable OS is always a good thing, but it'll never be enough for everything.

⬐ pjmlp

I am with him. It was putting performance above reliability that has brought the industry to the current state of security exploits that cost millions in tooling, bug fixes and paying damages.

We are still very far from other industries where reliability is taken seriously.

⬐ taeric

This argument cuts both ways. The emphasis on performance and, you know, working is what led to the massive amount of money in the market.

That is, yes there was a cost to this. However, leaving aside the unknowns of whether there would still be millions in tooling necessary for security even with microkernels, you have to provide some sort of reasonable argument that the current market (> billions) would still exist. You can't just take that for granted.

⬐ pjmlp

As I heard once at one software engineering talk, IT is the only field where one can buy a pair of shoes that explode randomly when tying the shoe laces and everyone will find it normal.

⬐ taeric

Which, of course, completely ignores all of the bat shit insane things people used* to sell and do to each other.

The difference isn't that IT has some dangerous crap in it. The difference is that you know some of these dangers. And, you feel that it doesn't belong for some reason.

Consider, you probably have highly charged electrical wiring to your house. Fairly high pressure water. Possibly gas. Literally explosive. Several drawers of sharp instruments in your kitchen. Possibly power tools that are literally spinning blades. Of sizes from less than a centimeter to upwards of a meter.

Think of a computer less as a safe device that only gives good results, and more as an extremely powerful tool that does what you tell it to do. Very fast. And then be thankful that it won't take your fingers off if you slip while pushing a board across it.

* Who am I kidding? We still do a fair bit of it. :)

Also... apropos of what? I'm not sure how this followed from my parent post.

⬐ tormeh

Reliability and security are different things. Industrial controller manufacturers often don't give a shit about security, but their products are much more reliable than x86 running Linux.

⬐ pjmlp

I think otherwise. Reliability implies security.

Summary of the links shared here:

http://blip.tv/clojure/michael-fogus-the-macronomicon-597023...

http://blog.fogus.me/2011/11/15/the-macronomicon-slides/

http://boingboing.net/2011/12/28/linguistics-turing-complete...

http://businessofsoftware.org/2010/06/don-norman-at-business...

http://channel9.msdn.com/Events/GoingNative/GoingNative-2012...

http://channel9.msdn.com/Shows/Going+Deep/Expert-to-Expert-R...

http://en.wikipedia.org/wiki/Leonard_Susskind

http://en.wikipedia.org/wiki/Sketchpad

http://en.wikipedia.org/wiki/The_Mother_of_All_Demos

http://io9.com/watch-a-series-of-seven-brilliant-lectures-by...

http://libarynth.org/selfgol

http://mollyrocket.com/9438

https://github.com/PharkMillups/killer-talks

http://skillsmatter.com/podcast/java-jee/radical-simplicity/...

http://stufftohelpyouout.blogspot.com/2009/07/great-talk-on-...

https://www.destroyallsoftware.com/talks/wat

https://www.youtube.com/watch?v=0JXhJyTo5V8

https://www.youtube.com/watch?v=0SARbwvhupQ

https://www.youtube.com/watch?v=3kEfedtQVOY

https://www.youtube.com/watch?v=bx3KuE7UjGA

https://www.youtube.com/watch?v=EGeN2IC7N0Q

https://www.youtube.com/watch?v=o9pEzgHorH0

https://www.youtube.com/watch?v=oKg1hTOQXoY

https://www.youtube.com/watch?v=RlkCdM_f3p4

https://www.youtube.com/watch?v=TgmA48fILq8

https://www.youtube.com/watch?v=yL_-1d9OSdk

https://www.youtube.com/watch?v=ZTC_RxWN_xo

http://vimeo.com/10260548

http://vimeo.com/36579366

http://vimeo.com/5047563

http://vimeo.com/7088524

http://vimeo.com/9270320

http://vpri.org/html/writings.php

http://www.confreaks.com/videos/1071-cascadiaruby2012-therap...

http://www.confreaks.com/videos/759-rubymidwest2011-keynote-...

http://www.dailymotion.com/video/xf88b5_jean-pierre-serre-wr...

http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hic...

http://www.infoq.com/presentations/click-crash-course-modern...

http://www.infoq.com/presentations/miniKanren

http://www.infoq.com/presentations/Simple-Made-Easy

http://www.infoq.com/presentations/Thinking-Parallel-Program...

http://www.infoq.com/presentations/Value-Identity-State-Rich...

http://www.infoq.com/presentations/We-Really-Dont-Know-How-T...

http://www.mvcconf.com/videos

http://www.slideshare.net/fogus/the-macronomicon-10171952

http://www.slideshare.net/sriprasanna/introduction-to-cluste...

http://www.tele-task.de/archive/lecture/overview/5819/

http://www.tele-task.de/archive/video/flash/14029/

http://www.w3.org/DesignIssues/Principles.html

http://www.youtube.com/watch?v=4LG-RtcSYUQ

http://www.youtube.com/watch?v=4XpnKHJAok8

http://www.youtube.com/watch?v=5WXYw4J4QOU

http://www.youtube.com/watch?v=a1zDuOPkMSw

http://www.youtube.com/watch?v=aAb7hSCtvGw

http://www.youtube.com/watch?v=agw-wlHGi0E

http://www.youtube.com/watch?v=_ahvzDzKdB0

http://www.youtube.com/watch?v=at7viw2KXak

http://www.youtube.com/watch?v=bx3KuE7UjGA

http://www.youtube.com/watch?v=cidchWg74Y4

http://www.youtube.com/watch?v=EjaGktVQdNg

http://www.youtube.com/watch?v=et8xNAc2ic8

http://www.youtube.com/watch?v=hQVTIJBZook

http://www.youtube.com/watch?v=HxaD_trXwRE

http://www.youtube.com/watch?v=j3mhkYbznBk

http://www.youtube.com/watch?v=KTJs-0EInW8

http://www.youtube.com/watch?v=kXEgk1Hdze0

http://www.youtube.com/watch?v=M7kEpw1tn50

http://www.youtube.com/watch?v=mOZqRJzE8xg

http://www.youtube.com/watch?v=neI_Pj558CY

http://www.youtube.com/watch?v=nG66hIhUdEU

http://www.youtube.com/watch?v=NGFhc8R_uO4

http://www.youtube.com/watch?v=Nii1n8PYLrc

http://www.youtube.com/watch?v=NP9AIUT9nos

http://www.youtube.com/watch?v=OB-bdWKwXsU&playnext=...

http://www.youtube.com/watch?v=oCZMoY3q2uM

http://www.youtube.com/watch?v=oKg1hTOQXoY

http://www.youtube.com/watch?v=Own-89vxYF8

http://www.youtube.com/watch?v=PUv66718DII

http://www.youtube.com/watch?v=qlzM3zcd-lk

http://www.youtube.com/watch?v=tx082gDwGcM

http://www.youtube.com/watch?v=v7nfN4bOOQI

http://www.youtube.com/watch?v=Vt8jyPqsmxE

http://www.youtube.com/watch?v=vUf75_MlOnw

http://www.youtube.com/watch?v=yJDv-zdhzMY

http://www.youtube.com/watch?v=yjPBkvYh-ss

http://www.youtube.com/watch?v=YX3iRjKj7C0

http://www.youtube.com/watch?v=ZAf9HK16F-A

http://www.youtube.com/watch?v=ZDR433b0HJY

http://youtu.be/lQAV3bPOYHo

http://yuiblog.com/crockford/

⬐ ricardobeat

And here are them with titles + thumbnails:

http://bl.ocks.org/ricardobeat/raw/5343140/

⬐ waqas-

how awesome are you? thanks

⬐ Expez

Thank you so much for this!

⬐ X4

This is cool :) Btw. the first link was somehow (re)moved. The blip.tv link is now: http://www.youtube.com/watch?v=0JXhJyTo5V8

Andrew Tanenbaum presenting on Minix 3's architecture: http://www.youtube.com/watch?v=bx3KuE7UjGA.

Regardless of one's opinions on microkernels vs. monolithic kernels, it's a very interesting but accessible talk for those interested in lower-level systems and fault-tolerant architectures.

MINIX 3: a Modular, Self-Healing POSIX-compatible Operating System.

By Andrew Tanenbaum =^.~=

https://www.youtube.com/watch?v=bx3KuE7UjGA

Andrew Tanenbaum explains: http://www.youtube.com/watch?v=bx3KuE7UjGA

⬐ thirsteh

The video is from 2010, but it provides an accurate introduction to MINIX 3.x by Andrew Tanenbaum himself. Version 3.2 was just released: http://news.ycombinator.com/item?id=5272980

Original post: http://news.ycombinator.com/item?id=1494386

It's also for security. If you find a security vulnerability in e.g. the sound driver, you're not going to get root access.

Andrew Tanenbaum explains it well: http://www.youtube.com/watch?v=bx3KuE7UjGA

⬐ CitizenKane

I find it fascinating that the actor model seems to be the model for creating reliable software. You can see it in Scala or Erlang at the user level, get it in MINIX3 at the kernel level. I think if you combined the two you could potentially create systems which stay up for years.

⬐ rbanffy

I have met a couple of QNX-based, also microkernel-based, systems that had uptime measured in years.

Lovely OS. I learned C on it.

MINIX reminds me of GNU/HURD. Since they are all Unix-ish from the application's point-of-view, I have hope they someday can more or less compete on equal footing with more traditional monolithic (it would be fair to call OSX "duolithic") OSs like Linux, BSD, Solaris.

Very usable desktops like the Gnome I am using to write this don't care much about what is under the libraries they link against. They would be fine on top of any OS as long as libraries provided the environment they expect. Oddly enough, the first time I ran Gnome was on top of IRIX.

⬐ phaedrus

The self-healing thing in Minix 3 really works.

Some years ago I installed Minix 3 on a very old Thinkpad that had an esoteric video hardware quirk that would crash most OS's. It was "designed for Windows 95" and by God they meant it because you couldn't run anything else on it. Neither Linux nor Windows 98 would run for very long before the hardware would cause a driver crash and a kernel panic/bluescreen (even in text mode). But with Minix 3 the video driver would just transparently restart after each crash, so smoothly that I wouldn't notice it unless I was watching the logs.

⬐ axman6

That's awesome, and I'm sure its the sort of story the developers would love to hear about.

⬐ yan

MINIX is a fantastic kernel to practice kernel hacking on and see how operating systems function without the real world cruft of performance and portability hacks.

⬐ ori_b

Personally, I found the code relatively painful to follow through. I'm not even talking about how the parts communicated - the code was difficult for me to trace through in the small. Even when compared to Linux or the BSDs, let alone Plan 9.

K&R prototypes, overly decorative comments that just echo the name of the function, defining macros like 'PUBLIC' and FUNCTION_PROTOTYPE(name, args) which are effectively no-ops, and a host of other little annoyances that make the code - at least in my opinion - rather unidiomatic and painful to read C.

⬐ subwindow

Not knowing much about OS design, is there any inherent reason why operating systems built on microkernels seem to be so unsuccessful? They always seem really elegant, but are never widely adopted (with the possible exception of Mach -> OS X).

⬐ yason

Since 80's, operating systems are a commodity and in the commodities market any elegant or premium product will end in the marginal.

People want their operating systems to manage disks, processes, cpus, network, and peripheral hardware: unless the operating system totally fails at these basic tasks, nobody will pay any attention to how the kernel was implemented. There are some folks who are interested in performance, and there are some folks who are interested in stability, but there are virtuall nobody who is, as a user, interested in elegance.

If you just buy a car to get from point A to point B, do you care if the engine has a carburetor and a purely mechanical fuel ignition system or if it comes with an engine control unit computer that electronically controls that its high-pressure fuel injection system and its computer-controlled ignition system are in sync, keeping the engine at its optimal parameters at all time and avoiding ignition knocks?

Given the nature of operating systems as for the markets in general, it's really great that people like Linus Torvalds and his fellow gurus keep making their kernel better and better. For most people, it sounds like really gritty, mundane work to do.

⬐ rbanffy

It's because Richard Gabriel was right.

Monolithic kernels are good enough, perform well enough and are available right now with a large enough body of software.

Being Unix-ish is a great thing as it allows you to be creative with the implementation while presenting a familiar API to the applications. Netscape ran on about 30 different platforms (27 of them more or less identical under the hood because they were Unix ports).

I see a bright future for microkernel and other architectures that can provide a unix-like appearence to programs, but only after we get rid of Windows.

⬐ ori_b

the OSX version of Mach has subsumed so much back into the kernel that it can barely be considered a microkernel any more.

The thing is that for most systems, microkernels are a performance liability, (it's possible to make them perform well, but it's very hard to do while keeping memory protection) and with lots of hardware they tend not to be a big safety win - if you can wedge the hardware through bad commands, it doesn't matter if the commands originated in userspace or kernelspace.

⬐ dflock

This happened to the Windows NT microkernel too - they just kept shoving more and more stuff back into kernelspace for performance reasons, starting with Graphics drivers in NT4, and going from there. I think they may have yo-yo'd on Graphics Drivers later for stability reasons, but I've taken my eye off Windows since I stopped using it.

⬐ Avshalom

Unsuccessful on the desktop, qnx for instance is a microkernel and is quite successful. On the desktop there's a lot of inertia, you need apps (which is a chicken and egg thing). As wmf said, you see a lot of poor unix on top of the kernel so there isn't even a compelling reason to use it.

⬐ wmf

I think the main reason is poor marketing. Teams built microkernels, but of course a microkernel is not useful on its own so they had to build a complete OS. But being already tired (or near graduation) from building the kernel itself, they simply ported Unix to run on top of their microkernels, which led to a slower OS with no new features. There are benefits to microkernels, but all these benefits were hidden or wasted by Unix. By this time, patience (and thus funding) for microkernels had run out and researchers had to find something else to work on.

⬐ rwmj

I tend to think that Linus was right and microkernels are in reality a bad design.

Case in point: Minix 1.5 (which I hacked on before Linux came along).

Minix 1.5 has two "daemons" called mm and fs which run the memory management and filesystem respectively. Now consider process creation and loading (fork and exec). Creating and loading a process intimately involves both mm and fs, so in Minix 1.5 the program sends a message to mm [IIRC] which sends a message to fs and both daemons have to coordinate with each other. This makes it a lot more complex than if there was just one daemon (ie. a monolithic kernel).

Another example is that if mm or fs die, your OS dies. You can't restart one or the other because there's so much process state spread across the two daemons. So the claim that microkernels are more resilient because you can restart daemons seems to be nonsense (but I should say that QNX can apparently restart some(?) components transparently).

Nevertheless it's not all roses for monolithic kernels either. There's no process protection and they're usually written in deeply unsafe languages like C. Exokernels might be the answer to this because they have monolithic qualities (fast calls and shared state) but keep virtually everything running in userspace so you can use sane, safe programming techniques.

⬐ 0bfusct3

This shows nothing more than a badly implemented api. POSIX is a bad api for anything modern (distributed).

⬐ vital101

In my undergraduate operating systems class we hacked away at an early version of Minix. The level of detailed code documentation always just blew me away.

⬐ rbanffy

Absolutely fascinating lecture. Very much worth watching.

⬐ switch007

His book, "Modern Operating Systems", is a fascinating read for any geek. He discusses MINIX a fair bit (unsurprisingly!)

⬐ michael_dorfman

Or, just get his "Operating Systems: Design and Implementation", which is a great read, and contains the full MINIX3 source code (printed and on CD).

⬐ rbanffy

Wow... I remember having read that in 88 or 89... It's a bit sad MINIX 1 would still be considered pretty modern.

⬐ strlen

After watching the lecture, just download Minix, setup it up in VMWare/Virtualbox/Bochs and hack something in it. Really, do it. It's amazingly cool OS and fun to play with.

⬐ pedrocr

This 6k vs 6M LoC comparison is pretty dubious. If your disk driver has a bug and overwrites some data you're just as screwed up, be it running in userspace or kernel space. The argument these bugs are less powerful because they are now in userspace "can't do very much" is at best limited. Drivers can screw up hardware just as easily, filesystems can screw up your data just as easily, etc. The memory grant stuff seems interesting though and I see how that can protect you from some forms of memory corruption.

Running all these processes in userspace seems to gain you some capabilities to more easily respawn/reset drivers. Live upgrade seems exciting. I wonder if you are paying for that in added complexity that is hard to debug. The tests they did are simulating basically hardware errors, and it does seems to be quite resilient to those. In practice I fear the software bugs more. The bugs I actually see in Linux tend to be oopses generated by bad code that simply disable the driver. The bad ones screw up the hardware they are controlling and require a reboot. It is unclear in these situations that the driver can be correctly reset without resetting the hardware too.

I wonder how much of this respawn/update functionality you can apply easily to a Linux kernel right now. Most drivers and filesystems can already be built as modules and loaded/unloaded in a live kernel. I wonder how much that can be extended.

⬐ arohner

Exactly. Linux is also useful enough to run the laptop Tannenbaum's presentation is running on while no version of Minix is. (He's running windows, but it could have been Linux)

Minix drivers would still have to exist, and to be usable, they still need access to hardware. There's no presented reason why Minix drivers should have fewer bugs / LoC than linux, except that individual drivers can't take down other "kernel" processes.

That might get you minor improvements in reliability, but it won't fix this issue of bugs existing in hardware drivers in the first place.

IMO, the way forward in reliability is a kernel written in a more correct, more expressive (haskell-ish) Programming Language.

⬐ rbanffy

> There's no presented reason why Minix drivers should have fewer bugs / LoC than linux

On a monolithic kernel, each and every driver is aware (or at least can be) of each and every other driver linked into the kernel. It may create added complexities that should be absent and abstracted in microkernel-based OSs.

Microkernels are not a silver bullet and buggy drivers will be able to put the system - buses, coprocessors, whatever - in weird states that can hang or crash the machine.

I like the idea of parameter validation on every function. That and extensive unit tests built into the kernel. Tests could be invoked on boot with a specific switch and the kernel would perform a self-test (with crash detection) before bringing up your system. This could help find hardware incompatibilities and other weird defects that could affect reliability of the whole box.

I don't think we would need a Haskell-like language. You can do very Haskell-ish things in C. All it takes is lots of discipline (just as much as writing Haskell)

Maybe forcing kernel developers to learn Haskell would help ;-)

⬐ axman6

I've never found that writing Haskell demanded much discipline at all, just follow a few simple rules (like making total functions etc.), and your code is well on the way to doing exactly what you think it should.

⬐ axman6

What I'm hearing is "It helps make things more stable, yes, but doesn't solve everything, so why bother?". The advantages Minix has to offer doesn't solve every problem, least of all the wetware problem that there will always be bugs. But if you're after a high reliability system, say a plane, you don't want a bug in the audio driver making the navigation systems crash. Who cares if the warning signals sound a little weird for a second, if the system is still running correctly, apart from the audio?

The less that is tied into the kernel, the less can crash the system easily. The disk driver could certainly overwrite the kernel on disk, and that could be... pretty bad, but no system that doesn't in some way verify that the code is correct can ever protect against this sort of programmer caused bug.

It should also be noted that Minix isn't exactly aimed as a replacement for linux, and I'm not one hundred percent sure that making such a comparison is that useful.

⬐ 0bfusct3

This 6k vs 6M LoC comparison is pretty dubious. If your disk driver has a bug and overwrites some data you're just as screwed up, be it running in userspace or kernel space. The argument these bugs are less powerful because they are now in userspace "can't do very much" is at best limited

They are less powerful in user-space he states this about 25 minutes into the talk "moving bugs to user-space will do less damage"...roughly.This is true instead of getting full ring0 access to anything I can only do what the driver is allowed to do if I exploit a bug in the driver.

Running all these processes in userspace seems to gain you some capabilities to more easily respawn/reset drivers. Live upgrade seems exciting. I wonder if you are paying for that in added complexity that is hard to debug

Why would there be added complexity? Linux has an api just as well but less defined than simple ipc - even more complex. Hard to debug? you do understand that having parts of the kernel in userspace makes it easier to debug

The bugs I actually see in Linux tend to be oopses generated by bad code that simply disable the driver.

Read LWN theres roughly a root exploit every 2 weeks

An ideal operating system would be a sort of exokernel but proven and a 'relaxed' api that would allow distributed computing. Unverified applications would run under a vm with proven parts being compiled as well as compiling heuristically verified parts and jitting the other needed parts.

⬐ pedrocr

>They are less powerful in user-space he states this about 25 minutes into the talk "moving bugs to user-space will do less damage"...roughly.This is true instead of getting full ring0 access to anything I can only do what the driver is allowed to do if I exploit a bug in the driver.

I did listen to the talk and that justification. That's why I said it was pretty limited. That may be true for security bugs in drivers. For a network driver that may even be very important. In practice what I see is that the actual bugs I care about in Linux drivers are code bugs that disable the device, or in a filesystem cause disk corruption. None of those are solved by a microkernel. Microkernels give you a bunch of provable advantages in areas that monolithic kernels don't seem to do too badly at.

>Why would there be added complexity? Linux has an api just as well but less defined than simple ipc - even more complex.

This is anything but simple IPC. You're sending async messages around and wanting to handle restart of whole pieces and reissuing of commands. It is much more complex and with many more edge cases that the equivalent Linux call stack.

>Hard to debug? you do understand that having parts of the kernel in userspace makes it easier to debug

Because now you're trying to restart a driver for a device that is in an unknown state and then restarting the operation of the filesystem accessing the driver that now has to make sure its operations are idempotent otherwise it will screw up. The number of new edge cases is immense. It could get hairy really fast. That is even touched upon in the presentation with the async messaging and deadlock avoidance. That's why it's harder to debug. Because you're adding a bunch of complex code in error handling paths that get executed once in a blue moon.

>Read LWN theres roughly a root exploit every 2 weeks

I read LWN every week, there are some local root exploits once in a while. The memory protection stuff could be good for that and you could implement it in Linux if you wanted. I specifically stated that part is interesting for this. My point is that the non-security bugs I care about wouldn't be prevented by this technique.