HN Theater @HNTheaterMonth

⬐ IanSanders

Is it time to redesign the entire technology stack?

Edit: Feels like half measures won't move us very far (and taking a step back and refactoring everything, taking into account lessons we learned, while awesome, would likely not be financially viable)

⬐ mamcx

Totally, and probably to be like the older but better stacks :)

ie: Amiga, BeOS, Plan9, HyperCard, FoxPro, ...

ie2: Is not as much a start from zero. Good ideas are not in short supply, is that the bad idea go a full speed and by pure mass it become bigger and bigger. Eventually, a clean start is 100% the cheaper option.

We are due that point!

Sadly, the BEST starting point for that was the introduction of the iPhone (to make stuff like this possible helps a ton to be along a game-change)

⬐ KronisLV

For better or worse, we'll eventually reach a stage a few centuries from now when the idea of rewriting all of the software that we're using won't be a matter of not having enough finances or manpower, but rather that the capacity of the human brain won't be able to deal with all of the layers upon layers of abstractions that will have accumulated.

Of course, perhaps it's therefore useful to improve the things that we can for now and push for more and more open software in all the levels of the stack, as opposed to introducing more and more abstractions and shiny new technologies before they're truly ready. Is Rust ready? I have no idea. Of course, people will always prefer having working software now rather than the ideal software for their grandchildren, as evidenced by the mentions of PL/I and the evolution of Linux.

⬐ IanSanders

> capacity of the human brain won't be able to deal with all of the layers upon layers of abstractions that will have accumulated

That's already the case though.

Maybe we need to take a step back and design an elegantly simple, unambiguous, auditable/analyzable infrastructure, one that's more robust against becoming too complicated in the future. "Structural minimalism", to complement minimalism in user interfaces becoming more and more popular in the recent years.

⬐ kaba0

There is no such thing. Useful computations are by necessity complex (in both a human-centric view, as well as from a computability view).

There are bad abstractions, of course. But this is just a hay dream that we can design a system that would be less complex than the essential complexity of the problem at hand - and dealing with all the hardware, networking, etc is just simply hard.

⬐ KronisLV

> Maybe we need to take a step back and design an elegantly simple, unambiguous, auditable/analyzable infrastructure, one that's more robust against becoming too complicated in the future. "Structural minimalism", to complement minimalism in user interfaces becoming more and more popular in the recent years.

This sounds good, but in practice there are two types of complexity:

  - domain complexity, which pertains to the domain that we're working in and essentially is forced upon us by it
  - accidental complexity, which is created by all of the sub-optimal decisions and implementation details

You can and should minimize the latter, absolutely, but a lot of the former isn't viable - making systems simpler in some ways (memory allocation and access, for example) could make them perform noticeably worse. No one (amongst the end users) wants to have their hardware become slower, especially when Wirth's Law is still in full effect ( https://en.wikipedia.org/wiki/Wirth%27s_law ).

It would be nice if we could, but even booting operating systems nowadays gets more and more abstractions and complexities added, just look at what Microsoft are doing with Windows 11, essentially creating e-waste for controversial reasons. However, i think that people can at the very least still be competent within small subsets of the larger system, with clearly defined boundaries amongst them, as well as interfaces.

⬐ worrycue

> capacity of the human brain won't be able to deal with all of the layers upon layers of abstractions that will have accumulated

Isn't the point of abstraction so we don't have to deal with the complexity?

⬐ naasking

I doubt that software will look anything remotely like what we have now, so I don't think I can agree about all of the layers of abstraction. If I were to hazard a guess, I'd say that software will be described by high-level specifications and compiled to running code subject to a separable executable specification (defining things like resource limits, allocation behaviours, latency requirements, etc).

Superoptimizers and machine learning will be used to ensure the final code conforms to all required specifications, so people generally won't be dealing with lots of abstraction layers in the way they do now.

⬐ KronisLV

> Superoptimizers and machine learning will be used to ensure the final code conforms to all required specifications, so people generally won't be dealing with lots of abstraction layers in the way they do now.

Have you seen the current state of model driven architecture and the state of trying to do metaprogramming of any sort?

The latter is very stack specific and is really hard to get right, whereas the former has never made its way out of academia. Admittedly, even the people who were forced to use something like the Eclipse Modeling Project ( http://www.eclipse.org/modeling/emf/ ) won't deny that there is definitely potential there, but to a large degree it has failed to materialize.

If there was as much hype around code generation as there was around the new releases of React, then maybe things would be different, but at large i personally doubt that the industry wants to take a step back and spend a decade on the building blocks, rather than getting things done in their day to day lives. Perhaps i'm really skeptical.

⬐ naasking

> Have you seen the current state of model driven architecture and the state of trying to do metaprogramming of any sort?

Sure, but you described circumstances "centuries from now". Your objection about not taking a step back for a decade or two due to immaturity evaporates on that timeline.

⬐ guerrilla

Anyone who watched the hour long video want to summarize? I dropped out after five minutes of pseudophilosophical rambling.

⬐ KronisLV

I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other).

That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": https://youtu.be/30jNsCVLpAE

So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.

⬐ KronisLV

Okay, so here's a slightly delayed summary (had to fix some prod issues):

  - a discussion that the parent commenter took an issue with: "What's software? What's hardware? It's hard to answer that."
  -   essentially, an OS is a program that abstracts away hardware
  -   kernel: a piece of the OS that runs with the highest level of privileges, but only a part of the OS
  -   the OS as a whole includes libraries, daemons etc.
  - expansion on the history of OSes and how we got to where we are now
  -   developing OSes isn't always lucrative, you don't hear about the innovative companies that didn't survive
  -   mentions of https://en.wikipedia.org/wiki/Second-system_effect
  -   a brief story about how trying to outsource the PL/I compiler wasn't a good idea
  -   the Unix approach was way more organic in comparison to PL/I, less of a waterfall
  - a little bit about programming languages
  -   a little bit about the history of C and how it wasn't created the exact time as Unix
  -   some words about languages that now seem esoteric, like https://en.wikipedia.org/wiki/Language_H
  -   thoughts on the imporance of macros and the C preprocessor
  - more about OSes in the 1990s
  -   languages like C++ and Java got more popular
  -   many of the OSes of the time suffered from the aforementioned second system effort, were overcomplicated
  -   oftentimes overcomplication also lead to great resource usage with little tangible benefit
  -   with the arrival of Linux, the C based OSes became more entrenched
  -   at the same time, the actual languages that focused on the ease of development (Java, Python, Ruby) also gained popularity, though in a different context
  - software systems in 2010s
  -   without a doubt, it's nice to be able to use higher level abstractions
  -   Node.js got surprisingly popular due to a high-performance runtime with the aforementioned benefits
  -   Go was also developed, though its garbage collector is mentioned as a problem here, because it makes C interop harder
  -   a bit of elaboration about GC and the problems with it, how easy it is to have a reference into a huge graph
  -   essentially, it has its use cases, but at the same time there are problems it's just not suited for (a certain class of software)
  - how Bryan got started with Rust and a bit about it
  -   initially he didn't want to go back to C++, because of a variety of past issues
  -   he got increasingly interested in Rust and its potential benefits
  -   curiously, there is a category of people who are curious about Rust, but haven't actually written any
  -   it's nice to have a language that's built around safety, parallelism and speed
  - more about Rust
  -   its ownership system allows for the power of garbage collection, but the performance of manual memory management
  -   being able to determine when a memory object is no longer in use and being able to do so statically is like a super power
  -   the compiler itself just feels really *friendly* by pointing you to directly where the problems are
  -   composing software becomes easier all of the sudden, when compared to C, since it's hard to get it right there
  -   going back to C or porting C software can actually be somewhat difficult because of unclear ownership
  -   some of the Rust performance gains are actually from good implementation of language internals, like them using b-trees
  -   algebraic types are also nice to have and the FFI in Rust is really well thought out
  -   there's also the "unsafe" keyword which allows loosening the security guarantees when necessary
  - about OS development and Rust
  -   no-one cares about how easy OS components were to develop or how long they took to develop, everyone just wants things to work
  -   a bit of information about having to deal with failed memory allocations, design discussions, language features etc.
  -   lots of OS projects out there in Rust
  -   however, writing your own OS essentially forsakes Linux binary compatibility, so a lot of software won't run anymore
  -   you have to consider what is the actual advantage of rewriting existing software in Rust, safety alone might not be enough
  -   a callback to the fact that an OS is more than just the kernel! you could rewrite systemd in Rust and other pieces of software, however not all software is a good candidate for being rewritten
  -   firmware in user space (e.g. OpenBMC) could probably benefit greatly from Rust as well, in addition to just having open software in the first place

tl;dr - Rust is promising, yet isn't a silver bullet. That said, there are certainly domains and perhaps particular OS components which could benefit from the safety that Rust provides, especially because its performance is also good!

⬐ guerrilla

Wow, impressive notes. Thank you.

That reminds me of the GNU coreutils being rewritten in Rust: https://lib.rs/crates/coreutils

⬐ puzzledobserver

Haven't watched the video, but I recently watched Timothy Roscoe's OSDI keynote on operating systems and hardware [0], where he argues for the operating system to include all of the kernel, device drivers and blobs that manage the SOC. He points out that the system emerging from the interaction of these components is complex and essentially unarchitected.

After watching his talk, I wanted to figure out how a computer, say a Raspberry PI, _really_ works. And build an operating system for it. Maybe in Rust?

[0] https://www.usenix.org/conference/osdi21/presentation/fri-ke...

⬐ dijit

TL;DW: No, it's not that simple, replacing working code is probably not a good idea, hybrid approaches are best.

⬐ IshKebab

> replacing working code is probably not a good idea

Depends what you mean by "working".

⬐ dijit

I think that's a pretty moronic take when we're talking about operating systems.

They obviously work. I'm using one right now and the site we're communicating on is using one also.

Or do you have another definition which encapsulate the basic notion of "work" and adds to it?

⬐ lnsp

I highly recommend to take the time. First, I find Bryan's presentation style very engaging and fun to watch. The short recap on language history is also quite fun. I also would not consider myself to be part of any Rust hype, but the language has sparked my interest once again.

My TLDR: Rust is a great systems language, however in-kernel C code in itself is very safe, its the ecosystem (drivers, firmware etc) around it that could benefit greatly from being rewritten in a safer language.

⬐ kaba0

> in-kernel C code in itself is very safe

Any citation on that? Obviously it went through numerous people hours where many bugs were reported/fixed, but afaik it still contains plenty of race conditions, and whole other areas of bugs that are just being discovered through better static analysis programs, and any future code can easily introduce new problems.

⬐ lnsp

It's around 0:56:12, AFAIK kernel dev has relatively strict guidelines regarding memory management. Cantrill also says that he considers the borrow checker less important if you only write code for a system that does not interact with other libraries, however as soon as you interact with others, things get messy on who owns what and when stuff should be free'd. Since the kernel is a sense a sealed system (ignoring kernel modules), the memory argument probably isn't that important anyway.

I totally agree regarding race conditions though, however I don't think Rust does anything to solve this.

⬐ throw0101a

> pseudophilosophical rambling.

But that's the best part of any Cantrill presentation. :)

⬐ selectnull

No.

"Any headline that ends in a question mark can be answered by the word no."

-- https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

⬐ jfk13

I'm waiting for a post with the headline: "Is Betteridge's law of headlines always reliable?"

⬐ mijoharas

He addresses this in the second slide. :)

⬐ bitofhope

The conclusion slide appears at about 1:02:00 so you can skip there if you just want a four-sentence answer to the question posed in the title.

The first five minutes or so lead up to a rough definition of what he means by "Operating System" for the purposes of this talk, essentially systems software that abstracts some physical or virtual model of hardware and runs programs that use those abstractions. However, in addition to the privileged mode OS kernel he includes things like system libraries and services in the scope of the subject.

This is followed by a historical overview of various operating systems and programming languages — particularly systems languages — and how they shaped each other.

Finally, Rust is compared and contrasted with historical and incumbent languages and the systems developed in those languages.

Despite his enthusiasm about Rust, Bryan finds that rewriting extant OS kernels in Rust should not be the top priority. The Operating System as defined in the introduction includes firmware and user mode services that can benefit from Rust's memory safety even if the system core remains in C or C++. Since Rust interoperates well with other systems programming languages, there is no need for an all-or-nothing approach.

I don't think my summary does justice to the talk. Bryan's talks very often look at things from a historical perspective, which I and others quite enjoy. The rambly tangents are actually my favourite parts of a Cantrill presentation and the conclusions may feel rather mundane if taken on their own. If you don't enjoy the first minutes, you may need to adjust your expectations or simply decide this style is not for you.

⬐ Verdex

I mean ... sure? However, I won't be funding it. At least not upfront. [That is, I might buy it after you finish and show it to be good.]

Rust is probably better than C or C++ for writing an OS ... however it has some behaviors that don't make it ideal. Rewriting the OS in Rust will perhaps simply have to start over again when some frustrated OS devs go off and make an even better language for writing the OS.

Heavily augment the OS in Rust is a much more believable effort. The main architecture can be in C and then all the utilities, misc functionality, etc can be in Rust. This is also kind of what Rust was built to do (heavily augment a web browser already written in C++).

Finally, my money is on the far future having most OSes built with multiple languages. Each one specializing in what is required for the given domain. It sounds like Windows is already taking this route (iirc Windows 8 and 10 have some drivers (probably usb) written in the P language).

OSes are important enough for our society that I think it makes sense for us to put some extra effort into their construction. Several custom languages and 50ish years of calendar time to get an output feels like it's going to be worth the cost.

⬐ steveklabnik

You should watch the talk.

⬐ scrubs

Assume Rust is a better system language than C, which Linux is written in. In order to get something 10x better ... you're gonna need better algos, or come with something that solves more problems in less code. Otherwise I don't see the point. A better, stronger claim is: let's rewrite sub-system 'X' in Rust which implements simplifications 'A,B,C,...' Unless there's an order of 10 improvement ... the cost will not be overcome by benefits.

⬐ matt_s

Just stop at "rewrite" - when is a full rewrite of anything really necessary? Whenever I see this, I immediately jump to "its because they want to".

⬐ ink404

isn't it possible for there to be overlap between "because they want to" and "because it's necessary"?

⬐ ashtonkem

Usually. But then again Linux was a rewrite too. Sometimes unnecessary things become critical, and the distinction is only clear in hindsight.

⬐ matt_s

From what I recall, Linux was also Linus doing something because he wanted to, not necessarily because there was a business driver. I'm not saying don't, its just good to understand the motivation.

The concept of OSes use file cabinets of papers as part of the abstraction layer. Breaking from that mold might prove fruitful.

⬐ colejohnson66

> From what I recall, Linux was also Linus doing something because he wanted to, not necessarily because there was a business driver. I'm not saying don't, its just good to understand the motivation.

People rewrite things in Rust, not because of business reasons, but because they want to. What’s different here?

⬐ Shadonototra

Rust is nice, it tries to solve certain kinds of memory bugs

But it introduce even bigger issues; slow build times

Being able to iterate SUPER QUICK is very important, you want to be able to test and see results as fast as possible, so you get to fix bugs or implement feature with ease

Imagine you get to work on super import product that needs super low latency and close to 0 downtime

You notice a bug, you have to deploy a fix ASAP

If your language makes you test/deploy the issue in hours, then it's very bad

⬐ mlindner

Most development bugs in other languages are issues of type mismatch (in dynamic languages) or memory issues, both of which aren't a problem in Rust. You don't need to iterate as fast.

⬐ Shadonototra

> You don't need to

I do need to

⬐ nso95

Rust isn't a silver bullet

⬐ thesuperbigfrog

No silver bullet exists.

"There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement ... in productivity, in reliability, in simplicity." -- Fredrick Brooks

Source: http://worrydream.com/refs/Brooks-NoSilverBullet.pdf

Nevertheless, Rust has shown great promise and implemented great ideas for high performance and safer low-level / system programming.

We need to keep on improving and using Rust and its ideas to improve the high performance, low-level / system programming space.

⬐ thewakalix

Well, of course. Those are two different metals.

⬐ Annatar

Rust is an oxidation of a metal with less than eight electrons in the outmost energy orbit, not a metal.

⬐ hoseja

Perhaps we need a new lang, "Tarnish".

⬐ jamincan

Projects to integrate Rust into an existing code base are often referred to as oxidation going back to the beginning with Mozilla and Firefox.

⬐ politician

I’d be onboard with a Rust rewrite of Windows. Go for it! It’s the least memory-safe OS with most problems and the widest desktop deployment. Do the Rust folks do rewrites of closed source?

⬐ daniel-thompson

The business side puts a very high premium on backward compatibility. Until that changes, they need to keep around the cruft / shims / architectural bridges / glue that make that possible. Working with the raw Windows API is already painful enough^; I can't even imagine the effort it would take to reimplement it in Rust.

^ random example: https://caseymuratori.com/blog_0025

⬐ masklinn

That’s probably in the works (on an exploratory basis at least) behind closed doors. Microsoft’s infrastructure team has noted they were switching away from C++, and the Security Response Center team considers it the best chance to move forward.

⬐ sys_64738

No.

⬐ bcantrill

This talk proved prophetic for me; since giving it three years ago, most of my technical work has been in the development of a de novo operating system in Rust -- not for its own sake, but rather because it is the right approach for the problem at hand. I talked about this a bit in a recent talk[0], but expect much more here soon, as we will be open sourcing this system in a few weeks.

Beyond the new system (called, appropriately enough, Hubris), we have been using Rust at more or less every layer of the stack: microcontroller firmware, early boot software, operating system kernel, hypervisor, and (of course) user-level. Again, this is not by fiat or to do it for its own sake; we are using Rust emphatically because it is the right tool for the job.

More generally: Rust has proved to be even more important that I thought it was when I gave this talk. My blog post from a year ago goes into some of this updated thinking[1], but honestly, my resolve on this continues to strengthen based on my own experience: Rust is -- truly -- a revolution for systems software.

[0] https://www.youtube.com/watch?v=cuvp-e4ztC0

[1] http://dtrace.org/blogs/bmc/2020/10/11/rust-after-the-honeym...

⬐ infogulch

I liked your talk, and the talk you referenced by Timothy Roscoe [0]. My understanding of your talks is that the issue we seem to be running into with system architecture design is that OS and userspace developers are clinging desperately to a dead model of the system as a homogeneous array of cores attached to large bank of unified memory. This falsehood is so deep that systems basically lie to us about 95% of their internal structure just so that we can continue playing out our little fantasy in obliviousness.

The biggest component of that lie is the unified memory assumption. To be fair to OS & app developers, writing for NUMA is hard, there are enough invariants that must be continuously upheld that it's impossible to just expect authors to keep everything in their in their head at all times. And using a language like C, described as "portable assembler", does not help at all.

Enter Rust, where the novel ownership system and strong type system allows encapsulating special knowledge of the system and packaging it up into contained bundle that can't be misused. Now you can compose multiple of these un-misuse-able (it's a word now) lego bricks reliably because the compiler enforces these invariants, freeing the author from the burden of reflecting on their design from N! perspectives every time they add a line. (Well, they still reflect on it, but they are privileged to reflect on it right after they make a mistake when the compiler complains instead of in an hours-long debugging session using a debugger or, worse, a specialized hardware debugging device.)

---

Your talk focuses on no_std, which is really a foundational necessity of such a new OS (the term "Operating System" feels too small now, maybe "Operating Environment"/OE? idk). I think the next important component is a path out of UMA-land, which I don't think is fully solved in Rust at the moment (that's not a knock, it's not solved anywhere else either). There's an ongoing Internals thread that started as a discussion about usize vs size_t across different architectures and now has dug down to questions such as "what even is a pointer?", "how are pointers represented as data?", and "how should you convert a bucket of bytes into a usable pointer?" [1] -- these are exactly the type of questions that Timothy's talk reveals as important (and has hitherto remained unanswered) and that you hinted at.

During the discussion, Ralf Jung presented an interface that would enable constructing a pointer from an integer by also separately identifying its provenance; I feel like this is a good direction.

    /// Returns a pointer pointing to `addr`, with the provenance
    /// taken from `provenance`.
    fn ptr_from_int<T>(addr: usize, provenance: *const T) -> *const T

---

What do you think of my summary? What do you think of the ongoing discussion on this Internals thread about the right way to construe Rust's memory model? What do you think of think of this idea presented by Ralf?

[0]: https://www.youtube.com/watch?v=36myc8wQhLo

[1]: https://internals.rust-lang.org/t/pre-rfc-usize-is-not-size-...

[2]: https://internals.rust-lang.org/t/pre-rfc-usize-is-not-size-...

⬐ infogulch

I should start a blog instead of using HN comments as an outlet.

I'm not a person you asked but in my opinion answer is very simple: YES

Example: https://www.youtube.com/watch?v=HgtRAbE1nBM&t=43m15s

> The reason that C programs often don’t perform as well as an equivalent rust program[1] is that it’s so incredibly hard to do anything at all in C (especially something reliable) that one can usually only do the simplest thing possible and this typically means simple data structures, simple algorithms and arrays

Brian Cantrill talks[1] about exactly this: in his C version he was using AVL trees because they are easy to implement, while his Rust version was using B-trees just because he could. And as a result, his naive first attempt in Rust outperformed his long-optimized C code.

[1]: https://www.youtube.com/watch?v=HgtRAbE1nBM&t=43m15s

⬐ kenhwang

I've noticed this as well. With C, I wouldn't dare do anything other than basic array-ops.

With Rust, I'd be far more inclined to just use a create with a well optimized implementation of an obscure structure/algorithm, or attempt it myself knowing the compile will safeguard me from foot-gunning too badly.

It applies to other languages other than C as well like Python, Ruby, JS, Scala, Java. Rust might be a difficult language in general, but it makes it so much easier to build difficult things.

⬐ jstimpfle

So he's comparing the AVL tree he wrote himself to a B-tree optimized to death by someone else?

Out of interest though, does he mention somewhere which actual implementation he used?

⬐ aldonius

Yes, and I suppose that's some of the point: a performant library ecosystem is one hell of a feature.

⬐ jstimpfle

It sure must be a nice feature - for fast results. Counterpoint, though. I barely used Rust, but when I wanted to play with the Xi editor I got to see the bad places that buying into such an ecosystem can get you to - places formerly pioneered by NPM. I had to download 100s of packages, (IIRC?) there were some build problems, all for something where I don't really see the value of noteworthy dependencies.

If you really need a Btree (like, if you want to make a fair benchmark for a presentation) then you'll absolutely find a reasonable implementation in C. After all, why would you need dependency management for something that should have 0 dependencies?

As an example of library data structures implemented in C - if you want, check out my Red-black tree (not a Btree): https://github.com/jstimpfle/rb3ptr/blob/master/rb3ptr.h . It's really easy to integrate into your project. I think the API (found in rb3ptr.h) is totally usable, and it might also be faster than what you can get with safe Rust - unless you can easily use intrinsically linked data structures in safe Rust.

Wow, tough crowd! I don't know that I -- or anyone that knows me -- would accuse me of "ever-evolving advocacy", but yes, I did see promise in node.js in 2010[1]. And yes, we at Joyent built systems on node.js -- but we also added unprecedented debuggability and observability along the way.[2]. While my views may not be "ever" evolving, they're also not incapable of it: by 2014, it was clear that node.js was diverging from my personal hopes for it -- that it was not going to become suitable for infrastructure software. While this was distressing at the time, I came to view this as a divergence in values -- a realization that I described at length in a talk in 2017.[3]

To the matter at hand: yes, I have fallen in love with Rust.[4] Yes, I think it is a good fit for system software.[5] And yes, this has served as a bonding force at Oxide -- but as much (if not more!) for the values that it reflects than for it itself.[6]

[1] http://dtrace.org/blogs/bmc/2010/08/11/the-node-js-demograph...

[2] http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-mem...

[3] https://vimeo.com/230142234

[4] http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...

[5] https://www.youtube.com/watch?v=HgtRAbE1nBM

[6] https://www.youtube.com/watch?v=2ajos-0OWts

I only linked that because most people don't know what B is, and I suppose the difference between "innovation meant to fill the space" and "extention" wasn't so important to me.

My source for the claim itself is from 14m30s of Brian Cantrill's talk "Is It Time to Rewrite the Opersting System in Rust" https://youtu.be/HgtRAbE1nBM

> When I see syntax like: var array=something[4...$] I don't see security there, all I see is someone pushing the problem further under the carpet, into the runtime.

The runtime (whichever one that is) is safe because its behavior is defined by design for all inputs. Yes there could be a bug, but the design is that all inputs should be defined, and doing something for any given input isn't rocket science. Out of bounds array indexes either panic/throw or return empty lists. That can lead to crashes and logic errors, sure, or maybe DOS attacks, but it cannot lead to undefined behavior or remote code execution. That's a world of difference.

C and C++ by design expose heaps of undefined behavior to the caller. The tools for detecting UB are imperfect and don't work if the relevant inputs never show up in test cases. Even when veteran C coders develop the techniques and habits that let them write correct C, mistakes tend to show up at the integration points between libraries written by different people with different techniques and habits (https://youtu.be/HgtRAbE1nBM?t=2344).

My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s

I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM

⬐ Chyzwar

I disagree with the suggested approach of gradual introductions of rust, this would require kernel developers to master both languages and introduce mismatch anyway.

I think before we jump into writing OS in rust we should first understand what wrong with current dominant Linux. I think we could make a major revision on how computers are built and how should be OS architected to last for another few decades.

⬐ blub

Well, I guess C's wrong with it :)

But as nickpsecurity likes to point out, C projects can make use of good, if labour-intensive tools for ensuring safety, so that seems like the logical choice instead of rewriting to languages which haven't proven themselves at the target time and complexity scale.

⬐ tatersolid

> C projects can make use of good, if labour-intensive tools for ensuring safety

I think this has been repeatedly proven false over the last several decades. There really is no way to make C programs safe in practice. It’s time to move on.

⬐ adamnemecek

> I think before we jump into writing OS in rust we should first understand what wrong with current dominant Linux.

Lack of an async API. Bad IPC primitives (how many are there? They are all terrible). The whole "everything is a file" idea. Real time audio. Bash. File systems. The process vs thread distinction. Package management. Permissions. Security model. Scheduler. General cruft. X Server. Battery life. OOM killer.

I would much rather hack on a Rust kernel than C.

⬐ iknowstuff

How does macOS's kernel compare to Linux?

⬐ AnIdiotOnTheNet

There is really just so much that could be done better now that we have decades of hindsight. Sadly, I doubt it'll happen because supporting even a reasonable amount of existing hardware has become an exercise in futility. Theoretically one could start with a well defined subset of hardware, but that would make growth extremely difficult unless that hardware was both inexpensive and common enough for anyone to acquire it easily while also being good enough to cover a lot of use cases (and your OS amazing enough that people would willingly sacrifice to use it). A Raspberry Pi fits this requirement pretty well, but you still don't see useful new OSs on it do you?

⬐ adamnemecek

Raspberry Pi is a bad target because you are unlikely to make money with your OS and fundamentally you need that to turn the OS into anything serious.

The advantage of Rust is that you can split things into modular code bases. Redox is split very nicely.

I think that one could start a hosting company and write a hypervisor in Rust.

⬐ AnIdiotOnTheNet

I guess that's why it hasn't been done. Like too many things people are much more concerned with making money than making things better, so we just keep building on the same pile of garbage we've had since the 70s.

I'd hardly call Redox serious, it's a "research OS" with no explicit goals to actually be useful.

⬐ adamnemecek

Correct however Redox paves a way. The project is very modular, I can imagine someone coming along and using it as a starting platform and reusing parts of it.

⬐ AnIdiotOnTheNet

I can imagine a lot of things that will never happen.

⬐ adamnemecek

Break down your reasoning to me. Don't look at what it doesn't have, look at what it has.

⬐ AnIdiotOnTheNet

My reasoning has nothing to do with what it has, it is about what it is: unambitious. It is unwilling to label itself as something that could ever potentially be a useful replacement for anything, probably because that means it can't be judged for failing to achieve it. Its developers are free to engage in academic exercises which amuse them without regard for practical applicability.

And that's fine really, but it is representative of the larger problem: no one capable of pulling it off is willing to be so ambitious. No one is willing to commit to the hard work of making a usable and practical platform, which will undoubtedly involve distasteful compromises and uninteresting work sanding down the rough edges.

I'd be glad to be proven wrong on this, but I just don't see it happening.

⬐ adamnemecek

I'm not talking about redox in particular but like, you have for example a bootloader that someone can reuse.

I'm saying in the next 10 years, there will be a serious OS written in Rust.

⬐ Redoubts

By Bryan Cantrill, which makes it totally worth it.

⬐ bullen

He's the programmer equivalent of a stand-up comedian!

⬐ karmakaze

Start at 29:31 to skip past 'what is an os?' history lesson.

> Go, JavaScript and others are garbage collected, making interacting with C either impossible or excruciatingly slow.

I don't see why interacting with C is a requirement. Android (SDK) could be considered an OS that doesn't have this either.

⬐ ldng

Humm, I don't think you can equal an SDK to an OS. Before getting this high level, you really have to start with a kernel to handle/manage device. And this as a lot of constraint that yous simply can not fulfill with a garbage collected language.

edit: I wasn't at the point you mentioned yet. Well, IMHO, it is not just the GC, it is not having access to native pointer prevent you to implement algorithm that are keys in term of performance for a kernel. IIUC more savvy dev than me it might not even be possible to implement a kernel in Rust without "unsafe" (which give you access to those raw pointers)

Happy to be proven wrong though.

⬐ karmakaze

OK that was a bad example. I just meant to say that C compatibility is an over-specification. We just need machine level interfacing like maybe locking to hardware threads, fixed address referencing, etc.

Edit: I don't know much about this but it might be a better example "GRiSP 2 – Erlang and Elixir on bare metal." I'm assuming this is possible and includes the BEAM VM with GC.

https://news.ycombinator.com/item?id=19358133

⬐ ldng

Oh I see, you mean "unvirtualizing" the VM and run on the hardware that comply to the VM spec.

I was under the impression that generally VM are idealized hardware that is friendlier for software development but they would make terrible actual hardware. I suppose it also depend on the language/VM.

But I'm totally out of my comfort zone. Let wait and see if someone more knowledgeable chimes in :-)

⬐ brightball

The BEAM GC is a little bit special because there’s not a shared memory model. Millions of little Erlang processes will be GC’d independently which simplifies things a ton.

That isolation model probably wouldn’t translate to the greater code ecosystem even though it’s very effective for BEAM applications.

⬐ littlestymaar

Also, Android is more than the SDK: almost every game running on this platform uses the NDK.

⬐ ldng

The NDK is a wrapper above a slew of C/C++ lib and those are not garbage collected.

It make me think of PHP. The language is actually really slow but because most of it's standard library are wrappers around actual C library, a lot of people perceive it as a really fast language.

But compared to raw C/C++/Rust, it is still slow.

⬐ pjmlp

The NDK is very limited, its sole purpose is for 3D, real time audio, ML and porting ISO C and ISO C++ code into Android, everything else requires JNI, even reading files outside the APK sandbox.

⬐ samus

There are Java APIs that give access to raw memory. It's one of the infamous, largely undocumented sun.misc-stuff though. There are libaries that used it to implement efficient off-heap data structures. Because of modularization its future looks grim though. After all, Java was not designed with having an unsafe subset of language constructs like Rust has. It would degenerate to C++ in terms of safety.

⬐ badsectoracula

GC doesn't prevent you from writing an OS, see Project Oberon[0] for an example of a system with its own CPU (implemented in FPGA), OS, language and applications by Niklaus Wirth which actually does use a system-wide garbage collector.

The language does allow for direct memory access though through some special functions that the compiler converts to direct memory accesses.

[0] http://www.projectoberon.com/

⬐ jhasse

There's the Android NDK though.

⬐ pjmlp

Which is very constrained and requires JNI to interact with OS APIs.

Even the stable NDK APIs are actually JNI wrappers, with exception of the 3D and real time audio ones.

⬐ arghwhat

It is not technically a matter of interacting with C, but rather a matter of being able to interact in a very controlled fashion.

Both JavaScript and Go are fantastic of examples of languages deeply incompatible with OS work, as they expect the ability to manage threads, memory, and even networking as a core part of their language runtime and feature-set. None of this is going to work when you're trying to implement those building blocks itself. It is also not acceptable for a language to decide to do something else outside of the authors' control when working on a kernel, like both JavaScript and Go wishes to do.

Attempts to write kernels in Go have been made, but doing so involves defining a subset of Go so small that what you end up with is really just a small new language with a Go-ish syntax, without any of the features.

So no, C interaction is by itself not a requirement, but it's more of an indicator for the ability to be in control.

⬐ badsectoracula

> they expect the ability to manage threads, memory, and even networking as a core part of their language runtime and feature-set.

These can be implemented using a subset of the language that does not need them, pretty much like C++ requires 'new' but when writing a kernel you do not have enough functionality to use it - so you don't, until you have provided enough infrastructure for its use. Or in a similar fashion...

> None of this is going to work when you're trying to implement those building blocks itself.

...most C stdlib functions are also not available when writing a kernel in C (e.g. malloc/free, fopen/fclose, etc) and you simply do not use them until you have enough infrastructure in place to implement them.

⬐ arghwhat

> These can be implemented using a subset of the language that does not need them, pretty much like C++ requires 'new' but when writing a kernel you do not have enough functionality to use it - so you don't, until you have provided enough infrastructure for its use. Or in a similar fashion...

Definitely. That subset for languages like Go and JavaScript is, however, so small that nothing is left. And for a kernel, that subset would have to be used in the vast majority of the code base.

If your goal is to write C with Go/JavaScript syntax, then that might be good enough for you, but that's hardly the point of "Should we rewrite in Rust" talks.

> ...most C stdlib functions are also not available when writing a kernel in C (e.g. malloc/free, fopen/fclose, etc) and you simply do not use them until you have enough infrastructure in place to implement them.

Of course, that's also how you'd do it in Rust: Don't use things that aren't there.

The Go runtime won't give you that choice, though, so you'd have to remove that, which removes pretty much all the language features that makes Go... Go.

⬐ scoutt

Write from scratch, maybe. Rewrite, I don't think it will happen anytime soon. In fact, the more time it passes, the more code needs to be rewritten.

Regarding firmware, if the talk was about bare-metal embedded development, then I think Rust is the solution to a problem that no one has.

Since the food on my table comes from firmware development, I passed several days trying to understand the reasons to move to the approach described here: http://blog.japaric.io/brave-new-io/ (Embedded in Rust blog, great reading BTW).

But found little. It's a nice toy but it may be too much. If that is the trend, then I think Rust for Embedded is the answer to a question no one asked, because in my work, if I need to set a bit into a register, I have to be sure it's fking set; not playing around "just because it may be problematic".

⬐ blub

I hope this idea of taking the complexity of memory management and encoding it in the language syntax is an evolutionary dead end, because that Rust code looks genuinely painful.

Otherwise I agree that a rewrite is nonsense. In these kinds of huge projects, only a gradual solution that can co-exist with the existing code base could work, if anything. An example would be tightening the safety of the existing code through static analysis of the kind offered by Frama-C.

⬐ pjmlp

The problem with static analysis of the kind offered by Frama-C, is that many developers not even at gunpoint bother to use them.

⬐ xemoka

Prior comments: https://news.ycombinator.com/item?id=18405804

⬐ andy_ppp

Call me crazy but I'd love an OS the emphasised latency rather than overall performance and essentially used the Actor model to provide concurrency. You could also use this message passing to do amazing things like combining programs in interesting ways if you could hook into the stream of events. Obviously rust would be a great candidate language for this.

⬐ OJFord

Redox (an OS written in rust) has a related page: https://doc.redox-os.org/book/introduction/will_redox_replac...

⬐ None

None

⬐ Animats

Is there a summary of this hour-long video?

⬐ chubot

Here's my summary from 2 months ago:

https://lobste.rs/s/qz9d82/is_it_time_rewrite_operating_syst...

Overall I was sort of disappointed in this video. I would rather hear about someone writing their OWN code in Rust. I watched one of Cantrill's previous talks about this which I liked, so I watched this one too.

It's not that interesting to hear "well Rust has problems for MY domain (kernels), but OTHER people should write their code in Rust (user services and firmware).

There's an obvious asymmetry there, because you know all about the problems in your own domain, but you're less familiar with the problems of other domains (e.g. ones that require a lot of async code, a lot of text parsing, etc.)

-----

I just watched this video. I liked the part about failed C++ operating systems from the early 90’s (Apple’s Copland, an effort from Sun, and Taligent).

Overall he’s excited by Rust, but points out some problems with using it for kernel development:

- Multiply-owned data structures are all over Unix kernels, and Rust’s ownership system doesn’t like those. (i.e. the doubly linked list problem)

- Rust doesn’t allow you to handle memory allocation failure. This is being worked on? I didn’t know this about Rust and it seems odd for a low level language.

He says that instead of kernel development, Rust could be used for:

- user-level services like systemd, or

- firmware.

This doesn’t seem like anything more than a wish though. I don’t see that Rust really shines in those areas.

- I think you could probably write systemd in Go. I don’t see that Rust has any advantage there. Whereas a kernel in Go has some obvious downsides (I know it’s been tried).

- Rewriting firmware in Rust sounds nice, but I don’t think vendors will do it. They have enough problems writing decent C, and they don’t write open source code. He is wishing for them to write Rust and release it as open source, but I’m not optimistic about either of those things.

I think the kernel and the browser are where Rust is interesting – when you need both performance and memory safety. I’m not really excited about Rust for other applications like web services, i.e. doing the job that Go, Python, or Ruby can do. So it does seem unfortunate that there are still some open problems with using Rust for the kernel.

He also points out the many nascent Rust OSes kernels, and the fact that they are “from scratch” systems, i.e. not compatible. I agree that you need a compatible kernel to have any hope of adoption.

⬐ viraptor

> I think you could probably write systemd in Go. I don’t see that Rust has any advantage there.

With a lot of low-level operations, go may have issues with mapping its threads to native. I can't remember which kernel interface was it, (there was a post about it on HN) but you have to call it for the same object from the same system thread. Since go didn't guarantee the mapping, you could randomly fail there.

⬐ steveklabnik

> Rust doesn’t allow you to handle memory allocation failure. This is being worked on? I didn’t know this about Rust and it seems odd for a low level language.

The standard library panics upon allocation failure, but allocation is a library thing, not a language thing. You can handle failure if you do it yourself. Eventually we have a path forward to handling it for those data structures too, but we’re not quite there yet.

This wasn’t a big concern because when you’re doing OS level stuff you’re generally not using the standard library anyway.

⬐ anp

I could have sworn std aborted on allocation failure, not panicked. Am I misremembering?

⬐ steveklabnik

Ah yes, my bad https://github.com/rust-lang/rust/issues/26951

⬐ chubot

OK, thanks for the clarification. C++ also has this issue, because vector<> and so forth can raise std::bad_alloc. So AFAIK basically nobody who writes kernels in C++ uses STL.

Although on second thought, I guess you can handle it because it's an exception rather than a panic. But exceptions have other downsides so people don't use them in that domain.

⬐ steveklabnik

No problem. And yeah, it’s effectively the same. You can catch panics if you really need to, but it’s certainly not idiomatic generally. (And most OS-level code makes them aborts which cannot be caught.) This is a situation where it would be... fine. Not as nice as making the api fallible from the start.

⬐ steveklabnik

My summary (I saw the talk live, this is from memory):

Some history of OSes, a bit about Rust. Is it worth re-writing the OS? No, but what about incorporating Rust into OSes, instead of creating a new one from whole cloth? There’s three approaches: in the kernel? Probably not worth it. In userland? Probably worth it. For firmware? Probably worth it.

⬐ drpixie

Thanks - I'm completely over video, when I can read a little text at my own speed, and skim much more effectively. Video is reallyn inefficient, both in bytes and in my time.

⬐ xuesj

Freedom is not free, what people make what ruler.

⬐ PhantomBKB

Oh god

⬐ steveklabnik

You should watch the talk.

⬐ jeffrallen

No. (https://en.m.wikipedia.org/wiki/Betteridge%27s_law_of_headli...)

⬐ littlestymaar

It's literally the first slide after the title!

⬐ jeffrallen

My comment was aimed at typical HN readers who cannot be bothered to visit the link...

⬐ ncmncm

Since they started putting Rust in Firefox, it crashes literally hundreds of times a day. (Usually the crashes are in subprocesses. Not always.) Is Rust to blame? I don't know. But I would rather see some attention to the crashes than any new features, or new OSes, right at the moment.

⬐ lytedev

If you're truly seeing crashes "literally hundreds of times per day", I'm sure the Firefox team would appreciate the information you have! I suspect that the problem is not Firefox itself, which I put through the ringer daily and am extremely happy with it.

⬐ beatgammit

Usually this is graphics driver related, but there could definitely be some bugs that OP's unique usage triggers.

⬐ tdurden

I find it hard to believe that you continue using software that crashes "literally hundreds of times a day".

⬐ user5994461

Don't we all use software that crash every day? Maybe not a hundred times but more than zero.

⬐ arghwhat

Why in the world would you use software that crash every day?

Unless it's a piece of code I wish to help improve, I see no reason to waste my time with broken tools.

⬐ zamadatix

Err... no? That would be ridiculously annoying. If you're having issues every day it's either time to find new software or new hardware.

⬐ tdurden

I don't, but I can see how one could put up with it for a critical piece of software with no alternatives. In this case though, we are talking about hundreds of crashes a day for software that has multiple alternatives.

⬐ littlestymaar

For a “critical piece of software with no alternatives” you could imagine having one to a few crashes a day, but not hundreds of them. Using such a buggy software for “critical” stuff would be close to insanity ;).

⬐ None

None

⬐ beatgammit

My computer crashes up to every couple days (Ryzen CPU lockup bug), but aside from that, I just don't put up with it. And I only put up with my computer crashing because it's more costly to fix than it is to deal with a couple minutes of downtime every couple days. And even then, I have spent a couple hours trying out patches for the crashes, which involves patching the kernel.

I can't think of any software I use with any regularity that crashes daily (well, other than code I'm working on), and yes, I use Firefox as my main browser, so I'm not seeing the behavior mentioned here. I tend to leave applications open for days on end (or until my computer crashes). If something crashes often enough that it becomes expected, I look for solutions and replacements.

So yes, crashing isn't normal.

⬐ alkonaut

Not even a program I use full time that crashed once per week would be acceptable.

⬐ xiphias2

Of course, for me it's the Julia interpreter that crashes all the time. After I fix my program, and write some new code it crashes again, but differently.

But for programs that I haven't written: no, they generally don't crash too often.

⬐ arghwhat

Rust is not Firefox.

Rust does not disallow crashes, it disallows that your code will run in unsafe manners, either with compile-time errors or runtime panics (with some exceptions for `unsafe` code).

Browsers are to some extend more complicated than the operating system they run on, so it's no surprise that there are bugs in there.

If you experience crashes, please do submit the crash reports, and if you already know what the cause of the crash is, please file a bug on bugzilla. Thanks!

⬐ robin_reala

it disallows that your code will run in unsafe manners

To be picky, it specifically allows unsafe code as long as it’s marked up as such.

⬐ tasty_freeze

I have three different (windows) machines which use firefox daily, and I get a subprocess crash less than once a month.

⬐ majewsky

How does a subprocess crash look like? Not sure if I've ever seen one.

⬐ tialaramex

Firefox will tell you that the tab you opened or are looking at or whatever has crashed, it will apologise and offer to try loading it again. In my experience that usually works, but in principle of course it might not.

I've used Mozilla M-series browsers back at the turn of the century and we crashed those "hundreds of times per day" but that's because we were DOM mangling back when that was new and exciting, not how everything works in every popular web page. Also the M-series weren't finished, hence the milestone number instead of a release version.

⬐ smolder

If that's really the case, I think you have a hardware stability problem. Perhaps firefox is using AVX instructions or doing something else your system doesn't like.

⬐ stouset

On the flip side, I switched to Firefox once the first “quantum” version was released (62?) and I haven’t experienced a single crash.

Then again, I don’t remember seeing a crash in either Safari or Chrome in ages, but the point being that Firefox seems at least as stable as it’s competition.

⬐ pornel

As someone who's rewritten several things in Rust, and even written C to Rust "transpiler", I totally agree with the approach of keeping working C code as-is, and switching to Rust for new code.

Rewrites from scratch are time consuming, and at risk of second system syndrome (especially when you suddenly have all the shiny new language features).

Direct 1:1 rewrites are underwhelming. After the initial C to Rust pass you end up with the same program, with a C-like architecture, and all you've done is replaced gcc with a slower rustc.

But wrapping a C implementation in a Rust API works great. You can translate C's implied ownership semantics into explicit Rust types. You can wrap all these C structs/handles with their init and free functions into RAII that never leaks.

It's something you can do in days, not months to years. You keep a fully working program at all times. You benefit from Rust for new development, and you can gradually refactor C out of it when and where it's sensible to do so.