HN Theater @HNTheaterMonth

These discussions always remind me of some talks by Jonathan blow where he argues rust doesn’t solve the hard problems he needed it to. There are podcast episodes with better discussions, but the main one people link to seems to be this

I haven’t used rust, but I sympathize with his problems with pointers and learned a lot from the various ways people try to solve them

https://m.youtube.com/watch?v=4t1K66dMhWk

⬐ None

None

⬐ tbillington

He also admits he hasn't written more than a hundred lines of code with it before, and that was 4 years ago, more than half rusts lifetime ago.

My understanding is his priorities are C level control over memory, fast compiles, and ergonomics for the kinds of operations he commonly does.

Rust solves for memory/ergonomics, though the strictness it holds you to is in opposition to what (I understand) he considers ergonomic access & control of memory.

Therefore rust solves ~1 of his priorities, so it's not really a good fit. To extrapolate that to "doesn't solve the hard problems" is another logic step that I probably don't agree with.

Right, it sounds like you are circumventing the borrow checker and then experiencing some of the classes of bugs it was supposed to prevent. And this seems common: https://www.youtube.com/watch?v=4t1K66dMhWk

⬐ verdagon

> Circumventing the borrow checker

Programs often require inherent state with data that refers to other data. In these cases, one must circumvent the borrow checker, whether it be with indices, IDs, Rc, or whatever. The borrow checker simply does not allow changing data when someone has a reference to it (except for the rare case where we can use Cell).

It's a myth that we can rewrite any program to not circumvent the borrow checker.

The hvasilev's comment got flagged and I could not reply to it anymore, so I'll reply here (sorry) and copy-paste the hvasilev's comment verbatim below, for the sake of commenting on it's claims:

---

Reality is not on the side of this language. 11 year old, has a very low adoption with virtually no jobs associated. (https://www.tiobe.com/tiobe-index/)

On the other hand if you search for "Rust" in the latest "Who wants to be hired?" thread, you will see it is quite popular with unemployed people.

The reality is that the language has a lot of friction, the ergonomics are bad, the syntax is heavy and some poor decision making has been made there for a systems-level programming language.

There are a lot of ideological traps in this industry and many people that fall for them. Why people are interested in ideologies and cults is beyond me.

[ A screenshot: https://imgur.com/a/tgAETjh ]

---

<rant>

Now my comments on the issues mentioned in the hvasilev's comment.

> Reality is not on the side of this language. 11 year old, has a very low adoption with virtually no jobs associated. (https://www.tiobe.com/tiobe-index/)

Tiobe index is shit. The most flattering thing I've read about it states that it (poorly) depicts quantity of educational materials available online for particular programming language. Unfortunate naming of programming languages after letters of alphabet, symbols (++, #) and real-life stuff (like islands) doesn't help this rating either.

That said, Rust isn't that popular and isn't growing much according to other better language ratings:

1. https://tjpalmer.github.io/languish/

A Github-based rating created by the author of Context Free YouTube channel. For 2021Q3, Rust is on 18th place with Mean Score of 0.82% (up 0.01% from 2021Q2).

2. https://madnight.github.io/githut/#/pull_requests/2021/3

A Github-based rating. For 2021Q3, Rust has 0.64% of Pull Requests, 0.30% of Pushes, 1.29% of Stars and 0.65% of Issues. Growth dynamic is quite flat.

3. https://redmonk.com/sogrady/2021/08/05/language-rankings-6-2...

Latest RedMonk language rating, dated June 2021. Rust is on 19th place with 0 growth.

---

About jobs comparison.

Indeed.com for California.

Rust: 527 jobs (with some unrelated stuff mixed in)

https://www.indeed.com/jobs?q=rust%20developer&l=California

JavaScript: 16,792 jobs (31.8x)

https://www.indeed.com/q-Javascript-l-California-jobs.html

Java: 12,418 jobs (23.5x)

https://www.indeed.com/q-Java-Developer-l-California-jobs.ht...

C++: 4,172 jobs (7.8x)

https://www.indeed.com/q-C++-Developer-l-California-jobs.htm...

Indeed.com for New York.

Rust: 85 jobs

https://www.indeed.com/jobs?q=rust%20developer&l=New%20York%...

Java: 4,815 jobs

https://www.indeed.com/jobs?q=java%20developer&l=New%20York%...

Javascript: 4,037 jobs

https://www.indeed.com/jobs?q=javascript%20developer&l=New%2...

C++: 1,126 jobs

https://www.indeed.com/jobs?q=C%2B%2B%20Developer&l=New%20Yo...

glassdoor.com without location set.

Rust: 492 jobs

https://www.glassdoor.com/Job/rust-developer-jobs-SRCH_KO0,1...

Java: 45005 jobs (91x)

https://www.glassdoor.com/Job/java-jobs-SRCH_KO0,4.htm

Javasript: 30952 jobs (62.9x)

https://www.glassdoor.com/Job/java-script-developer-jobs-SRC...

C (with unrelated stuff): 9594 jobs (19.5x)

https://www.glassdoor.com/Job/c-developer-jobs-SRCH_KO0,11.h...

Go(lang): 1406 jobs (2.85x)

https://www.glassdoor.com/Job/golang-developer-jobs-SRCH_KO0...

Judge for yourself if a few hundred jobs in places like CA and NY count as "virtually no jobs".

For comparison, in Ukraine (population 35-41 millions) there is 6 Rust jobs:

https://jobs.dou.ua/vacancies/?search=Rust

... 190 C++ jobs:

https://jobs.dou.ua/vacancies/?search=C%2B%2B

... and 676 Java jobs:

https://jobs.dou.ua/vacancies/?search=Java

... listed on the largest Ukrainian programming site.

---

> The reality is that the language has a lot of friction, the ergonomics are bad, the syntax is heavy and some poor decision making has been made there for a systems-level programming language.

Well, this is matter of taste, largely. But I have a few issues with Rust syntax too (IMHO):

1. F--king single quotes. Eww, really?! IIRC, a tilde (~) character was used for lifetimes until some Europeans (?) complained that their keyboards have no tilde. I wonder, how they programmed in C++ all that time? For years, if I met online a piece of code that was highlighted as a comments mishmash I knew exactly in what language it was. Ugly as f--k.

2. Closures using pipes (|). With no arguments they look like OR operator (||). Distracting.

3. Using angle brackets for generics.

4. Double colons (::) as "path qualifier" produce too much visual noise. Java likes long pathes too and uses dot (.) as separator just fine.

5. What with this arrows (->) before return types? Seems unnecessary. Couldn't return types be purely positional as in Go?

I don't use Rust so it's mostly "glimpses from the outside".

Speaking of friction. This reminded me of a video by Jonathan Blow (creator of Braid and The Witness games and Jai programming language). The video is worth watching whole but piece about friction in gamedev starts approximately at 49:23.

"Rant: Entity systems and the Rust borrow checker ... or something."

by Jonathan Blow

Sep 14, 2018

https://www.youtube.com/watch?v=4t1K66dMhWk&t=2962s

As for poor decision making, it was pretty poor decision to include an npm knock off into the language. I'm speaking of crates.io repository. For some time it has a squatting problem that isn't fixed yet:

https://old.reddit.com/r/rust/comments/9aaanw/cargo_crate_na...

3 year old thread, the squatter is still there holding 104 packages. At least npm has namespaces.

I wonder, if cargo will turn into malware-ridden micro-dependency hell too?

---

> There are a lot of ideological traps in this industry and many people that fall for them. Why people are interested in ideologies and cults is beyond me.

"MongoDB is web scale" video nicely illustrates "ideological traps" and cult-like behaviors in "this industry".

http://www.mongodb-is-web-scale.com/

https://www.youtube.com/watch?v=b2F-DItXtZs

And what else illustrates cult-like behaviors? Flagging an innocent comment you don't agree with. What was in the hvasilev's comment that warranted its removal? In my opinion, nothing. It contained no insults, no personal attacks, and was more or less factually correct. Rust is relatively unpopular, complex, ideological, syntactically-heavy language with relatively few job offerings, i.e. pretty much what the hvasilev's comment said.

Complaining about the comment being "off topic" is somewhat funny given Rusters' penchant for inserting their language into discussions about other programming languages (especially C, C++ and Go).

</rant>

⬐ the_only_law

> Complaining about the comment being "off topic" is somewhat funny given Rusters' penchant for inserting their language into discussions about other programming languages (especially C, C++ and Go).

The childlike “well they do it too” argument is almost the perfect example of how the anti-rust crowd is becoming even more obnoxious than the infamous Rust evangelism strike squad.

For the record , I’ve never written a line of rust in my life and am not particularly invented in its success or failure. And no, I didn’t flag GP.

⬐ fee1-dead

> What with this arrows (->) before return types? Seems unnecessary. Couldn't return types be purely positional as in Go?

I guess you could say `->` is too verbose and you can omit it in other languages. Rust has a complex type system and there can be confusing code when you omit `->`.

1. Closure Return Types.

You can define a closure with an explicit return type:

``` let my_closure = |i: u32| -> u64 { i as u64 }; ```

Now how do you omit the arrow here? How do you know `u64` is the return type and not constructing a struct?

2. Parsing stuff

It becomes impossible to parse none-delimited types. Is `fn() fn()` two different types or a function pointer returning a function pointer?

3. Readability

I mean, tokens can be read out loud and omitting it stops making sense.

`fn foo(bar: i32) -> f32` can be read as a "function named 'foo' that takes an argument named bar with type i32 and returns f32". The word returns directly corresponds to the `->` token.

Rust also has the `!` (read: never) type. Poorly formatted code when `->` is omitted is very confusing: `fn a()!` when compared to `fn a()->!`, or just one character generic types: `fn a<T>()T` compared to `fn a<T>()->T`.

⬐ arkush

Thank you for clarification.

Seems like arrow (->) is the best choice for Rust, given arrow's visual distinctiveness and search-ability.

On why ECS gets in the way for him:

> Because it is far more complicated, thus takes far more work, than what you actually need to do. That work has a large opportunity cost. https://twitter.com/Jonathan_Blow/status/1427378984145154048

To me this has more nuance than, “you ain't gonna need it”. I don't think “concrete implementations” would do all that much to strengthen his argument that unnecessary complexity gets in the way of shipping for indie devs:

> Even 10% friction more than I ever had would have killed me. I wouldn't have been able to make the things I had. Even 5% more friction would have been really bad. https://youtu.be/4t1K66dMhWk?t=3635

And…

> I have, several times, built games where I barely managed to finish. … I've just experienced that too many times to increase friction. I need to decrease friction. https://youtu.be/4t1K66dMhWk?t=3072

And again, in relation to entities rather than Rust's borrow checker:

> If you are trying to focus on the way your entities are set up, you are mis-directing your effort and that's going to make it harder. Try to solve the problem that makes your game interesting. What is it about the gameplay that makes it interesting … that users can see? Focus on that, solve those problems. https://www.youtube.com/watch?v=w7W3xM2tzRA

On what he uses instead of ECS/components in his engines:

> One struct per entity type, with a base struct that is common to all of them. https://twitter.com/Jonathan_Blow/status/1427376307453665280

⬐ meheleventyone

Frictions a funny thing in this sense and what hurts one person helps another.

One example is that a concrete entity is much harder to change than one that is composed of concrete components. So for example an artist I work with took some components from an FPS game we made, some other random components we had and a couple of components made in people’s spare time, made a bunch of art and ended up with a pretty convincing prototype of a multiplayer shooter. He couldn’t have done that at all following a concrete entity approach.

Composition and in particular making composition data-driven and runtime malleable is very flexible even if you don’t care for the ECS approach.

But a large part of the problem is that we end up promulgating opinion divorced from context and often a lot of that context is not really more complex than the approach someone is used to. Like if you have no problems with the concrete entity approach which has been a successful pattern since forever then there isn’t that much compelling you to change. That doesn’t make it the one true way or that because someone famous and opinionated likes it that there isn’t an alternative that will better serve someone else’s needs.

Horses for courses.

I just recently came across these two videos.

Using Rust For Game Development by Catherine West: https://www.youtube.com/watch?v=aKLntZcp27M

Counter-rant from Jonathan Blow: https://www.youtube.com/watch?v=4t1K66dMhWk

⬐ badsectoracula

I'm not sure if it was that first video or i saw/read about this approach somewhere else (i'm almost certain that i heard about it years before that video though), but my reaction was pretty much the same as Jon Blow's (though not so long winded :-P): aren't you just working around the borrow checker and making your own allocator with its own faux pointers via indices? Sure, it wont crash the game if your index is invalid, but you'll still access the wrong data - which can end up with hard or weird bugs and corrupted state (e.g. in savegames and/or editor).

⬐ andrewflnr

Yeah, that's pretty much working as intended. The first priority of Rust's safety is crashes and security. Any other bugs that get squashed are a bonus, and you can't expect it to get all of them (whatever any overly enthusiastic evangelists might say).

⬐ adwn

> Counter-rant from Jonathan Blow

I've just watched the first 20 minutes of that video. So far, it's all "yeah, she's basically right". When does he get to the point? I.e., what does he actually object to?

⬐ indy

An issue that Jonathan Blow had was that one of the touted benefits of Rust is it's ownership semantics, yet the ECS (Entity Component System) that the talk demonstrated was effectively bypassing that.

⬐ adwn

> ownership semantics, yet the ECS (Entity Component System) that the talk demonstrated was effectively bypassing that.

I don't see how that's the case [1]. The entities are owned by the ECS (or the game state, or whatever), not by each other. Entities can conceptually reference each other, but not own each other. The only time this is problematic, is when one entity is destroyed while another one is still holding a reference to it. At the beginning of the video, he discussed basically all approaches to solve that problem:

1) Raw pointers. Bad for obvious reasons.

2) Smart pointers which keep the referenced object alive. No good, he says, because one entity should not keep another one alive, if the game logic says it should be removed.

3) Weak pointers which are safely invalidated when the referenced entity is removed. No good, he says, because keeping track of back-references is inefficient.

4) Weak pointers which check whether the referenced entity is still alive before accessing it. Which is exactly what the ECS does. Rust's ownership semantics still help here, because entities are unambiguously owned by the ECS, which returns a type-safe None value when you try to access a deleted entity.

[1] I'm not arguing against you, indy; I understand that you paraphrased Blow's argument.

⬐ meheleventyone

Yeah the confusion comes because entities are no longer a concrete element of the program but a concept tying components together. It’s just as valid to look at a subset of the components that are referenced by an entity id as a view as it is to look at them all. Like a relational database.

And by breaking things into components the granularity of ownership is increased compared with the equivalent concrete representation of the same data. So whilst you can’t reason about ownership of an entity as it primarily exists conceptually the ownership of its constituent parts is well defined.

⬐ gameswithgo

tldnr is that Rust guides the programmer only halfway to the solid engineering solution, the programmer is on their own to make the arena solution safe.

its a pedantic nitpick, as is his way. also keep in mind that of late Blows focus has been single player video games made by very small very talented teams. A domain where the benefits of Rust are very small, possibly even a net loss compared to the fast compiling language they use now.

⬐ pmarin

Keep watching.

⬐ neutronicus

Her talk is basically "Introduction to Entity Component Systems for Rust Programmers," and it has an implicit secondary thesis that implementing the ECS in Rust adds value vs implementing it in e.g. C++.

He seems to disagree with that secondary thesis. He thinks she did a good job implementing a toy ECS, but that Rust itself wasn't particularly helpful to her.

⬐ beigeoak

Idk who that guy is in the counter-rant portion, but if he can make a 1-hour long video against a language, why doesn't he make his own language instead or write his own game engine in his own language or something like that? I highly doubt he would get sub 1-second compile times, like he claims at 1:03:52.

Talk is cheap, SAD!

⬐ Thaxll

The sutdio Catherine West worked for stopped using Rust and went back to C++ fyi.

https://www.reddit.com/r/programming/comments/atyzz4/halley_...

⬐ ImprobableTruth

... because Catherine West left for personal reasons and so it was easier to shift the project over to their existing and already working C++ engine (that they were continuously developing and using, so they didn't really 'go back').

Jon Blow addresses this in some earlier rant here: https://www.youtube.com/watch?v=4t1K66dMhWk

His point being that now you're effectively implementing a form of manual memory management to circumvent the borrow checker.

Pointers living through a render pass aren't the main use case he is interested in. It's a problem space that many engineers are familiar with so it's easy to talk about. Game developers have some very domain-specific problems that are less common outside of game engines. A more detailed description of his opinions are laid out here where he goes into much more detail about his experiences porting The Witness to mobile (I think): https://www.youtube.com/watch?v=4t1K66dMhWk

And the response by Jonathan blow https://youtu.be/4t1K66dMhWk

One of his points is that you still have to solve synchronization for allocations and deletions from the arrays.

Jonathon Blow had a really interesting but subtle response to this talk: https://www.youtube.com/watch?v=4t1K66dMhWk&feature=youtu.be

⬐ brodo

Thanks, very interesting. He is right, the Rust borrow checker can not deal with arbitrary graphs and she added a memory management layer on top of the Rust one. However, you would have to do the same in C++. It is an interesting obervation. I spoke to Benedikt Meurer from the Google V8 team recently and he pointed out the same for Javascript Engines. If you do not have DAGs, Rust can‘t help you with safety. (at least it can not prevent use after free) I still like the whole ECS idea and it makes sense for me that using ECS in Rust will provide a performant and clean architecture.

jonathan blow is not a big fan of ECS

https://www.youtube.com/watch?v=4t1K66dMhWk

⬐ gameswithgo

It isn't appropriate for all games. Which is important to keep in mind. Sometimes people get crazy and do these really complex high performance ECS systems for a 2d game with 10 entities. Or a game with only 2 kinds of entities. Just unnecessary complexity in those cases.

⬐ platz

I am not a game dev, but from watching some of his stuff, I have a feeling he would invert that claim, and say ECS is fine for smaller games but for really involved systems with time-to-market pressures the framework is too limiting.

⬐ mcphage

> but for really involved systems with time-to-market pressures the framework is too limiting

It was specifically designed and developed for involved systems.

⬐ jungler

Jon is also known for being hugely opinionated with only his direct personal experience backing him up.

His basic thought on ECS is that it's "obvious". And if you have worked on an involved game a few times and tried to examine the performance bottlenecks of the architecture, it really is. There are different flavors of "how to ECS" that make tradeoffs between static compile-time composition(theoretical fastest ECS: an entire scene is fed into a compiler and it emits a custom bespoke memory layout and an API for it) and runtime dynamism(most flexible ECS: dynamic types, reflection, some code to automatically maintain indexes). But the basic principles of working directly with compositions of plain old data are ingrained throughout, and emerge naturally from trying to extend a simple game with simple data structures and an imperative code style, without resorting to OO inheritance(which was tried and discarded because properties inevitably crept upwards into the parent class leading to a memory-hungry "God Object").

⬐ platz

I don't think I'm claiming that Jon would advocate for OO inheritance. I also think he thinks that SoA instead of AoS is a good thing - no qualms there. I guess he would probably have the most comments on how to deal/access/manipulate that SoA at a higher level

⬐ dfsdfsdfsdfs

couldn't make it through as he says "right?" at the end of every sentence

⬐ Impossible

Jonathan Blow's commentary isn't anti-ECS. It's more skeptical of Rust's borrow checker, more specifically he's pointing out how the original talk ends with bypassing the borrow checker and copying a pattern that would have equal safety in C or C++.

⬐ platz

That is true + there is an offhand quote he makes about ECS in general, but 99% is about the issue you mention

This was extensively discussed on reddit and a few other places (not HN afaik, or I couldn't find the thread), after the video^ Jonathan Blow posted on the topic.

I think the tldr; is, opinions vary.

Using indexes manually doesn't by-pass the borrow checker, its just a different, manual, memory allocation and management strategy.

It preserves memory-safety, but, it's questionable if you're better or worse off in terms of correctness of application logic when you use it.

...you're probably better off using the borrow checker (that's why it exists) or an abstraction in a crate to deal with this sort of problem, regardless of whatever implementation strategy it uses internally (unsafe, this, etc).

[1] - https://www.youtube.com/watch?v=4t1K66dMhWk

I thought this was going to be a response to Jonathan Blow's video about how doing your own memory management is effectively turning off the borrow checker: https://www.youtube.com/watch?v=4t1K66dMhWk

The takeaway being that the borrow checker doesn't magically prevent the use-after-free class of bugs. Although you will never experience a segmentation fault in safe Rust, the bug is still there and your program keeps running in an invalid state. The symptoms are changed, but no less dangerous.

To make the problem even more obvious, think of allocating a large array to be used as a heap and handing out indices to implement your own malloc. You have bounds checking to prevent indexing outside the bounds of the heap, but it doesn't really help when the elements have logically different lifetimes and occupy different parts of the array. I don't think this is a contrived example either. A less obvious version of this can easily creep into large or complex systems, as evidenced by the Entity Component System in Rust example.

⬐ pcwalton

> The symptoms are changed, but no less dangerous.

The symptoms of a use-after-free-style logic error are less dangerous in Rust, because it's much harder to get RCE.

⬐ rectang

What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us.

For memory management, some of the major algorithms available are global allocation, stack allocation, malloc/free, reference counting, and tracing garbage collection. (The Entity Component System model is doing your own allocation so it's a subtype of either malloc/free or ref counting.)

Stack allocation and global allocation with borrow checking seem very attractive in terms of safety, but you have to know more in advance about how the memory is going to be used because growing the allocation while you are deeper in the stack is not possible. Are there any idioms or algorithms which are particularly friendly to this model? For instance, traversing a preliminary object graph to establish max allocation requirements before allocating a large block on the stack?

⬐ steveklabnik

> What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us.

We all would. They're still emerging! I think the generational index idea is one of them, though.

⬐ zrm

> The symptoms are changed, but no less dangerous.

Potentially even more dangerous. Having the program segfault immediately is much preferable to, say, Heartbleed.

⬐ barrkel

If only we could guarantee immediate segfaults.

⬐ zrm

> If only we could guarantee immediate segfaults.

It should be possible to create a malloc implementation that does that by making the minimum allocation size a page and then not reusing virtual addresses for new allocations. Then once an allocation is freed, any access to it is permanently a segfault.

That may not be practical on existing architectures with 48-bit virtual addressing though, since you could plausibly exhaust the address space. The full 64 bits might be sufficient for most things at least.

You could also get most of the benefit by not reusing virtual addresses until you run out.

⬐ steveklabnik

It's not, because that relies on actually doing the de-reference. Thanks to UB, that may never actually happen, the code may get removed entirely.

⬐ zrm

If the code is removed entirely then what memory is being improperly accessed?

⬐ steveklabnik

It's impossible to tell, as that can cause other issues. See the link I posted about time travel elsewhere in the thread.

⬐ zrm

The day will come when the compilers that do things like that will be righteously categorized as malware.

⬐ the_why_of_y

The day will come when language specifications that allow compilers to do things like that will be righteously categorized as archaic.

⬐ kibwen

That's literally what Rust compiler errors are doing. :P

⬐ kibwen

Any situation where C would segfault immediately, Rust would also segfault immediately. The only difference is that Rust references are an alternative to C pointers that provide stronger safety guarantees at the cost of restricting what is possible (in other words, the same tradeoff that all static analyses make, including every type system). You can also use C-style pointers in Rust, but you can get away with just using references the vast majority of the time, which gives you better memory safety guarantees at no runtime cost since references are just pointers at runtime (and they're often automatically marked as restrict pointers at that). You're not going to get Heartbleed in Rust by using references.

⬐ kbwt

> Any situation where C would segfault immediately, Rust would also segfault immediately.

For equivalent code, yes.

The borrow checker can however compel you to write code that would not immediately panic in Rust, when the C code you would have written has at least a chance of segfaulting immediately or being detected with valgrind.

⬐ kibwen

I'm afraid I don't follow. Can you give me an example of the borrow checker compelling one to write such a piece of code? I admit I can't think of an example of Rust code that would fail to panic when the equivalent C code would tend to segfault.

⬐ zrm

The issue isn't the equivalent of Rust code in C, it's C code without an equivalent in Rust.

Rust should prevent the typical UAF bug in C, but if the programmer unwisely insists on using the same pattern in which they fail to track object lifetimes correctly, they can still implement it by using array indexes instead of pointers. Which can also be done in C (as was the case in Heartbleed), but it isn't as common, and is potentially worse than the usual C UAF because it produces a consistent successful out of bounds access rather than at least the possibility for a segfault.

It's possible to prevent someone from doing something bad and have that cause them to do something worse. "Nothing is foolproof to a sufficiently talented fool" etc. Safer tools are not a replacement for competence.

⬐ kibwen

> The issue isn't the equivalent of Rust code in C, it's C code without an equivalent in Rust.

I'm afraid I still don't follow; Rust also has C-style pointers, they just require one to use the `unsafe` keyword to dereference. I can't think of any C code that can't be expressed in Rust in this way.

> Rust should prevent the typical UAF bug in C, but if the programmer unwisely insists on using the same pattern in which they fail to track object lifetimes correctly, they can still implement it by using array indexes instead of pointers.

I don't see how this shows that Rust is any more dangerous than C; a C programmer is no less capable of mistracking object lifetimes. Furthermore, array access are checked by default in Rust; even in the degenerate case of using the `get_unchecked` method (which requires the `unsafe` keyword as well), that's no more dangerous than every C array access.

> Safer tools are not a replacement for competence.

Why can't we as an industry have both safe tools and competent programmers? These are not mutually exclusive.

⬐ zrm

> I'm afraid I still don't follow; Rust also has C-style pointers, they just require one to use the `unsafe` keyword to dereference. I can't think of any C code that can't be expressed in Rust in this way.

What I mean is without an equivalent in Rust that doesn't require unsafe, which people are correctly admonished to avoid using whenever possible.

> I don't see how this shows that Rust is any more dangerous than C; a C programmer is no less capable of mistracking object lifetimes. Furthermore, array access are checked by default in Rust; even in the degenerate case of using the `get_unchecked` method (which requires the `unsafe` keyword as well), that's no more dangerous than every C array access.

The equivalent things are equivalent. The issue is that if you call one dangerous thing unsafe but not another thing that is as or more dangerous, the naive programmer assumes that not having to use unsafe means whatever they're doing is safe.

And it's little help to bounds check the array because that wasn't the problem. The data the programmer was expecting to be at that index isn't there anymore but the array itself still is and there is now some other data there.

> Why can't we as an industry have both safe tools and competent programmers? These are not mutually exclusive.

It's not that we can't have both, it's that we should have both.

Whenever we get better tools, people are tempted to hire worse programmers, because better programmers are more expensive and worse programmers with better software should be as good as better programmers, right?

This is obviously at odds with the ideal of using the new tools to create better software, and doesn't actually work, because a smoke detector can't save you if you don't know what to do when it warns of low battery.

⬐ Rusky

The example that kicked off this discussion, replacing references or pointers with array indices, is presumably one example you're thinking of?

The problem with using that example to make this argument is that it is the equivalent code. Jonathan Blow's argument was not that the borrow checker forces people to use a worse pattern - he uses basically the same pattern in C - but that the borrow checker doesn't help you with that pattern, and so the borrow checker is extra friction.

Now, Catherine West's argument in the keynote video was that the borrow checker pushes people toward this better design, albeit without helping solve the use-after-"free" aspect, while Jonathan Blow says he would just start there as an experienced game developer, so he doesn't need to be pushed.

There's also the fact that the borrow checker isn't completely gone, and still applies to any references you put in the array or temporarily take to the array or that don't involve the array at all; as well as the fact that Rust also helps with things like parallelism, which definitely applies to games and doesn't need to be "bypassed."

⬐ sidlls

I think the primary argument for Rust advocates here is that what you describe is only possible by using explicitly "unsafe" code (i.e. code using the "unsafe" block that is required).

The weakness of this argument is exposed in your comment also: in libraries "unsafe" is often hidden behind an abstraction. To take a simple example, consider the standard Rust Vec: plenty of "unsafe" code resides in this object, but you'll rarely if ever see "unsafe" wrap typical vector operations ("unsafe { myvec[0] }"). Partial mitigation is that this does reduce the surface area where one might have to look if odd behaviors start to appear in a Rust program.

Overall it's still a net benefit.

⬐ saghm

I might be in the minority here (and I don't do game development, which is the topic of this thread), but I tend to start by using `vec.get(i)` (which returns an option) and then only switch to the index notation if I end up checking the length immediately beforehand and want to do an early return or something to avoid extra indentation in the code where I use the accessed element.

⬐ majewsky

I think modern C++ compilers are smart enough to eliminate the bounds check if they can prove that it never fails.

⬐ saghm

I mainly do it this way to reduce the chance of my code panicking, not specifically for any performance reasons.

⬐ steveklabnik

Yup, Rust will too.

⬐ steveklabnik

> ("unsafe { myvec[0] }")

Part of the point in my post is that this does not remove the bounds check. This unsafe block does nothing.

This actually might be a better example than the one I picked...

⬐ kibwen

And just as in the OP, the compiler makes it clear that this unsafe block does nothing:

    warning: unnecessary `unsafe` block
     --> src/main.rs:4:3
      |
    4 |   unsafe { myvec[0]; }
      |   ^^^^^^ unnecessary `unsafe` block
      |

⬐ sdegutis

So it’s the difference between crash early (C) and don’t crash but run wrong (Rust), inherent by design in the main selling point of Rust (borrow checker)?

⬐ andrewflnr

Broadly speaking, C is the king of running wrong without crashing, so no.

The borrow checker isn't really making anything worse, it's just failing to save you from this particular problem. In C you could still write the exact same bug, or you could even write it with actual pointers. I gather a lot of browser exploits start that way these days.

⬐ Tuna-Fish

No. Most modern C++ ECS take the same approach as the Rust ECS that is being discussed.

Also, since I started using generational indexes, I have never had a bug caused by using stale data, so at least for me it hasn't been a real issue.

⬐ gpm

I'd say that rust crashes early in a strict superset of when C crashes early.

The case where rust fails to crash early is you have are using indexes into an array as a pseudo pointer, but you've "freed" the object you wanted to use and put another (of the exact same type) in it's place. If you do the same with malloc/free in C you also aren't going to crash early. But in C you can also fail to crash early because of things like "and now an object of a different type is in it's place".

⬐ ummonk

I believe the example here is where you bypass the borrow checker by keeping your own pool of memory and then allocating objects on that pool, using a custom heap implementation with manual memory management (or with a custom garbage collector). You can do that in either C or Rust (or any other language for that matter).

⬐ tree_of_item

I don't see what there is about C that forces things to crash early. It's more like the difference between full of bugs that no one sees (C), and get the problem with your approach highlighted by the type system (Rust).

⬐ gliptic

Why is everyone assuming C is "crash early" as opposed to "undefined behaviour will crash if you're lucky and cause RCE in the worst case, or any weird thing in between". The selling point of the borrow checker (at least one of them) is that it doesn't allow undefined behaviour unless you explicitly enable unsafe operations. If there was a way to detect UB and predictably crash in a performant way, you could probably implement that in Rust as well. In fact, that's often what is done with generational indexes and similar.

⬐ shawn

https://doc.rust-lang.org/nomicon/print.html

Rust is otherwise quite permissive with respect to other dubious operations. Rust considers it "safe" to:

Deadlock

Have a race condition

Leak memory

Fail to call destructors

Overflow integers

Abort the program

Delete the production database

However any program that actually manages to do such a thing is probably incorrect. Rust provides lots of tools to make these things rare, but these problems are considered impractical to categorically prevent.

⬐ dx87

None of those are related to anything in the post you're replying to. What point are you trying to make?

⬐ steveklabnik

So, I had been thinking about this post for a while, and Blow's video caused some more discussion that made me post it. But it's not a direct response, I still haven't watched the video, and so I don't know what he actually said. If I wanted it to be a response, I would have linked to it.

> The symptoms are changed, but no less dangerous.

I would take issue with this sentiment. There's a world of difference between "logic error and/or panic" and "undefined behavior".

Yes, Rust doesn't fix all bugs. But it's still an improvement here.

⬐ kbwt

Thanks for the response. I wasn't trying to speculate on your intentions for publishing the blog post.

> There's a world of difference between "logic error and/or panic" and "undefined behavior".

Is is really so different for the programmer who wrote the bug?

If you have undefined behavior, the language implementation can do whatever it wants. It won't actively work against you, but the implementer is given permission to ignore what would happen if you violate their assumptions.

With a logic error in custom memory management, the program execution will still be following well-defined rules but the invariants assumed by the programmer will no longer hold. The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken.

Defensive coding with panics/asserts will definitely help catch some of these mistakes during development.

> Yes, Rust doesn't fix all bugs. But it's still an improvement here.

I applaud your efforts with Rust, it's great to see someone actually trying to improve the state of programming languages.

⬐ steveklabnik

It’s all good, it’s a totally reasonable thing, which was also brought up in all the other threads :)

> it won’t actively work against you

I guess it depends on what you mean by “active.” Consider the Option<NonNull<T>> case. We can do the null check in safe Rust. We know the check is done. Now consider the case with UB: https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=...

These kinds of things can cause lots of subtle issues. The rust code won’t.

⬐ oconnor663

> The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken.

I still think there are big differences here, especially when we think about these things as security issues.

If you write a program that's supposed to draw some pixels to the screen, and you have a logic bug, you program is going to draw the Wrong Pixels (https://xkcd.com/722). But your program isn't going to install a rootkit on your machine, or mine bitcoins, or send all your spreadsheets to the bad guys. If you never call the `mine_bitcoins()` function anywhere in your program, there's no way a logic bug can make you somehow call that function.

Not so with undefined behavior. An attacker who exploits a buffer overrun in your program can make you do anything. This almost sounds paranoid, but as soon as your code is taking input from the internet, this is really truly the problem you have. This sort of problem is why projects like Chrome spend millions of dollars building security sandboxes for their C++ code, and researchers still keep coming up with ways to break it.

⬐ Rusky

> Is is really so different for the programmer who wrote the bug?

Yes, absolutely. The symptoms may be similar (though the logic error will still never lead to memory corruption or segfaults), but debugging is much easier when you can still rely on the language's invariants, if not your own.

⬐ earenndil

If you segfault, then the error is clear. You're crashing, because (as the debugger or valgrind will show you) you tried to dereference this memory after you already freed it. You can then figure out why you freed it this early and change that. If you're in an undefined (according to your application's internal logic) state, it can be much harder to track down why it's acting erratically.

⬐ unrealhoang

Logic bugs with undefined system state are much easier to debug than UB, that is why people use memory-safe programming language.

⬐ kbwt

Citation needed.

There is great tooling to pin down incorrect memory accesses when you are using the system allocator (valgrind, clang sanitizers). You're truly on your own if you access logically repurposed memory within a persistent system allocation.

⬐ steveklabnik

> If you segfault, then the error is clear. You're crashing,

if you segfault. UB means anything can happen. Sometimes that's a segfault. Sometimes it means worse things.

⬐ archgoon

To point; if you free memory, and reuse it (without nulling the reference), you likely won't segfault.

Simple example (compiled on OSX)

  #include <stdlib.h>
  #include <stdio.h>

  struct X {
      int x;
  };

  int main() {
      struct X *a = (struct X *) malloc(sizeof(struct X));
      a->x = 4;
      printf("%d\n", a->x);
      free(a);  // a is no longer a valid reference
      struct X *b = (struct X *) malloc(sizeof(struct X));
      printf("%d\n", b->x); // b is probably reusing the memory used by a
      a->x = 5; // updating a probably updates b
      printf("%d\n", b->x);
  }

If you compile this without optimization (clang test.c) you'll probably get

  4
  4
  5

'Probably' because this is both relies on both undefined behavior (which is partially why turning on -O2 changes the result), and the way malloc is implemented.

Fortunately, in a simple case like this, compiling your application with '-fsanitize=address' will give a very nice error in this case. :)

⬐ earenndil

No, but valgrind will tell you.

⬐ archgoon

Not necessarily. It can tell you if it is triggered by your tests; but it won't tell you if it isn't. So if you run your test suite under valgrind, and you don't trigger the problem, valgrind won't tell you that there is a potential issue for certain inputs. Which, in this case, will result in silent corruption of the heap.

So, trivially adding argc to main,

    if (argc > 5) {
       free(a); // a is no longer a valid reference
    }
    // valgrind won't catch this issue
    b->x = 2; // valgrind complained about us dereferencing b before intializing
    a->x = 5; // valgrind won't complain about this if argc <= 5

results in a program that valgrind won't catch. Valgrind is great; but your users won't be running it when they use your program.

Now sure, you combine valgrind with other tools like afl (https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer)) or KLEE (https://klee.github.io/), and insist that your test suites have full coverage (however, code coverage isn't the same as input space coverage), but the point is, you're stuck doing runtime analysis (and need to know that you need to do that analysis) to make sure you did this right. Baking this type of error checking into the type system itself is valuable.

Given that large projects like Google's Chrome keep hitting these issues, it seems reasonable to say that they aren't strictly trivial to solve. :)

https://www.cvedetails.com/cve/CVE-2017-5036/

https://vuldb.com/?id.100280

⬐ steveklabnik

Just for fun, on Windows, I get

  > cl.exe foo.c
  > foo
  4
  10372440
  5
  > cl.exe -O3 foo.c
  > foo
  4
  9025424
  9025424

I've seen stuff like this work on OS X, and segfault on Linux. Yay UB!

⬐ CUViper

Use-after-free and other memory errors won't necessarily segfault anywhere near the source of the problem. It could also limp along, corrupting memory in weird places, until something totally unrelated segfaults instead.

ECS might let you continue with an outdated index, but that problem is contained.

⬐ nemothekid

>There's a world of difference between "logic error and/or panic" and "undefined behavior".

I recently started using Rust daily as a break from $dayjob because I've never really liked C++. I took the time to watch Blow's full rant because I think we made an interesting point, that would take issue with your retort.

The naive version of West's "memory allocator" (without the generational index), in the context of her game, would have also had undefined behavior (in the game world). The naive system still defeats the borrow checker, but you can still end up in a situation where you try to deference something that no longer exists, and worse still since the object that lives there was the same type as before, your state corruption is even more silent. This necessitates the need for a generational index, however West only knew to use a generational index because she is an experienced game dev, not because the borrow checker told her to.

For Blow (who hasn't written rust), believes the borrow checker (and/or language) should prevent these kinds of logical bugs in his game code and bypassing it in this way effectively turns it off ("entity safety"), while the Rust borrow checker only really guarantees memory safety.

His final argument is then, since the borrow checker doesn't provide "entity safety", it impedes on games development because Blow (and any other modern game developer) would have been smart enough to start with a proper ECS system anyways and the borrow checker wouldn't have bought him anything. This final argument is where I disagree with Blow, but 1.) I don't think any programmer is smart enough 100% of the time however I will concede he comes from a different world (Game Dev) which has stricter deadlines than most other industries so he may be more sensitive to tools like the Borrow checker. 2.) Something I've noticed with Go as well is when the language developers tend to say something like "you can't use this toy because you will shoot yourself", it becomes a personal attack on developer ego rather than a nuanced trade off on system stability.

⬐ kibwen

> His final argument is then, since the borrow checker doesn't provide "entity safety", it impedes on games development because Blow (and any other modern game developer) would have been smart enough to start with a proper ECS system anyways and the borrow checker wouldn't have bought him anything.

But even in C++ gamedev using a proper ECS, one is still using plenty of plain-old references all over the place for things unrelated to the world state, no? If so, then saying "the borrow checker doesn't help manage world state" doesn't imply that the borrow checker doesn't benefit the codebase in other places, especially considering that use of an ECS circumvents the places where we have determined that references (and hence the borrow checker) are already poorly suited to model.

⬐ steveklabnik

> would have also had undefined behavior (in the game world).

"Undefined behavior" is a term of art in programming languages. She would have a logic error, but not UB. See my comment over here: https://news.ycombinator.com/item?id=17995007