HN Theater

The best talks and videos of Hacker News.

Hacker News Comments on
Spec-ulation Keynote - Rich Hickey

ClojureTV · Youtube · 293 HN points · 45 HN comments
HN Theater has aggregated all Hacker News stories and comments that mention ClojureTV's video "Spec-ulation Keynote - Rich Hickey".
HN Theater Rankings
  • Ranked #27 all time · view

Hacker News Stories and Comments

All the comments and stories posted to Hacker News that reference this video.
Obligatory Reference to Rich Hickeys talk on versioning and interface specs:

https://www.youtube.com/watch?v=oyLBGkS5ICk

Discussion:

https://news.ycombinator.com/item?id=13085952

Oct 30, 2019 · finalfantasia on Thank You, Guido
To anyone who's interested in this topic, I strongly recommend this talk from Rich Hickey about "giving something to somebody that they can use and making a commitment": Spec-ulation (https://www.youtube.com/watch?v=oyLBGkS5ICk)
Rich Hickey explains a similar thinking in this talk https://youtu.be/oyLBGkS5ICk but it’s more directed at code libraries. The same arguments apply.

I 100% think this is the way to go for internal APIs.

For public APIs I may change my opinion here for more sociological/political reasons. The rational/technical reasons all still remain.

chrisweekly
Thanks! Helpful link, and args simply / well-put. :)
You're looking at this Python code and blaming its deficiencies on lack of types. It's deficiencies are not lack of types, but lack of separation of stable vs volatile code, lack of contract specification and documentation, lack of refactoring and versioning idioms (see https://www.youtube.com/watch?v=oyLBGkS5ICk).

Why does the blame always go to lack of types?

Whether or not you have types you're still going to suffer if you don't have the other things I mention. And if you have the other things I mention then static typing become much more of a nuisance. Ergo...

> What happens if you discover a security vulnerability in your stdlib?

As the user of the stdlib, it really depends on what the security issue is. For most users, if they can verify they're not using module X, then there's no problem and no rush for them to upgrade. In a language like Python that's very dynamic, you also might be able to download a hotfix .py file provided by the language maintainers (or any other party) as an alternative to upgrading, or as something for the interim while you wait for the upgrade to get released, backported and released for older major versions, and available for the deploy environment -- this might even take the form of an iptables rule, depending on the issue. In short my point is that there are alternatives besides bumping one release's release version and hoping people upgrade.

> Except, users don't upgrade stuff.

Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.

For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.

There's also another party of users worth mentioning, those that don't upgrade specific things because past upgrades have been a terrible experience. This is a reputation issue.

> And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.

Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.

What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.

You might be interested in this talk that makes the case to stop breaking your API especially if you're requiring more and providing less. https://www.youtube.com/watch?v=oyLBGkS5ICk

_hardwaregeek
> Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.

Sure, some people definitely upgrade. But there's a big difference between requiring someone to actively hear and heed your warning and having them automatically upgrade whenever they create a newer project. Or even just prompting them to upgrade like NPM does for package vulnerabilities.

> For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.

That's not the point. If your standard library is a package, any new project will automatically download the latest version of the standard library. If I have a vulnerability in an npm package, I simply push a new version and people will automatically download that new version when they create a new project. If there is a vulnerability in the Python standard library, users will continue to have that vulnerability until they actively patch it.

> Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.

Not a canonical one. They're starting to add a new promise based API, but now we have two different APIs doing the same thing. What if we come up with a newer way to do async? Do we add yet another API?

> What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.

That's a bit of a straw man. With packages, you can still support multiple versions. It's just that fs would no longer be tied to Node. You could make a newer fs package, fs 2.0 or whatever, that uses promises. The older one would still be supported, of course, but it wouldn't be stuck in Node.

Jach
Wouldn't new projects also be using the latest stable released version of the language (and hence standard lib) that's shipped with their distro or wherever they're getting it from, especially in the deployment environment? Or at least I think they'd be as likely to do so as getting the newest versions of libs from the lib repository...

Unless, and maybe this is a source of disagreement, it's still the standard in Node land to specify dependencies as "whatever the latest version is, I don't care"? That approach has teeth. After you've been bitten a few times you learn to say "no more" and version pin. There are other nice benefits to version pinning too (like reproducible builds). It's pretty common in Java land, even though some people still specify "version x or higher". It's a tradeoff.

One downside is that if you don't occasionally check maven central or wherever for new versions you might miss a security issue, and if you're copying dependency specifying files over to new projects the issue will persist. This is solvable with tooling -- http://www.mojohaus.org/versions-maven-plugin/ comes to mind and I was pleasantly surprised to get an email from Github that even a toy project of mine pointed to a version of a JSON parser with a known issue.

> What if we come up with a newer way to do async? Do we add yet another API?

Sure. Especially if people were just fine using the older API. For the fs situation, putting it in its own package would be great, up until the point I have to modify years-old code that's been working just fine through upgrades until someone decided to break things... for JS that's even more intolerable since nothing in the type system forbids the function updating to accept a promise or a callback. That (as would be a new API for async) would be a non-breaking change, it only provides more and the requirements for old code are the same.

If it's possible to break up the core in a way that preserves legacy code, I'm all for it. Java's Project Jigsaw did this to an extent with its module system. Another way to help that is to separate what's standard (such as with an official ANSI or ISO standard) for the language and what a particular implementation of the standard bundles with its releases on top of the standard. I recall at one point the old IO fork of Node promised to never break "core javascript APIs", but the industry consortia that defines what that means makes me think it was more a hollow promise.

To take an example from a language that has official standards, C, across decades of versions of C compilers and standard libraries from various vendors, if my code is C89 then it compiles with compilers that support C89. People might be offended that a function like strcpy exists, but it's there, I can use it if I want, or a separate library like bstring, or maybe the compiler provides their own __builtin__strcpy_chk that I can use directly or switch between with a flag (_FORTIFY_SOURCE). This might end up being less work for the compilers, too, since they don't need a security bulletin saying "some uses of strcpy can lead to a buffer overflow, removing it in v2.1.2, update immediately to strcpy2". They can (and did) just provide a new strcpy2, made available a seamless upgrade (I believe even Node has done this with some shims for deprecated APIs) if you don't want to change the source text, and warn people about the old one.

:clap: :clap: Don't make breaking changes in public things :clap: :clap:

https://www.youtube.com/watch?v=oyLBGkS5ICk

An item about versioning is always a good time to revisit Rich Hickey's thoughts on growing software and his criticisms of semantic versioning.

https://www.youtube.com/watch?v=oyLBGkS5ICk

I prefer Rich Hickey's definition of compatibility [1]:

A function is compatible if it requires no more than it did, and provides no less than it did in the previous version. A library is compatible if it provides no fewer functions, and requires no additional context (transitive dependencies) to use it, and all of its functions are compatible as defined.

When you wish to require more, or provide less, you make a function with a new name.

[1] As linked elsewhere: https://youtu.be/oyLBGkS5ICk?t=1189

Feb 11, 2019 · jt2190 on What's next for SemVer
ruuda pointed out an excellent talk/rant by Rick Hickey about the difficulties with versioning: https://youtu.be/oyLBGkS5ICk?t=1792

(edit: change video link to start right at the ranty part about Semantic Versioning)

jt2190
Rich Hickey, creator of Clojure.
Feb 11, 2019 · ruuda on What's next for SemVer
For an interesting take on versioning, see the Spec-ulation keynote by Rich Hickey [1]. It specifically touches upon SemVer being itself versioned.

[1]: https://www.youtube.com/watch?v=oyLBGkS5ICk

joppy
It's a great talk. I can't remember whether he says it directly in the talk, but the upshot of his argument is that the "major version" part of SemVer is pretty much useless information to library consumers, and suggests that rather than using major versions, libraries should just change their names.
badsectoracula
This is an excellent talk, i found myself fully agreeing and kind of relieved that i'm not the only one thinking that you shouldn't break backwards compatibility and if you do, then rename the API. I'd only add that for libraries renaming "FOO 1" (with "1" often implied) to "FOO 2" isn't a good new name since someone else might want to pick up the slack and continue developing a "FOO 2" that is backwards compatible.

One of my favorite parts of the talk is the X.Y.Z changes in semver essentially translating (from a user perspective) to imscrewed.dontcare.dontcare :-P.

Sadly it seems that this talk has been largely ignored. I'd like everyone who provides any library, framework or anything else that other people are supposed to build on - especially on the perpetually broken Linux desktop - to watch this talk.

slowmovintarget
SemVer bit starts at https://youtu.be/oyLBGkS5ICk?t=825

Have to agree, excellent talk.

Rich Hickey made a similar (but more clear) case against semantic versioning in his Spec-ulation keynote https://www.youtube.com/watch?v=oyLBGkS5ICk

He advocates talking about "breakage" and "accretion" instead of the vague term "change".

His most controversial point, as far as I remember, was that any breaking change (i.e. a major version bump in semver) might as well be considered another product/package/lib and should better choose another name instead of pretending to be the same thing with a bumped number. For me that is taking it too far in most cases (e.g. when the intent of the software remains the same), since it would also have very disruptive effects on the community, documentation, reputation, ... of the package.

Another thought about semver: a bugfix requires only a minor version bump, but IMHO this could also be breaking if people were relying on the buggy behavior. I see the value of semver, but I guess it will always be a too-neat abstraction over something that is inherently too complex to communicate in a simple version string.

ctroein89
> Another thought about semver: a bugfix requires only a minor version bump, but IMHO this could also be breaking if people were relying on the buggy behavior. I see the value of semver, but I guess it will always be a too-neat abstraction over something that is inherently too complex to communicate in a simple version string.

I find this approach to versioning and bugs problematic, because public APIs are about the contract with the user. It's hard to work with contracts with undocumented portions. If my API says that the behavior is supposed to be one thing, and in certain cases its different, my obligation is to bring the behavior to be consistent with what my API documentation says it's supposed to do. I feel like any other position would be a slippery slope into arguing that any change in behavior needs to be considered a breaking change, and therefore there's no such thing as a patch version change.

vorpalhex
It's easy to treat all things in programming as if they're algorithms, but we should remember that there's people behind them too.

As a maintainer, it's on you to communicate effectively to your users. If you're patching buggy behavior but it's likely to break code, then a major bump may be appropriate. Likewise if your refactor is massively changing your basic contracts, then it may be wise to fork your own work and move to a new repo and name.

Semvers value is that it offers a clear enough guideline most of the time, which is about the best we can ask for.

chongli
was that any breaking change (i.e. a major version bump in semver) might as well be considered another product/package/lib and should better choose another name instead of pretending to be the same thing with a bumped number

Rich Hickey is a proponent of "hammock-driven development." He's the antithesis of the Zuckerberg "move fast and break things" culture. If you're considering a breaking change, then maybe it's a good idea to hang on to it for a while until you've got a clearer picture of what you're doing. Then maybe you can aggregate a bunch of breaking changes into a partial (or total) redesign and ship that as a new product.

Yes, sometimes you may end up with the Python 3 scenario. But I think that's the right place to have a philosophical debate, rather than between only the insiders who participate in the decision to merge the breaking change.

vincentdm
As a Clojure user I very much agree with Rich's careful way of designing software.

But in case of the Clojure language itself, I'd personally prefer a big-bang version 2.0 with significant backwards-incopatible changes to clear up some accumulated inconsistencies, over a fork into yet another language with a new name. Such a fork risks splitting the community in two, each below a critical level, and also makes the decision to upgrade more open-ended (i.e. a 2.0 release explicitly supersedes 1.x and thereby communicates that it is supposed to be better and recommended. A fork is much more open-ended and provides more arguments to part of the community to stay behind).

But there is obviously a point where the original is sufficiently unrecognizable that a new name is more fitting.

Yes, Datomic is the killer app for Clojure [^1]. Have a look at Datascript[^2] and Mozilla's Mentat[^3], which is basically an embedded Datomic in Rust.

Hickey's Spec-ulation keynote is probably his most controversial talk, but it finally swayed me toward dynamic typing for growing large systems: https://www.youtube.com/watch?v=oyLBGkS5ICk

The Clojure build ecosystem is tough. Ten years ago, I could not have wrangled Clojure with my skillset - it's a stallion. We early adopters are masochists, but we endure the pain for early advantages, like a stable JavaScript target, immutable filesets and hot-reloading way before anyone else had it.

Is it worth it? Only if it pays off. I think ClojureScript and Datomic are starting to pay off, but it's not obvious for who - certainly not very ever organisation.

React Native? I tore my hair out having to `rm -rf ./node-modules` every 2 hours to deal with breaking dependency issues.

Whenever I try to use something else (like Swift), I crawl back to Clojure for the small, consistent language. I don't think Clojure is the end-game, but a Lisp with truly immutable namespaces and data structures is probably in the future.

[^1]: In 2014 I wrote down "Why Clojure?" - http://petrustheron.com/posts/why-clojure.html [^2]: https://github.com/tonsky/datascript [^3]: https://github.com/mozilla/mentat

I don't find that too surprising, Rich tends to let things bake internally (in his head and at Cognitect) for a while before releasing anything publicly for consumption and feedback.

Most of Clojure's current development is around tooling [1] and some changes to move clojure.spec out of alpha. I believe they want to accomplish this for 1.10, but I'm not sure where I remember hearing that.

Alex Miller has mentioned that Cognitect built and continues to work on a tool that tracks all of your functions with spec definitions and generatively tests them only when the function or one of its dependencies changes. This would be extremely helpful since generative testing can be expensive to run, especially if you want to generate huge amounts of test cases. It'd be nice to say "test all of my codebase's functions 10 million times" and then have that information stored locally with the code.

In Rich's "Effective Programs" talk [2] he outlines the "10x problems" that Clojure was targeted at solving. In it there are two 10x problems that weren't solved by Clojure: resource utilization and libraries. There's not much Clojure can do to solve resource utilization as it's dependent on the host (JVM), but the library problem (especially Clojure libraries with spec definitions) is solvable (though there seems to be some disagreement on whether only allowing growth in libraries in a practical).

Pure speculation: All of their tooling projects (spec, codec, tool.deps, and the above mentioned generative test runner/tracker) are going to come together in a tool that attempts solves the library dependency problem. Rich outlines a lot of his thinking about how to accomplish this in his "Spec-ulation" talk [3]. He wants Clojure to "become the first community to make [dependencies] great". I hope that we'll see more tools and guidance from the Clojure team on this soon (and maybe improved out-of-the-box error messages)!

[1] https://github.com/clojure/tools.deps.alpha [2] https://youtu.be/2V1FtfBDsLU?t=29m53s [3] https://www.youtube.com/watch?v=oyLBGkS5ICk

puredanger
It's a journey. :) Do not expect "one grand tool" but as you've noted, all of these things are pointed in the same direction.
shaunparker
That's a better way to put it. Thanks Alex. I'm excited to see where we end up.
This is essentially the model the Go community is aiming to do with versioning standards. It's also the philosophy the linux kernel takes to e.g. syscalls.

It's something a lot of libraries could learn from. Most users want to build software for the long term and tend to avoid upgrading libraries too often.

Fwiw, this is a large part of Rich's Spec-ulation [0] talk.

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

sethev
I don't know the specifics of the Go standards but semantic versioning doesn't add anything if you don't break your consumers.
A very interesting talk by Rich Hikey (creator of Clojure) on the topic: https://m.youtube.com/watch?v=oyLBGkS5ICk

To summarize, any change that offers more (as in, more fields in the output) and/or requires less (more of the original parameters made optional) is non-breaking.

Edit: Of course, this is all under the assumption that the semantics of methods should never change. Instead of changing semantics, he proposes to just create a new method with a new name.

Rich Hickey's take on SemVer makes for a really fantastic talk.

"Change isn't a thing. It's one of two things: Growth or Breakage." Growth means the code requires less, or provides more, or has bug fixes. Breakage is the opposite; the code requires more, provides less, or does something silly like reuse a name for something completely different.

I recommend the whole talk, but the specific beef about SemVer starts here: https://youtu.be/oyLBGkS5ICk?t=1792

Jun 27, 2018 · pjmlp on Go 1.11 Beta 1 is released
I the context of versions I think Rickey puts it best, actually there are no guarantees of compatibility.

Even a minor update can eventually break someone's code that relied on the bug.

"Spec-ulation – Rich Hickey"

https://www.youtube.com/watch?v=oyLBGkS5ICk

Russ actually mentions Rich Hickey's keynote talk, Spec-ulation (https://www.youtube.com/watch?v=oyLBGkS5ICk) in one of his blog posts about vgo
Feb 20, 2018 · Jach on Go += Package Versioning
But it just delays the problem and introduces new ones...

Patch changes, don't care, minor changes, don't care, major changes, you're screwed [maybe].

https://www.youtube.com/watch?v=oyLBGkS5ICk (https://github.com/matthiasn/talk-transcripts/blob/master/Hi...)

I wasn't aware of greenkeeper, thanks. Can you give some examples of "Semver-based package managers"?

I'm curious what you think of the two posts I was basing mine on:

"Spec-ulation" by Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

"Volatile Software" by Steve Losh: http://stevelosh.com/blog/2012/04/volatile-software

I think there may partly be a "universes colliding" effect here, and partly just the future being non-uniformly distributed.

hobofan
Based on pornels' example, "Semver-based package managers" are most modern ones (npm, bundler, cargo) that have the concept of Semver embedded in the version constraints for dependencies.

E.g. the tilde (~) and caret (^) operators in npm[0] allow you to specify version constraints on dependencies, that allow them to be resolved/pinned to higher minor/patch versions of that package, but not to a higher major version since that will by the Semver definition contain breaking changes and those might impact your project.

[0]: https://docs.npmjs.com/misc/semver#tilde-ranges-123-12-1

akkartik
If that's true then Rich Hickey has refuted this approach to my mind. See my link in OP. I also mentioned npm and bundler in OP. This is precisely the approach I'm against. It's good locally, but deteriorates the eco-system at large when everyone follows it.
LTS would be a ton less necessary if people took backwards compat as seriously as some do.

Rich Hickey has a great talk here https://www.youtube.com/watch?v=oyLBGkS5ICk

That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained.

Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.

moomin
Backwards compatibility is exactly why clojure.set’s JIRA is full of won’t fixes, though.

It’s not just a cost for maintainers.

bpicolo
Yeah, for sure. I think the best takeaway for me from that talk was that major versioning is actually a kinda crummy part of dependency managers, because the way we structure them / code tends to allow you to only use one version of some library at a time. I've definitely been bit on many occasions by libraries doing entire restructures of the whole thing and then pushing under the same namespace.

A better strategy for namespacing would go a long way.

The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.

Rich Hickey has a wonderful talk about it.

"Spec-ulation" - https://www.youtube.com/watch?v=oyLBGkS5ICk

Which I fully agree with, basically there are no major, minor, patch level changes, in reality every change is a possible breaking changes, because we don't control what downstream is doing with our software.

tome
Oh well, he should try a language with types ...

/me runs ...

pjmlp
All my favorite languages are typed and I do agree with him.
tome
Really? How so? If you have abstract types then you can easily make internal changes without making those changes visible externally.
pjmlp
Some examples,

1 - Not all changes can be kept internal, for example adding a new case to an ADT

2 - Internal changes can still alter the semantic of the API, even if from the outside everything still looks the same from the type system point of view

tome
Yes, both are true. 2 is a particularly good example.
gmfawcett
As a simple example, suppose I swap out one container type with another one in the internal code. Both implement the same interface/traits, so the change isn't visible to users. But the time complexity of the container operations are different on the new type, and the user was depending upon the time characteristics of the old container. E.g., deletions are now O(n) instead of O(1). I've violated a user expectation without breaking type safety.
tome
Nice counterexample!
spraak
What are you trying to say? There are already a lot of explanations for why Clojure is dynamically typed and how it fills its role nicely that way.
tome
I know, I know. We've been having some interesting discussions about Clojure, Haskell and dynamic and static typing on these stories

https://news.ycombinator.com/item?id=15602658

https://news.ycombinator.com/item?id=15593069

My comment was intended to be a wry one referencing those very interesting discussions and hinting that if you have types then you can more easily make breaking changes internally that aren't visible externally.

nickbauman
Dubious. The maintenance argument is weak. https://www.youtube.com/watch?v=2V1FtfBDsLU&feature=youtu.be...
tome
I'm not sure what you mean. I thought it was self-evident that if your interface is an abstract type then you can change the implementation without breaking consumers. If you don't think it's true then please can you explain how? I thought the Clojure argument is not that types don't provide beneficial encapsulation, but that it's not worth paying the cost for them. Again if you know any different I'd be pleased to hear.
nickbauman
Apples and Oranges. We're talking about changing the interface. Did you watch the video segment?
tome
> We're talking about changing the interface.

Who's "we"? I'm talking about changing the implementation.

> Did you watch the video segment?

I watched the bit where he said if you add a constructor to a sum type you have to change everywhere it's pattern matched. True. I'd love to see Clojure's solution where you don't have to change any existing code!

EDIT: Oh wait, I think it's even worse than that. I think he's talking about product types. In which case you should use projection functions and indeed you need to make no changes to old code!

eternalban
> there are no major, minor, patch level changes, in reality every change is a possible breaking changes

It is "possible" that my blowing my nose will disturb butterflies in China.

Internal to the organization that produces the software, they are highly meaningful. Extended to user base that reads the documentation, it remains equally meaningful.

Downstream that resists information and can not be controlled is on its own. Good luck.

EGreg
I agree that platforms should stay backward compatible as much as they can. That said, if upstream changes, sometimes you've got to change too.

There should be a mechanism for releasing breaking new major versions. And it is this:

1) Release a new version for all your DIRECT dependents. Ideally they should be subscribed to your updates.

2) After they have had ample time to test it, they will release an update for THEIR direct dependents.

At no time should C update A if C depends on B and B depends on A. In fact, ideally, if C depends on B1, ..., Bn then C will NOT upgrade A until all B's have upgraded. C should push the B's to do it and supply patches.

This means version pinning for all dependencies, my friends!!

The alternative is "dependency hell" or making multiple copies of A to live inside each B.

If some B's are taking their sweet time then it may be better for some C to submit a patch to B's repo, which even if not accepted can be pulled by other C's. The question is who "owns" or "maintains" a B project if they don't react to changes of some of their dependencies. The answer, I think, should be - fork the B if it's not being actively maintained.

Edit: I think HN needs a "reason for downvote" field, perhaps seen only by the person being downvoted. As it is, many downvotes convey very little useful information.

cgdub
If upstream changes, pick a better upstream.
EGreg
So if a large platform like facebook changes, simply stop supporting facebook? Wouldn't that kind of make your library very limited?
majewsky
> there are no major, minor, patch level changes, in reality every change is a possible breaking changes

I noticed this when recently writing a spec that defines backwards-compatibility. There's surprisingly few things you can do when you want to stay backwards-compatible.

greggyb
It's much like an immutable data store. You get one operation: insert.

With full backward compatibility, you agglomerate features, and changes are implemented as optional, with the default being the old behavior.

You can add layers that depend on existing code, but you cannot make existing code depend on those new layers.

Sep 25, 2017 · bad_user on Project Jigsaw: Complete
> I have a hard time being too impressed because more modern languages are doing this out of the gates

The problem with Java is that it's an already established and very popular platform and it's been so for the past decade at least. When starting from scratch, it's easy to just throw it all away and start fresh, it's easy to fix mistakes that were made in the past.

The irony though is that we still have a hard time learning from history. Just look at Go. Sometimes this industry feels like Groundhog Day, the movie.

Also, no platform or language that I know of has gotten "modules" right, with Java being one of the platforms that has gotten closest to fixing the problem actually (by means of OSGi). To see why modules are still a big problem and why we won't have a fix for the foreseeable future, I invite you to watch this keynote by Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

jcadam
OSGi is a bottomless pit of pain and despair. Never again will I work on any project built on OSGi modules (well, every man has his price, but you know what I mean).
bad_user
I agree, OSGi sucks, but it's one of the few attempts at fixing the problem with compatibility breakage in dependencies.
Whenever an idea about API versioning springs into mind it is always worthwhile to watch Rich Hickey's Spec-ulation talk, https://www.youtube.com/watch?v=oyLBGkS5ICk for an alternative view of the world.
See this talk [0] about this addition to Clojure called core.spec [1]. I read the paper first, and it isn't until the end of the presentation that I understood how it was related to the feature at all. Basically, core.spec is a kind of formal verification tool designed to deal with the growth of software. It is not a type system, though it resembles one in some ways. The objective is to support comparisons between the signatures of a function at two different points and say, are these compatible? Is this a breaking change? You have to design for growth: make no assumptions about unspecified keys and always assume that later versions may return more than they used to. And so on.

If you're a fan of semver, be warned.

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

    https://news.ycombinator.com/item?id=13085952
[1] https://clojure.org/about/spec
Jun 08, 2017 · eriknstr on How to Version an API
> My point was that there is no need to anticipate such world-breaking changes with a version ID. We have the hostname for that. What you are creating is not a new version of the API, but a new system with a new brand.

I think Rich Hickey (the creator of Clojure) would agree. He said something to this effect in the "Spec-ulation" keynote at Clojure/conj 2016. https://www.youtube.com/watch?v=oyLBGkS5ICk

the very first slides talk about the deprecation and removal of old APIs. if there are any HN readers who influence go, consider Rich Hickey's argument [1] regarding deprecation and code removal.

Bottom line: he proposes that one should never take away a function/API (particularly when you are a library or programming language).

https://www.youtube.com/watch?v=oyLBGkS5ICk

edit: superficial word choice

Someone on HN posted the spec-ulation talk by Rich Hickey recently [1]. I found it really persuasive, with the fundamental point being: when you update your software in an incompatible way, you shouldn't bump the "major version" number; you should rename it.

I hope the firefox devs consider this approach. Firefox has great name recognition, but it will be a deficit if users learn that it is a name that means breakage.

1. https://www.youtube.com/watch?v=oyLBGkS5ICk

This reminds me of Rich Hickey's keynote called "Spec-ulation": https://www.youtube.com/watch?v=oyLBGkS5ICk&t=0s

He's classifying changes in library as either:

    - (backwards compatible) growth
    - breakage
He's also saying that "semantic versioning" is a recipe for breaking people's software and that if you're going to break things, then you should change the name / namespace. I agree with him.
_pmf_
It's an unfortunate side effect of the limited way some ecosystems handle multi-version dependencies; I would not blame this on semantic versioning (which clearly states which changes are to be considered breaking). With OSGi I can use one module that uses an older version and one module that uses a newer version and use these together in an application (as long as the dependencies are not exposed, which they unfortunately will be in almost all real-world code).
Chyzwar
Problem is not semantic version but .Net platform that have multiple branches of the same thing[0]. If MS was not braindead they would promote Mono to become .Net Core instead of building a new thing.

Hickey's Clojure introduce breaking changes even in minor versions[1] how this is more sensible?.

[0]https://docs.microsoft.com/en-us/dotnet/articles/standard/li... [1]https://github.com/clojure/clojure/blob/master/changes.md

bad_user
Even though its history isn't perfect, Clojure's changes are not breaking and it's one of those projects that sets a high bar for what backwards compatibility means.

Clojure code written for 1.0 (May 2009) works just fine in Clojure 1.8 and you'd find it challenging to find an item in that linked document that actually breaks compatibility.

Spec-ulation keynote by Rich Hickey at clojure/conj 2016. He discusses the worst part about semver https://www.youtube.com/watch?v=oyLBGkS5ICk
This seems inspired and perhaps even duplicated from what Rich Hickey recently said about Semantic Versioning (found here: https://www.youtube.com/watch?v=oyLBGkS5ICk)

I really enjoy how the package-manager for the Elm programming language automates semantic versioning by diffing your apis based on the type system.

I think a bit part of the real criticism of SEMVER stems from this inability to either automate it or use it correctly.

Hickey's discussion of the subject obviously has a lot more great points to it, and even when I don't agree, I can't help to enjoy listening :)

Rich Hickey's recent talk https://youtu.be/oyLBGkS5ICk?t=2790 tore into semver. Recommended.
williamle8300
He's making good points, but what alternative is he offering?
lkrubner
Create a new namespace for your new API, or your new Interface, or your new version, or whatever you want to call it. But don't delete your old namespace. If your code is someone else's dependency, you should leave the old namespace in place for those who need it.

Left unresolved is how to deal with the accumulation of old code. I assume there will eventually be automated ways of turning it into a library, so the old code no longer needs to appear in the code that you are actually working on. Some automated solution seems like the kind of thing that the Clojure crowd will come up. I think it would only take me a day to write a macro to recreate namespaces from a library. Zach Tellman has some code that does half of this (Potemkin).

nemothekid
The reasons that Rich Hickey tore into SemVer seem to be issues SemVer weren't concerned about solving and his alternatives doesn't solve the main issue SemVer was created for. The problem of "can I reasonably upgrade this library without breaking my own code, and if possible to so in an automated fashion" isn't solved by chronological versioning. If I need to upgrade my application due to a security vuln, SemVer lets me know if I can just "drop-in" the upgrade, or if I need to work more.

The problem of knowing whether 1.3 vs 3.7 was written 10 days ago or 10 years ago doesn't seem useful. Rarely has software I have written depended on how recent the library was released.

Ultimately, the talk tears into version numbers in general, but not semantic versioning - he doesn't discuss anything particular to SemVer. He could have been talking about Postgres' versioning conventions and the talk would still hold.

greenyouse
> Ultimately, the talk tears into version numbers in general, but not semantic versioning

He does talk explicitly about the trouble of making major changes in SemVer[0]. The gist of his argument was that minor changes in semver are relatively useless while major changes have a high probability of breaking your software. Major changes in semver are backwards incompatible and update the program's API in place. This leads to dowstream breakage and fear around doing upgrades.

> If I need to upgrade my application due to a security vuln, SemVer lets me know if I can just "drop-in" the upgrade, or if I need to work more.

I think the point he was trying to make was that upstream developers could change internals of a library but keep the API consistent so that downstream devs would never have to worry about scary updates. As you said with SemVer, if the security upgrade is a significant change, then you can expect breakage in the library. What he was advocating was patching issues like security vulns under the hood while keeping everything backwards compatible. Major upgrades could even add new namespaces, functions, and arguments but there's no real point to deleting or mutating old code, that just creates breakage. He wants software libraries to be immutable to take care of dependency issues and versioning to better reflect the changes made in the code.

[0]https://youtu.be/oyLBGkS5ICk?t=13m45s

https://youtu.be/oyLBGkS5ICk?t=30m6s

https://youtu.be/oyLBGkS5ICk?t=46m28s

viraptor
> As you said with SemVer, if the security upgrade is a significant change, then you can expect breakage in the library.

In practice that's very uncommon. If someone is actually doing security releases, then either they release a minimal change to supported versions, or the distributions do that for them. Actual security upgrades are normally single patches which take great care not to do any API or behaviour changes.

sethev
I thought he was advocating never introducing breaking changes.
unsoundInput
I think this talk makes his opinion on how breaking changes in libraries should be handled (in the context of the JVM ecosystem) very clear:

A) avoid breaking API changes

B) if you have to do large breaking API changes, that will probably affect alot of dependent projects, make it a new artifact / namespace / library that can live side-by-side with the old version

B is actually pretty common in large java library projects (rxjava -> rxjava 2, retrofit -> retrofit 2, dagger -> dagger ‡ all have independent maven artifacts and I think also packages) and imho this approach makes a lot of sense. It's also the more important part of this talk compared to his critique of semver.

fenomas
Isn't semver the best way to do what he's advocating, then?

I mean, it's not like people delete 1.3.0 from their packaging system when they release 2.0.0. Incrementing the major version number is semver's way of declaring a new namespace, and once declared, it lives alongside the old ones.

What is Hickey suggesting be done differently?

unsoundInput
It is about treating new versions with major API breakage as if they were completely new libraries, not as a "maybe, possibly drop-in replacement, iff we use just the right feature subset". E.g. RxJava changed their maven artifact ('io.reactivex:rxjava:1.y.z' -> 'io.reactivex.rxjava2:rxjava:2.y.z') and the package name where in which their code lives ('rx' -> 'io.reactivex'). This makes it possible for both versions to be used at the same time while transitioning, without breaking dependencies that rely on the older version and without having to bother with automatic multi-versioning in buildtools.

With that in place it is questionable what the actual advantage of semver vs a simpler monotonous versioning schema (like build number or UTC date) is.

Rich Hickey has a great talk on this: https://www.youtube.com/watch?v=oyLBGkS5ICk

Previously discussed: https://news.ycombinator.com/item?id=13085952

qwtel
Came here to say the same thing. Also related: http://hintjens.com/blog:85 "The End of Software Versions"
williamle8300
He's making good points, but what alternative is he offering?
Anderkent
@williamle8300 >He's making good points, but what alternative is he offering?

Your comment is dead, but to answer the question - the alternative proposed is to encode the contract of your library in itself, and never break it, only extend it.

If you need to change the API, you don't change behaviour or signatures of your functions while preserving the names; you just add a completely new mainspace, while letting people use the old functions if they still have reason to. You don't have to keep actively supporting the old namespaces (i.e. no new bugfixes), but just need to not break the stuff that already worked.

There's some scenarios where this is hard - for example it's a web api and supporting it is costing you $$. But for things like that devs are generally aware that what they're consuming might get shut down at some point; so doing a new api, in a new namespace, and shutting down the old one at some point is not completely out of the question.

This reminded me of "Spec-ulation" from Rich Hickey [0]

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

There was a great talk recently by Rich Hickey[0] about how change itself is breakage. Breaking early and often is catastrophically worse than python 2.7 in my opinion.

[0]: https://www.youtube.com/watch?v=oyLBGkS5ICk

nickbauman
Agreed. Hickey, in that talk, claimed if you really did break backward compatibility, you should just name it something else. So in real terms Python3 should have been renamed to Pythonidae or something like that. Eventually the last folks working in the 2.7.* branch would notice that nobody else was in the room and hopefully figure out where they went.
btrask
If you look at long-running, successful projects, they don't do this.

One example is the original V8 dev team that went off to create Dart (AIUI). Nobody followed them and V8 got replacement developers. (But perhaps that was a particularly egregious case.)

derefr
I feel like the only responsible thing to do in such cases—if you truly believe that the thing you're abandoning is just a plain-old failure of engineering, and it's effectively self-flagellation to be subjected to maintaining it—is to poison the well/salt the earth when you leave, so that people are at least forced to switch to something else, if not your replacement project.

Badly-engineered buildings get condemned, and we make it illegal to use them, even if it might benefit someone to use the building in exchange for the (likely not-very-well-considered) risk of the roof falling on them.

What sort of political/social infrastructure would it take to allow someone to condemn a codebase?

kibwen
Google has begun building "time bombs" into Chrome that will start inconveniencing users once the binaries reach a certain age without being updated (I believe this is specifically related to certificate validation, but I'm not going to look up details because I don't want to be put on a watchlist for googling "chrome time bomb" :P ).
caf
Is the watchlist for googling it worse than the watchlist for writing it?
kibwen
Nah, I was just on my phone and needed an excuse for my laziness. :P
scrollaway
https://bugs.chromium.org/p/chromium/issues/detail?id=664177
cookiecaper
I disagree. Python 2->Python 3 was not really a huge change. It seems that the Python 2 userbase is finally starting to dissolve but IMO the holdouts were mostly doing it for philosophical purposes. Renaming the project for Python 3 would've resulted in a massive decrease in useful searchability, history, etc, when really only a small handful of backward incompatible changes were at stake.

It seems the better lesson is a) keep compatibility for a LONG time; there are finally some old Windows programs that Windows 10 no longer runs; and b) when you do break compatibility, don't make a big deal out of it.

Python 3 probably would've been better if it was just a feature-heavy incremental release that didn't break compatibility. 3.2 probably should've introduced a change that discouraged but still supported the old code style (maybe print a warning any time the file didn't contain a shebang-style header like #!allow_old_style, kind of a simpler reverse-version of the future module), and then maybe 3.8 could finally remove compatibility after the old method has been deprecated and difficult and annoying for 10 years or so. Making a hard delineation between compatibility, while on the surface appearing useful, seem to have a much stronger negative effect in encouraging users to stick with "their" version out of principle than it has the planned positive effect of giving users plenty of time and notice about compatibility breakages.

This seems gradual enough that most users won't notice and there won't be a hubbub, nor an insurrection among users to shun your new iteration for making them do extra work.

Dealing with people effectively, including end users, is very difficult. It takes a lot of patience.

None
None
Which is a pity. We take backwards incompatible changes too lightly. I'm working with Scala for example and love the language, but it breaks binary compatibility between major versions because Scala's features don't map that well to the JVM's bytecode and the encoding of things like traits are fragile. Well, the community is kind of coping with it by having libraries support multiple major versions at the same time. Well, it's not much, but at least the tools have evolved to support this (e.g. SBT, IDEs, etc).

That said, if you're looking for a language that has kept backwards compatibility, that's Clojure. Rich Hickey has a recent keynote that I found very interesting: https://www.youtube.com/watch?v=oyLBGkS5ICk

I don't know how to feel about it. On one hand this means Clojure will probably never fix some obvious mistakes in its standard library, without changing its name (like what they did with ClojureScript). On the other hand, as I said, it is liberating to have code written in Clojure 1.0 still working in 1.8.

This talk [1] by Rich Hickey seems more pertinent than ever. It's a contrarian stance on semver (which, almost as much as vc itself, is lauded as "the Right way" among developers). He definitely challenged my beliefs, it's likely to be worth your time too.

https://www.youtube.com/watch?v=oyLBGkS5ICk

I'm curious to hear how folks how adopt GraphQL handle server side schema changes over time. I'm all for the idea of not breaking clients almost ever [1] but sometimes you need to coordinate between client and server logic; how do you do that if the dependency of any client can literally be your entire db schema? Perhaps there is a subset of the server side schema that is designed to be GraphQL-able and is kept very stable over time?

[1] https://www.youtube.com/watch?v=oyLBGkS5ICk

Dec 05, 2016 · Davertron on Dear JavaScript
Rich Hickey just did a great talk about this sort of thing at Clojure Conj: https://www.youtube.com/watch?v=oyLBGkS5ICk
Dec 02, 2016 · 6 points, 0 comments · submitted by ludwigvan
Dec 02, 2016 · 4 points, 0 comments · submitted by zaph0d
Dec 02, 2016 · 283 points, 72 comments · submitted by Moocar
DEADB17
I don't understand why he says semantic versioning does not work. In my experience (with NPM, not maven) it is very useful, adding meaning of intent by convention:

Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes.

I got the impression that the issue was maven not being able to handle multiple versions of the same package/artifact, not in the convention.

auganov
His core opposition is to introducing breaking changes under the same namespace. SemVer is the leading scheme that sanctifies such practice.
jjnoakes
Perhaps, but he repeatedly claimed that the minor and patch numbers conveyed no meaning, while dismissing the semver spec as a manifesto.

But if he read and understood it, he'd know those were important numbers. Maybe moreso than the major version.

Perhaps he should have argued his actual stance more, instead of the strawman stance. That put me off.

sheepmullet
From the point of view of a library consumer why should they care about the patch or minor versions at all?

Isn't later = better?

jjnoakes
Because if you start relying on something new or fixed in x.2.z of your dependency you want to make sure anyone using your code isn't using x.1.y.
sheepmullet
And doesn't automatic dependency resolution make this a non-issue for your consumer?

Edit: I.e. If you declare your own dependencies then tooling should ensure anyone who uses your code uses the same dependencies.

It doesn't work this way in Java world due to technical limitations, but it can in JS world

prodigal_erik
That limitation is healthy. If versions 1.1 and 1.2 of a class exist and I'm foolish and determined enough to use both in the same process (via multiple classloaders), Java will still ensure that I can't accidentally give a 1.1 instance to a callee expecting a 1.2 instance, or vice versa. Version mismatches at call sites fail quickly and loudly with classloading exceptions.

I think a hell of a lot of NPM packages only appear to work by accident, and over time they'll fail because of sloppiness about this.

jjnoakes
Consumers may want to use a different version.

Perhaps they want a newer one (bug fix, security fix).

Perhaps they want an older one (since another dependency was tested against an older version of the dep in question).

Semver gives you a way to decide what ranges of versions should be safe to move between in order to satisfy all of those occasionally conflicting requirements.

sheepmullet
Explicitly limiting your consumers to a specific version of another library is a breaking change. You have introduced a very specific dependency and are requiring the consumer to honour it.

By semvar rules you should be updating the major version?

jjnoakes
What?

If my library requires X version 1.2 or higher, how is it a breaking change if I don't work with version 2.0 or 1.0 or anything except 1.2 through 1.99999?

That's the whole point of software versioning, no matter what you call it (renaming things, semver, git hashes, anything). At some point you require something of someone else, and you can only use the versions of that other library which provide what you require (or more). Semver is just a way to lock those requirements into a machine-readable number scheme.

DEADB17
Wouldn't changing the MAJOR version be equivalent to changing the namespace without altering the human memorable identifier of the package?

Understanding that changes in MINOR.PATCH are backwards compatible, is the difference between NAME MAJOR.MINOR.PATCH and NEW_NAME MINOR.PATCH significant? They look to me as just two different conventions.

taeric
No. Because I don't have a way to keep the old and the new.

That is, the point is that you didn't change my use of the old. You just changed what I actually use. It may work. It may not.

This is especially egregious when I had to up versions to get some new functions, and old versions just happen to have changed.

DEADB17
I don't know about maven, but in NPM you can keep both. If you take away the limits of a particular implementation, I think that semantic versioning is a useful convention for the producer of an artifact to convey intent to the consumers.
DEADB17
Sorry @taeric, I can't reply to your post directly.

I do think that the intent of the producer is being communicated (unsafe to upgrade, safe to upgrade with new features, safe + automatic improvements).

I'm not disagreeing that "spec" adding more metadata to have better granularity and potentially reducing the amount of manual work is a good thing. But in the absence of it, "semantic versioning" is an improvement over safe and unsafe versions being indistinguishable.

taeric
No worries. In the future, you can almost always get a reply button by clicking directly on a post. (Click on the "time since post" to get to the direct link.)

I think I see the point. Yes, he is using hyperbole. However, I have found it is more accurate than not. In particular, the point that many projects feel a lot more cavalier about doing breaking changes.

taeric
That is the point. It doesn't convey intent. Outside of the "major versions could break." Which is somewhat worthless.

Consider, you are using a library that expose ten objects and functions. It is on version 1.2.8. it upgrades to 1.2.9. What do you do? You take the upgrade. Usually no questions asked.

It upgrades to 1.3.0, but only to add an eleventh function. What do you do? Probably take it, because you don't want to be behind.

It upgrades to 2.0. reason is "things have changed.". However, they kept the same function names. You think you can make the upgrade fine. Because, well they have the same names. However, you can't know, because some body thought it wise to reverse arguments of some functions. Which thankfully, is a compile time fail. What else changed, though?

MrBuddyCasino
The JVM can't handle multiple versions of the same jar, as there is only one classpath.

A depends on B and C, which in turn depend on incompatible versions of D.

Congrats, you're screwed, unless you're running in an OSGi container.

The only reason this seldom a problem in Java land is because many popular core Java libs, including the whole frickin standard library and all the official extended specs like servlet maintain backwards compatibility, and the successful core libs do too (Guava, commons-whatever).

They do what he preaches. Bam, successful platform!

taeric
Guava is actually a bad actor in this regard. To the point that it can sometimes teach people that "coding fearlessly" is an wonderful thing.

It certainly can be. Especially in monolith code bases where you can fix everything you broke.

As a platform, though, it is very frustrating.

sheepmullet
Like Rich mentioned from the point of view of a library consumer it's:

PATCH: Don't care MINOR: Don't care MAJOR: You're screwed

MAJOR is simply not granular enough and MINOR and PATCH are pointless.

Sometimes when I update to a new major version of a dependency it all just works. Other times I've got to spend weeks fixing up all the little problems.

Did you break one thing I didn't even use? Update the MAJOR version. Did you completely change the library requiring all consumers to rewrite Update the major version.

jjnoakes
Still don't see the big problem. If the major version is updated and it doesn't affect you, you have a 10 second job to do. If it does, you have a bigger job to do (or don't update).

What's the big deal?

sheepmullet
How do you know if it's a 10 second job or a bigger job?
jjnoakes
Read the release notes?

Read the code diff?

Try it?

sheepmullet
So lots of manual work. And you still don't see the issue?

Wouldn't it be good if there was some kind of automated way to know?

jjnoakes
Semver doesn't preclude automating anything, does it?
klibertp
But without semver you'd need to do this manual work even more often! Semver makes you do less manual labor, because you know that PATCH and MINOR don't require your attention. You don't know such a thing in many other versioning schemes.

Would it be better to eliminate even more manual labor? Yes, of course. But then is semver bad because it reduces manual labor?

prayerslayer
I get your overall point that it's better than nothing, but you'll have to admit semver makes promises that just don't hold up in reality:

> because you know that PATCH and MINOR don't require your attention

:)

In 99 % of cases, they don't. But you're never completely sure.

DEADB17
Being screwed would be the case if the consumer of the artifact is forced to upgrade. i.e.: if versions cannot coexist in a code base.

Otherwise I think the information that they convey is useful:

PATCH: improvement or correction that does not affect the consumers expectation (safe improvement)

MINOR: additional features that may be useful directly to the consumer or its transitive dependencies (safe to upgrade)

MAJOR: No longer safe to upgrade automatically. The consumer may need to investigate further or stay with the previous MAJOR.

In any case it is useful information being conveyed. The consumer decides how to act on it.

sheepmullet
> Being screwed would be the case if the consumer of the artifact is forced to upgrade. i.e.: if versions cannot coexist in a code base.

In theory multiple versions can co-exist in a codebase in js land.

In practice though the vast majority of js libs don't compose well with different versions.

At best I've found it can work as long as you consider them seperate and unrelated libraries (basically the version in js land is part of the namespace).

Edit: I definetly don't think it's as big of an issue in js land as in Java land because of transitive dependency handling.

gniquil
Haven't watched through the entire video, but the first part, analyzing dependency bottom up, and claiming version doesn't work, really reminds me of graphql. We've been doing traditional rest with /v1/..., /v2/.... This sucks for so many different ways. Graphql's solution of providing contract at the lowest level and just evolve the "graph" really feels a bit like what he was talking about in this video. And note that Facebook by not "provide less" or "require more" in their API is how they made their API still compatible with older clients. This talk is very interesting! (note I could be spewing BS here as I've not finished the video)
opvasger
exactly my thought, having dug into GraphQL recently and seen a number of talks about it.

Getting over the annoyance of looking at some initial functionality that is misplaced can be quite hard... It's really tempting to just get rid of those as you're bumping up the MAJOR.

anonfunction
Rich Hickey gave one of my favorite talks that I recommend to all programmers no matter which language they code in:

https://www.infoq.com/presentations/Simple-Made-Easy

taeric
The whole section on "just change the name" fits interestingly in with the way that the kernel is developed. They have been rather strict on this for a long time. A function released to the wild stays. Changes to it require a new function.
dmoney
Also part of the C4 methodology: https://rfc.zeromq.org/spec:22/C4/

* A patch that introduces new features to a Public Contract SHOULD do so using new names.

Rapzid
http://man7.org/linux/man-pages/man2/syscalls.2.html
tbatchelli
This is a worthwhile talk, albeit a bit too long. In this talk RH provides a framework to think about libraries, how they change over time and how we should be communicating those changes. He proposes a way forward in which the changes are communicated in a way that can be dealt with programmatically, a much better framework than versioning schemes (semver gets a beating in this talk!).
adwhit
Video is a high-bandwidth way of delivering information at a low bandwidth. But try watching tech videos at 1.5 speed, it makes them much more engaging. Also works for ponderous anime.
achikin
That's why I prefer youtube over the other services - you can speed it up. I'm not a native speaker, so 1.25 is the fastest I can understand at the moment.
asymmetric
Got any examples of ponderous anime? Lain? Akira? GITS?
Garlef
Dragonball. They sometimes charge up one attack over several episodes.
cygned
The planet will explode in five minutes.
faitswulff
Hah! Ponderous anime. I remember watching some shows at 4x speed. I'm not sure whether it's anime that introduced me to sped up video, but subtitles are a great help when watching videos at faster than 1x speed.
atemerev
I can't get any useful information from videos, it is so slowwwww. Even on 2x (the fastest I can watch), it is still 3 to 5 times slower than I read. Also, I can control my reading, skip the minutiae, or reread important parts or look up some reference — but I can't do that with videos or audiobooks or podcasts. I wonder how people even find it convenient.

Unfortunately, Rich Hickey really loves videos for some reason, so everything he says is lost to me.

(Yes, there are transcripts, but those are linear and unstructured.)

greenyouse
Hey, I usually take notes when reading or watching stuff. If you're willing to read through my semi-coherent org-mode notes, it summarizes the talk better than a full transcript.[0]

[0]http://pastebin.com/bmF0xc7y

atemerev
Thanks!
qwtel
I've revisited this post so many times, I know the number by now: http://hintjens.com/blog:85, "The End of Software Versions". I think it is related to, and similar to what Rich is saying.
j-pb
> Logic systems don't have "And nothing else will ever be true!".

Uuh. Closed world assumption? Everything that is not true is false. Most logic systems do have this. Prologs (not a logic system I know) cut operator even turns this statement into an operation.

I feel like Rich really gets it wrong this time. His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it. But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

The best example of this style going horribly wrong are the linux kernel file system modules. Different api versions all in use at the same time by the same code with no clear documentation on what to use when.

It's also ironic that the examples he uses to make his point namely, Unix APIs, Java and HTML, are horrible to work with especially because they either never developed good API's (looking at you unix poll), or they, like browsers, have become so bloated that nobody want to touch them with a ten foot pole. One of the reasons why it takes so long for browser standards to be adopted is that they have to be integrated with all the cruft that is accumulating there for almost three decades now.

"Man browsers are so reliable and bug free and it's great that the new standards like flexbox get widespread adoption quickly, but I just wish the website I made for my dog in 1992 was supported better." -no one ever.

Combinatorial complexity is not your friend.

I'd rather have people throw away stuff in a new major release, maybe give me some time to update like sqlite or python do, and then have me migrate to a new system where they have less maintenance cost and I profit from more consistency and reliability.

I think that Joe Armstrong has a better take on this. https://www.youtube.com/watch?v=lKXe3HUG2l4

Also even though I'm a fulltime Clojure dev, I would take Elms semantic versioning that is guaranteed through static type analysis anytime over spec's "we probably grew it in a consistent way" handwaving.

joe-user
> His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it. But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

It might actually be less effort to follow his method. You may need to create a new namespace or create a new function, but then you don't need to put out breaking change notices, handle user issues due to breaking changes, etc.

> "Man browsers are so reliable and bug free and it's great that the new standards like flexbox get widespread adoption quickly, but I just wish the website I made for my dog in 1992 was supported better." -no one ever.

It's not about better support for your 1992 website, it's about it still being accessible at all. Perhaps you've never had to deal with a file in a legacy format (ahem, Word) that had value to you, but was unrecognized in newer versions of software, but I can assure you that it's thoroughly frustrating.

> Also even though I'm a fulltime Clojure dev, I would take Elms semantic versioning that is guaranteed through static type analysis anytime over spec's "we probably grew it in a consistent way" handwaving.

An Elm function `foo :: int -> int` that used to increment and now acts as the identity function is merely statically-typed "we probably grew it in a consistent way" hand-waving, which may be worse than the alternative given the amount of trust people put into types.

pron
> But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

What percentage of the total software produced and used fits that and should discussions of general good practices address this anomaly? Obviously, if you work outside the normal industry practices, then industry best-practices don't apply to you. I don't think you should take what he says too literally to mean that every line of code you should write must obey this and there are no exceptions.

> that nobody want to touch them with a ten foot pole.

If by nobody you mean almost everyone. Those happen to be the most successful software platforms in history, and most new software outside embedded (with the big exception of Windows technologies, maybe) uses at least one of those platforms to this day.

> One of the reasons why it takes so long for browser standards to be adopted is that they have to be integrated with all the cruft that is accumulating there for almost three decades now.

So what? Look, large software gets rewritten or replaced (i.e. people will use something else) every 20 years on average. If your tools and practices are not meant to be maintained for 20 years, then one of the following must be true 1. they are not meant for large, serious software, 2. you are not aware of the realities of software production and use, or 3. you are aware, but believe that, unlike many who tried before, you will actually succeed in changing them. Given the current reality of software, backward compatibility is just more important to more people than agility.

> Combinatorial complexity is not your friend.

Every additional requirement results in increased the complexity, and backward compatibility is just one more, one that seems necessary for really successful software (especially on the server side).

> "we probably grew it in a consistent way" handwaving.

Why do you think it's handwaving? The spec is the spec, and if you conform with the spec then that's exactly what you want.

j-pb
When stating that nobody wants to touch browsers with a ten foot pole, I meant that nobody wants to contribute to browser development unless they are paid very very well.
freshhawk
> His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it

This is generally his focus, on big professional software. I'm also a Clojure dev and I'm on the other side of the fence on this one as well so sometimes I'm disappointed in the enterprise focus but I knew what I was getting into and it is still worth it. Am I crazy to think that maybe other lisps would have done better if they had demanded less purity?

Same with the examples of Unix APIs, Java and HTML. Sure, they are all bloated and horrible to work with. They are also massively, insanely successful. I think they are great examples because at that scale it's impressive that they work at all.

This is part of the pragmatism that makes Clojure great, they generally stay away from trying to solve the problem of huge projects being unwieldy and ugly and painful and instead they accept it as a given and work on tools to mitigate the problem. For a lot of people backwards compatibility isn't a choice, it's a requirement set in stone. Even though it always causes everyone to pull their hair out in frustration.

One day maybe one of these other research languages or experiments will find an answer to this, and prove that it works at scale. I will celebrate more than most.

kazinator
Almost everything is mutable in Common Lisp: lexical variables, objects, global function bindings. Code compiles to native and you can deliver a single binary executable with no dependencies.
junke
> Am I crazy to think that maybe other lisps would have done better if they had demanded less purity?

People associate Common Lisp with many different things, but not purity.

DigitalJack
The primary advantage of clojure to my mind are the lovely data structures and the interaction with them.

Common Lisp and scheme can implementation them, and through reader macros interact in probably the same ways, but it would always be second or third class.

Second big deal for clojure is the ecosystem.

I'd love a native language like clojure that was driven by a language specification.

ozten
As I understand it, one of the original problems Go was designed to solve was long compile times at Google.

This talk is fascinating and points out a possible solution space for real world problems in giant codebases and build infrastructures.

What if 80% of your dependency graph can be identified as dead code paths at build time and not require you to actually take dependencies on all that dead code?

_halgari
Sadly then Go went and shot themselves in the foot. Originally (I assumed this hasn't changed), you specified dependencies via git URLs. Sounds great, but then they went and said every dependency always used the head of master. So now you're in the horrific situation of your deps suddenly changing whenever your build decides to pull the latest from the git repos.

(Note: I don't use go, but this was the situation about 3 years ago when I last looked).

NightMKoder
One of the things Rich talks about in this talk is semantic versioning is fundamentally broken. In fact any versioning scheme that allows removal is broken. In some sense this model encourages Rich's ideas better - since you can't version directly, you either make a new library or maintain backwards compatibility.

Of course the current 'go get' model has a lot of downsides, e.g. Non deterministic builds. I still think it's worth considering building on, rather than trying to fix semver. All that's really missing is a stable monotonic identifier - something as simple as git tree height may be enough.

_halgari
Absolutely, I think the "perfect world" would be adopting some of the ideas Rich proposes but adding (git url + sha) as the dependency management system.

Of course for that to work well you'd also need to protect yourself from people deleting their git hub repos, or rewriting git history.

robto
Which is a great use case for something like ipfs, I think.
anonfunction
I've always found this to be overlooked or at least a misguided design decision. IMO would be very helpful to be able to specify a commit, tag or branch to pull down.

The Go team's solution to this problem was allowing vendored dependencies. Basically you can put the dependencies inside your project in a directory called vendor which will be used instead of whatever is in your $GOPATH/src/. This allows you to pin deps to a specific version and not need to pull deps from the internet at all.

eternalban
I was surprised by this. (Rich is rigorously perceptive and characteristically peels conflations apart.) Here, he seems to be conflating linker responsibilities with (unit-module) dependency management concerns.

If deployment-unit-A depends on du-B, but system-language level dependencies have additional subtleties, then that is fine, since someone like RH could be looking into code stripping of unreachable/dead-code on linking, and I'd be completely surprised if that hasn't been already looked at extensively to date.

Yes, you would sacrifice some transfer bandwidth/time on artifact acquisition but keep conceptual models local and their implementations simpler. Or, looking at it from another point of view, OP should consider that a 'dependency' is maximally the provider of objects in context of its namespace.

greenyouse
When I was listening to the part about breakage vs accretion, I kept thinking back to the Nix pkg manager from NixOS. It does a pretty good job of separating out the artifacts by indexing them by version number and a hash of the username (i.e. artifact-name = sha($user) + $project-name + $semver). There have to be other examples of pkg managers that do this kind of thing too...
sgentle
I think there is a missing piece to this conversation, which is about the definition of namespaces. It's easy to conflate "problems with semver" and "problems with semver+the way my package manager handles versioning".

In npm, or with any package manager that allows multiple independent versions to coexist, the version becomes part of the namespace. Multiple different versions of a package can be installed at the same time and referenced from each other. It's no problem if you depend on B1.0 and C1.0, and they depend on D1.0 and D2.0 respectively; you just get all four installed.

In Rich Hickey's vision, we would move the compatibility contract into the package name, so if we want to make a new version 2 of helloworld that does not obey the contract of version 1, we call it helloworld2. In practical terms, this is no different from the above, it's just that we add the version number to the namespace through the package name rather than the package version.

Not surprisingly, this latter strategy is already used by most systems where versions aren't part of the namespace. Debian has python2 and python3. Python has urllib and urllib2 (and urllib3). Unix has mmap and mmap2. It works fine, but it's a bit of a hack.

If your version is part of the package namespace, semver works fine and solves the problem it was intended to solve. If, like in Java, every version has to share the same name, then I agree that semver doesn't really buy you that much. After all, it doesn't help you much to know that your unresolvable dependency tree is semantic.

It's not a coincidence that npm handles transitive dependencies in the way it does; it's the whole point. I'm sure it makes npm much more unwieldy to allow multiple different package versions, but having used it it is exactly one of those "solves a problem you didn't realise you had" moments.

With that said, I definitely agree that more careful API design would allow far fewer backwards-incompatible (ie semver-major) changes, and more tightly defining compatibility (like with this requires/provides concept) is a good way to do that. Many web APIs already say things like "we will return an object with at least these fields, and maybe more if we feel like it", which I think is a good example to follow.

As far as changing package names, there's also an interesting question, not about how computers deal with those package names, but how we do. After all, why helloworld and helloworld2? Why not give it another name entirely? Part of that is because you want to make the connection between these two packages explicit. But if you change your package dramatically, perhaps that connection is not doing a service to your users.

One of the biggest complaints about major rewrites is "why does it have to be the new way? I liked it the old way!" If you genuinely believe the new way is better, why not let it compete on an even playing field under its own name? (And, if you know your new package can't compete without backwards compatibility, maybe don't make a new package.)

marysimpson
Ever require the services of a hacker with discretion and top servicing, i implore you to try your very best to hire only professionals. It will increase your chances of getting your job completed. i was able to hire the services of an elite, asides the fact that i was provided a permanent solution to the service that was rendered me but gave a very efficient customer experience. gmail- pauleta.steelbreaker. M-+19283233115 (text only)
programnature
This is a very thick talk, one Rich's best ever IMHO.

The first point is how we talk about 'change' in software, to center around what things 'provide' and 'require'.

Breaking changes are changes that cause code to require more or provide less. Never do that, never need to do that. Good changes are in the realm of providing more or requiring less.

There is a detailed discussion about the different 'levels' - from functions, to packages, artifacts, and runtimes, which he views as multiple instances of the same problem. Even though we now have spec, theres a lot of work to leverage it across all those different layers to make specific statements about what is provided and required.

michaelfeathers
That's Postel's Law.
mshenfield
I found value in dissecting the different levels of change. For the sake of sanity though, we should do breaking changes. Breaking changes exist because we have limited capacity as individuals and an industry to maintain software. This is especially true for infrastructure that is supported by (limited) corporate sponsorship and volunteers. Breaking changes limit our window of focus to two or three snapshots of code, instead of having our window of focus grow without bound. Our limited capacity can still be effective as a library changes over time.

The most important point of this talk is here: "You cannot ignore [compatibility] and have something that is going to endure, and people are going to value" [0]. Breaking changes provide a benefit for library developers, but it is usually damage done to end users. As consumers we should weigh the cost of keeping up with breaking changes against the quality of a tool, and the extra capacity its developers are likely to have.

[0] https://youtu.be/oyLBGkS5ICk?t=4177

solussd
Additionally there are safer, usually reasonable, ways to deal with what would otherwise be breaking changes. Give the changed functionality a different name, create a new namespace/module without the removed functionality, or create a new library if you have introduced something fundamentally different (e.g., w.r.t. how you interact with it). That way your users can choose to refactor their code to use the change, rather than discover their expectations no longer match reality when they upgrade.
blueprint
Who says you have to maintain old code? We're talking about simply not deleting it and establishing a discrete semantic for the new version as truthfully, a new version is new content which demands a new name to accurately and precisely describe it. If it didn't it would be like saying different content doesn't produce a different hash.
wtetzner
Except that naming things is one of the hard problems. I don't see why a major version bump can't be considered a different library.

I guess you can use version numbers in the name instead, since this talk is specifically targeting maven artifacts.

blueprint
Hickey's other talk says hard is relative, and I happen to agree, especially when it comes to naming. The question is to what degree of exactness you can confirm what exists (in problems). That is a function of your degree of truthfulness. So it's "hard" only in the sense it's hard to approach 100% truthfulness. However, I have observed that one doesn't need 100%, one needs to be beyond a certain threshold of effective sufficiency. And according to human history, special, rare individuals are born who do exceed that threshold.
mshenfield
You're right, there is no obligation to maintain it. I think that misses the point though. The value in keeping the code is to allow the end user to continue to enjoy improvements in parts of the library that don't have breaking changes without upgrading those that do. You could continue to have security patches installed, for example. That value is much less when you don't do basic maintenance implement bug fixes and security patches.
blueprint
Unless I'm missing something… the answer to that problem is to (a) factor the code sufficiently to then (b) create an abstraction (interface) that backs out the concrete implementation to the specifically desired version/functionality.
michaelfeathers
Agreed. Breaking changes can lead to alienation of user base, but I think there's a danger in lulling people into expecting that kind of constancy in software. It creates dependency of another kind. Maybe the trick is to vary features at some rate, getting users used to change and bringing them along.

In retail it used to be the case that you could go to the same store a month later and see the same shirt to buy. The Sears catalog [1] presented that sort of constancy for consumers. Today there's a lot of flux. Some of it actually engineered to prevent people from delaying purchasing decisions. In software we can and do introduce breaking changes for ease of maintenance, and that can be ok as long as people are used to it. It's making the choice to have a living ecosystem.

[1] http://www.searsarchives.com/catalogs/history.htm

HN Theater is an independent project and is not operated by Y Combinator or any of the video hosting platforms linked to on this site.
~ [email protected]
;laksdfhjdhksalkfj more things
yahnd.com ~ Privacy Policy ~
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.