HN Theater @HNTheaterMonth

I'm not sure why don't we send diff(url) along with version in the header (if you have ever watched this https://www.youtube.com/watch?v=oyLBGkS5ICk Rich Hickey's talk)

Diff format example: {added: [{"path-to-key": {..}}], removed: [], changed: [{"a/b/c": {prev: {type: "string"}, current: {type: "number"}}], moved: [{old: stuff1, new: stuff2}]}

> See, the scope and purpose of something changes faster than its name can.

There's your problem. Create libraries that aim to do one thing and do it well. Once you release your project and have users that depend on your code, you owe it to them to maintain it in the original scope. If you have an urge to change your project's scope so much that the name should change, create a new library with a better name instead.

p.s. Obviously does not apply to brand names, which need to be distinctive.

p.p.s. Rich Hickey talks about this here: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ another-dave

Also comes back to the thing of "products end up modelling the organisational structure".

If it's painless to create new internal products within your company & there is operational support for it, you're more likely to spin up a sister "EmailNotificationService" alongside "SmsNotificationService".

If you have to jump through hoops to get sign-off and business cases etc and eitherways it's the same team responsible for both, people are more likely to say "let's just put the functionality in here"

⬐ habitue

> Create libraries that aim to do one thing and do it well.

This is a thing people like to repeat over and over again, but just really isn't actionable. The world is very high dimensional, it's always a judgement call what to cluster together and call "one thing". Especially in software where everything is made up and is constantly changing.

People often like to refer to things like unix tools etc as doing one thing and doing it well, but all those tools have grown a ton of command line args, and have backwards compatibility issues that prevent them from doing their job "well" in the modern era. So people write new tools or just hardcodes a ton of command line flags into aliases by default. Many default settings on unix tools are awful. Why does grep not use regex by default? Why doesn't echo support newline escapes by default? Which tools use color by default?

Reality is a mess, and it's an illusion that you can do "one thing"

⬐ etothepii

Aim is the problem here. Many projects are built and maintained by people who have a use case.

It may well fit that use case to add one extra feature to the library rather than creating a new library.

I suspect that many of the examples provided (such as the Django project) evolved over time. It can also often be the case that once an implementation is done it becomes obvious that it actually does something very different to what you originally set out to do.

⬐ JosephGuerra

In theory, it makes sense that you would just create a new library with a better name, but if the original product has been on GitHub long enough to have amassed followers and a significant number of stars, they may be hesitant to have to ask those contributors to re-add the new repo.

⬐ jayd16

Exactly. Don't just cram functionality into the closest thing at hand without any thought and you won't have to worry about scope creep ruining the name of that thing.

If you want to make new and different things, just do that.

⬐ hulitu

like libcaca. Caca means sh*t in some languages.

⬐ Vinnl

> Create libraries that aim to do one thing and do it well.

That's nice and all, but once you combine that with giving them descriptive names, you're going to have a list of hard-to-distinguish projects like:

- css-min

- css-polyfill

- css-inline-min

- css-inline-polyfill

- css-script-min

- css-script-polyfill

- etc.

Good luck clarifying what you're talking about. And of course, people will start using acronyms for all of them, and now you're back at cryptical names, with the added benefit of all the names sounding and looking similar. Or, of course, have inaccurate or inconsistent names.

And then I didn't even get to the part where your categorisation turns out to not match the eventual scope, and they'll need to be split into css-server-min, css-client-min, etc.

⬐ dmitriid

> you're going to have a list of hard-to-distinguish projects like:

What's hard to distinguish? These are projects that do something with CSS. If you don't need css, you skip them. If you need a polyfill, you only look at polyfill ones to see what you need.

Now, let's imagine your list is "creatively" named like

- shelob

- arcana

- custard

- escalope

- virtuocsso

- jester-css

- etc.

Ah yes, this is so much easier to distinguish, isn't it?

⬐ Vinnl

They're hard to distinguish in the sense that a colleague might ask something like "hey can you publish a new version of css-script-min" and you accidentally publish a new version of "css-min" or "css-inline-min" or whatever instead.

⬐ dmitriid

That's such a minuscule problem compared to trying to figure out which of the insane names is the one you need.

⬐ woojoo666

A name is usually not enough to determine which library/framework is best for you. This is the _illusion of transparency_ that the author talks about. How would you rename React and Angular such that developers could tell at a glance which one to use? If it's going to require further investigation from the user's part, better to give the project an opaque name to make it clear that further investigation is necessary.

⬐ dmitriid

when I'm looking for a css minifier, I sure as hell don't want to wade through weirdly names projects that "require further investigation".

And definitiely not inside the company I work at

Edit "And I will never know love because of Galactus" https://www.youtube.com/watch?v=y8OnoxKotPQ

⬐ lelandfe

And then someone makes a package to combine all of them, and then someone makes a package that combines that combining package with another combining package, and then the author of `css-script-polyfill` deletes the repo, and we're dealing with another leftpad

https://qz.com/646467/how-one-programmer-broke-the-internet-...

⬐ TkTech

I can't be the only one that just includes any dependency with a permissive license? `git subtree` is so ridiculously easy for the larger dependencies and 90% of these projects are sub-500 lines in a single file, just include them.

As Ritch Hickey puts it semantic version is meaningless in practice.

https://youtu.be/oyLBGkS5ICk

TLA+ [1] and Idris come to mind. Also Clojure Spec [2] For the specific case you mention, "taint analysis" would be appropriate.

[1] https://softwareengineeringdaily.com/2018/11/09/tla-with-les... [2] https://www.youtube.com/watch?v=oyLBGkS5ICk Edit: links

> Yes they've invented that: It's called bumping the major version.

Rich Hickey's Talk:

https://www.youtube.com/watch?v=oyLBGkS5ICk

explains why this is not a solution. It makes a difference, yes. But it is the difference between "the incompatible changes in my library are going to break your application" and "the incompatible changes in my library are going to break your application, and I am telling you this beforehand".

> There has to be some kind of release valve for software to evolve and break backwards compatibility.

You kinda insinuate that breaking backwards compatibility is kinda necessary at times.

This is not the case. Projects like

* the Linux kernel, or

* the GNU C library, or

* the Numeric -> Numpy transition around Python 2.0, or

* Common Lisp (which is much older than Python) adopting Unicode

are good examples that this is not necessary. It is not true that you have to break backward compatibility.

There are domains where breaking backwards compatibility in libraries is not acceptable at all, like vendor libraries in industrial automation. You don't throw away a 15-year old printing machine or a chemical plant just because the vendor of the automation software is tired of supporting its old interfaces.

That might sound strong, but others have expressed it more strongly. Read this: https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

How it is done? It starts with well-designed interfaces. And when interfaces are changed, the old interfaces are kept and become special cases of the new ones. Numeric/Numpy is a good example.

Here is a talk, brilliant as always, by Rich Hickey which explains why and how:

https://www.youtube.com/watch?v=oyLBGkS5ICk

It is highly relevant to Nix and Guix.

Python3 could have gone the same way - keeping the interpreter compatible to Python2 code, making the semantics dependent on whether a source file has a *.py or a *.py3 extension, and so on. It would have been more work but the transition would have been nearly painless, and I guess much faster. Support for old stuff does not need to go on forever - for example, Linux does not support any more Intel 386 CPUs.

It boils down to whether keeping stuff backwards-compatible is a goal of the project leaders or not.

⬐ lamontcg

The problem with most open source software that is in package managers is that it is usually done by one person. It isn't started by someone with a decade of interface design, it is often their first large important project, they DONT do the well designed interfaces because they haven't made the mistakes in interface design yet which they'll eventually learn from. And then when it comes to backwards compatibility it is cheap for you to say they should just support their old interfaces forever, but that has a cost and creates more friction going forwards for the projected. When it is one person working on open source who isn't getting paid, that is all somewhat unreasonable to expect and you just won't ever get it. In your world what you'll wind up with instead of back compat breaking changes is just abandoned and rotting software as maintainers give up.

You could do this by arguing that languages need very thick and well-designed standard libraries, which means that hopefully there's a large business supporting the library and there are teams of reasonably well paid software engineers who are doing the design work up front for everything. You should be explicit about that though.

I'm kind of not surprised that you cite one of Linus' asshole rants to LKML as well. Try screaming that at a single-person open source maintainer and watch them decide it just isn't worth it any more and quit on the project entirely.

If you want that then don't use anything outside of your language's standard library and don't use package managers and contributed source code at all. Write everything else yourself, no dependencies, no worries about backwards compatibility breaks.

⬐ jnxx

I did not say that hobbyist packages which are used by few people and are unstable experiments should be kept stable at all costs.

But you see the linked discussion about stability in Nix is about packages like opencv, pillow, boost, pytorch, tensorflow, kubernetes, and I would expect them to behave professional.

And as said, as long too few people actually respect semver, it is pointless to suggest to use it, especially if the authors of a package do not know what a breaking change is, do not know how to avoid them to happen, and do not have a documented and specified API in some way. If you don't have an API, you can't use semver.

Although not git-tag specific, if you want to see a great rant on semantic versioning, I can highly recommend Rich Hickey's.

https://www.youtube.com/watch?v=oyLBGkS5ICk&t=30m

I was really surprised Rich Hickey's talk wasn't linked to: https://youtu.be/oyLBGkS5ICk

There's a really good Rich Hickey talk about versioning. Agree or not, it's a great prompt for considering what you want from versioning.

https://www.youtube.com/watch?v=oyLBGkS5ICk

Reminds me of the talk Rich Hickey gave about Clojure Spec and why semver fails.

https://youtu.be/oyLBGkS5ICk

The two are not mutually exclusive. Clojure has namespaced keywords and specs[0] to cover that. (There is also the third-party malli, which takes a slightly different appproach.)

The advantage is that maps are extensible. So, you can have middleware that e.g. checks authentication and authorization, adds keys to the map, that later code can check it directly. Namespacing guarantees nobody stomps on anyone else's feet. Spec/malli and friends tell you what to expect at those keys. You can sort of do the same thing in some other programming languages, but generally you're missing one of 1) typechecking 2) namespacing 3) convenience.

[0]: spec-ulation keynote from a few years ago does a good job explaining the tradeoffs; https://www.youtube.com/watch?v=oyLBGkS5ICk

The talk for those interested: https://youtube.com/watch?v=oyLBGkS5ICk

⬐ Pet_Ant

And the transcript: https://github.com/matthiasn/talk-transcripts/blob/master/Hi...

You might be interested in reading about spec[0] and as always, a Rich Hickey video on it[1].

[0] https://clojure.org/guides/spec

[1] https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ didibus

We do use Spec heavily, we use it to define our domain data model for all data which crosses the application boundary (data which we will persist to disk, database, or data which we send/receive from other services (over APIs), or data which we take from the user or return to the user.

What we've found happens to us, is that because Clojure is dynamically typed, we actually spend more time focusing on making sure we have well specified and properly designed our data model for data which enters/leaves the application boundary, and that we use runtime validation everytime the data is about to leave the app boundary or enter it from a non-trusted source.

When we used Java and Scala, we did not spend quite as much time here, because you've already got your internal types, and you kind of take them as your data model, and rely on them for validation and specification. The issue is the types only go so far to perform validation, and you've often designed them in terms of the implementation needs, and not the domain model. A dynamically typed language like Clojure teaches you to be extra careful with data to make sure you don't mess it up, and it gives you really powerful tools for it, such as Spec and the REPL.

This turns out to help I've found in an enterprise context, because even with a statically typed language, when data enters/leaves the boundary of the application it is no longer within the purview of the type checker.

So with Clojure, we end up spending way more time thinking through our domain model and validating it. Which worked pretty well for us.

You also need to learn how to properly design an app in Clojure, and it's very different than other languages, so it's possible to overcomplicate things and shoot yourself in the foot. But if you do it right, it makes for very simple code bases that are way smaller in LOC.

Generally, we have one module that specifies our data model for the bounded context of our app. Another module to construct, validate, convert, and manipulate entities from our data model. And then we have a main module with all our APIs, each API implementation lives in its own module. Shared logic exist in a shared module. And shared state exist in another shared module.

It takes a bit of a different mindset, like knowing to ski and then trying to learn to snowboard, but for me I ended up liking it much better, and the lack of static types didn't bother me, the strong functional immutable leaning of Clojure, along with the help of the interactive REPL development style makes up for it.

You might find Unison[0] interesting. Or the more recent developments (or intents) of Clojure spec[1] and the talk Speculation[2].

[0] https://www.unisonweb.org/

[1] https://github.com/clojure/spec-alpha2

[2] https://www.youtube.com/watch?v=oyLBGkS5ICk

I had some of the same thoughts reading the article, but I could not articulate them as well as you did.

A good resource for the details about what constitutes a breaking change is Rich Hickey's talk about Clojure's Spec. [1]

[1]: https://youtube.com/watch?v=oyLBGkS5ICk

An interesting talk about versioning by Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

Made me think of Rich Hickey's talk about versioning. "Avoid breakage, accrete!" https://www.youtube.com/watch?v=oyLBGkS5ICk&t=2462s

Subtraction increases chances of breakage.

This approach increases maintenance costs (duh) but form customers PoV, it's better if something I rely on doesn't stop doing it.

You know what you could also do? Don't break backwards compatibility. I know it sounds strange (if you're coming from modern JS development) but many ecosystems actually pull that off, like the entirety of Windows, or 99% of the ecosystem in Clojure/Script.

Then if you have over the years collected a bunch of incompatible changes that you would like to do, you create a new library with a new name, and do all the changes there.

Now you have the first library absolutely never breaking compatibility, and the second one for your "I absolutely must change the API interface here" changes.

There are some good ideas around versioning here as well (by Rich Hickey): https://www.youtube.com/watch?v=oyLBGkS5ICk

> just messy because it's new. I'm sure better tooling will eventually emerge.

I don't understand this. Either a new solution is better and we try to use that, or the solution is worse and we should avoid it. If it's worse, why should we build better tooling when we can just use a good solution from the get-go?

⬐ emteycz

> I don't understand this. Either a new solution is better and we try to use that

From my perspective coming from my experience both as a component kit developer and an application developer using that kit, having one component per versioned package is good. That we live in a world where larger packages are prevalent and thus tooling forces a developer to duplicate configuration everywhere is an unfortunate coincidence that doesn't take away the benefits of separating the project into one versioned package per component.

Also consider that it might not even be because of the development tooling, but rather VCS or CI/CD related.

To add, making the version part of the namespace is the right thing to do, also because it avoids conflicts with transitive dependencies. Changing the namespace means that the same project can depend on multiple different major versions of the same library.

And that's fine, because when you break compatibility, you're actually not talking about the same library. And this makes upstream developers think twice about breaking compatibility. Accretion of features should be preferable to breakage.

It's what Rich Hickey talks about in his Spec-ulation talk: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ grey-area

Accretion of features should be preferable to breakage.

I think most people, particularly those using Go, would agree with this.

It does not follow that we should make breaking changes easier or routine, nor that we should force people to use strict semver (which is not widely used for good reasons).

I see why they've done that as it simplifies assumptions but prefer the way other package managers handle this where it is left to producers and consumers to negotiate how strict they want to be.

⬐ stubish

> Changing the namespace means that the same project can depend on multiple different major versions of the same library.

Do you actually want that? Do you actually know that the multiple versions of the dependency you happen to be using do not conflict? If the dependency is in different namespaces their locks and other globals are also in different namespaces.

⬐ rad_gruchalski

Yes! This is exactly why people do shading in Java!

⬐ bad_user

Yes, I do, because this isn't just about my source-code, but about the dependencies of dependencies.

⬐ stubish

So yes, you want that. But how about the next question? The dependencies of your dependencies may conflict with different versions of themselves. Because nobody thought 'what happens if there is another version of this code running in a different namespace'. And nobody tested it, so you are trusting to luck rather than any sort of rigor. Thankfully, the conflicts when they happen are usually fairly obvious (two copies of the connection pool built into your database driver, or a crash on startup because the run-once initialization ends up being run twice). But it could be more subtle, like multiple versions of the dependency that handles logging not sharing a mutex and your logs end up interleaved or corrupt under load.

⬐ grey-area

The two approaches to this are to force resolving dependencies (can be painful and require humans), or duplicate code (can cause bugs).

There are disadvantages to both and in particular duplicating dependencies should be seen as a short-term fix for a transition, not something you do routinely. If you do it routinely you'd see bugs due to shared locks, state, or changed data structures - it breaks assumptions about pkg globals for example, an important part of the language used extensively in the stdlib.

⬐ yxhuvud

Or even conflicts rising from competing use of state space in some other app. For example, if your database have a limited set of connections it can set up then you absolutely don't want different dependencies to handle connection pooling by themselves.

⬐ imtringued

>There are disadvantages to both and in particular duplicating dependencies should be seen as a short-term fix for a transition, not something you do routinely.

If you don't duplicate versions then you will have to wait for every single library in existence to update to the latest major version. This can take decades and still fail. Just take a look at Python 3.

⬐ FreeFull

It's not like putting the version into the namespace is necessary for this, though. Rust's Cargo will let you use multiple versions of a library too, without extra namespacing.

⬐ blablabla123

...and needs a package manager to solve NP-complete dep hell: https://research.swtch.com/version-sat

Go's package management is really well designed I think and also actually semantically relying on semver

Rich Hickey, the creator of Clojure, has a wonderful talk where he made a case against Semantic Versioning: https://www.youtube.com/watch?v=oyLBGkS5ICk

If you write a package and you use pip to install and test it, pip will install the requirements according to what is requested, by default the latest acceptable version.

If somebody else installs your package at a later time, pip will again install the latest fitting versions.

This install will be different from your install, or from other installs other people made in the meantime.

If a newer version if a dependency breaks your package because it has a backwards-incompatible change, your package won't work - and it will not show up in your testing because pip sees the existing, already installed packages and will think they are new enough. So, you'll find "it works for me".

A good examples are packages like python-opencv which break on python2 because the versioning implies they are backwards-compatible but they (or their given dependencies) use syntax which is not supported by python2.

And these things tend to snowball quickly because the number of packages which other packages depend on tends to grow exponentially, without a real upper bound.

And while there is lots of annoyment and cursing about package managers, I think a huge part of the problem is a cultural issue in the python community because people accept and create libraries with backwards-incompatible changes without marking that in the version numbers.

Obligatory link to Rich Hickey's brilliant talk on how to do it better: https://www.youtube.com/watch?v=oyLBGkS5ICk

Anyone else interested it is available at https://www.youtube.com/watch?v=oyLBGkS5ICk

Rich Hickey's talk about this is enlightening https://www.youtube.com/watch?v=oyLBGkS5ICk

Personally, I try to read in a charitable way by assuming that the person writing isn't being robotic - we may (most of us here) be techies but we're still human. I enjoy the wit of others, especially served wry.

Now, to versioning. There is no one standard for versioning. Again, we're mainly techies here, we know this (I hope). I like semver[1] (though I'm having doubts[2]), some prefer calver[3], others use versions as branding[4] (which extends beyond computing). Hence, the choice of version scheme chosen by another organisation or project has no relevance to any other project, their being in the same category or field or competing or whatever simply have no more relevance to this subject than Ford Escort MkII moving to MkIII.

If you're into recursion you might also try applying you "who gets to choose the relevance" logic beyond your own comments, it'll be enlightening.

Finally, if you're going to suggest that all X are capable of something but ignore exceptions to that to me smacks of dogmatism, and more importantly, of no help whatsoever to the person you're responding to.

If you have something helpful or insightful for me then please respond with that, else you can comment further up the thread with your thoughts about versioning and how a hardware company that has been fined for planned obsolescence via software updates[5] isn't angling for more purchases when it stops its dev environment being backwards compatible so it can support a half finished UI library.

[1] https://semver.org/

[2] https://www.youtube.com/watch?v=oyLBGkS5ICk A fascinating talk by Rich Hickey where he questions semver and the way we handle change as developers.

[3] https://calver.org/

[4] https://hbr.org/2011/05/the-best-way-to-name-your-product-20 It's interesting, they did lab tests:

> In laboratory and field studies involving hundreds of subjects, we found that when consumers see a brand-name continuation, they expect improvements to existing features. When they see a brand-name change, they expect fundamentally new features, and they perceive the product as riskier (likelier to fail or more prone to compatibility problems with previous products) but potentially more rewarding (higher in quality, more satisfying to use).

If OS X to macOS isn't a brand name change then what's the point (pun intended) of the 10?

[5] https://www.itworld.com/article/3316958/apple-and-samsung-fi... I got a free battery out of this one, perhaps I shouldn't complain? (wry humour alert;)

Edit: I managed to muck up the numbering. Is Markdown capable of off-by-one errors? I can make it happen.

I'd really recommend listening to Rich Hikey's "Spec-ulation" talk, it's about changeing (growing and breaking) APIs.

The main part starts about 20 minutes in, but the whole talk is good. https://youtu.be/oyLBGkS5ICk

I would say Clojure more than substantial returns of invested time. Not just the language itself, but also the wider approach to software composition, feature accretion vs backward compatibility, schema'd dynamic typing seem benefictial no matter what your development tools you use. Maybe check out the talks below and see if they don't enRich you as software engineer.

https://www.youtube.com/watch?v=ROor6_NGIWU The Language of the System

https://www.youtube.com/watch?v=MCZ3YgeEUPg Design, Composition, and Performance

https://www.youtube.com/watch?v=oyLBGkS5ICk Spec-ulation

https://www.youtube.com/watch?v=YR5WdGrpoug Maybe Not

If you check his remarks from Spec-ulation (https://www.youtube.com/watch?v=oyLBGkS5ICk), he also thinks just plain old libraries should not remove or change functionality, but only accrete. If you want to make a breaking change, you should change the name of the library. You can see this philosophy in practice with clojure.spec.alpha-2: https://github.com/clojure/spec-alpha2

⬐ gowld

That's SemVer with the version in the name.

Have you watched his "spec-ulation" talk? It makes a very strong case against EVER removing a feature from an API. https://www.youtube.com/watch?v=oyLBGkS5ICk

Obligatory Reference to Rich Hickeys talk on versioning and interface specs:

https://www.youtube.com/watch?v=oyLBGkS5ICk

Discussion:

https://news.ycombinator.com/item?id=13085952

To anyone who's interested in this topic, I strongly recommend this talk from Rich Hickey about "giving something to somebody that they can use and making a commitment": Spec-ulation (https://www.youtube.com/watch?v=oyLBGkS5ICk)

Rich Hickey explains a similar thinking in this talk https://youtu.be/oyLBGkS5ICk but it’s more directed at code libraries. The same arguments apply.

I 100% think this is the way to go for internal APIs.

For public APIs I may change my opinion here for more sociological/political reasons. The rational/technical reasons all still remain.

⬐ chrisweekly

Thanks! Helpful link, and args simply / well-put. :)

You're looking at this Python code and blaming its deficiencies on lack of types. It's deficiencies are not lack of types, but lack of separation of stable vs volatile code, lack of contract specification and documentation, lack of refactoring and versioning idioms (see https://www.youtube.com/watch?v=oyLBGkS5ICk).

Why does the blame always go to lack of types?

Whether or not you have types you're still going to suffer if you don't have the other things I mention. And if you have the other things I mention then static typing become much more of a nuisance. Ergo...

> What happens if you discover a security vulnerability in your stdlib?

As the user of the stdlib, it really depends on what the security issue is. For most users, if they can verify they're not using module X, then there's no problem and no rush for them to upgrade. In a language like Python that's very dynamic, you also might be able to download a hotfix .py file provided by the language maintainers (or any other party) as an alternative to upgrading, or as something for the interim while you wait for the upgrade to get released, backported and released for older major versions, and available for the deploy environment -- this might even take the form of an iptables rule, depending on the issue. In short my point is that there are alternatives besides bumping one release's release version and hoping people upgrade.

> Except, users don't upgrade stuff.

Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.

For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.

There's also another party of users worth mentioning, those that don't upgrade specific things because past upgrades have been a terrible experience. This is a reputation issue.

> And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.

Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.

What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.

You might be interested in this talk that makes the case to stop breaking your API especially if you're requiring more and providing less. https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ _hardwaregeek

> Actually users do. Not all users, but your sweeping statement isn't true of all users either. Experience and reading reports tell me that many if not most users actually upgrade, especially when there's a significant security issue and especially when upgrading is easy. 2.7.14 to 2.7.15 was a problem for no one.

Sure, some people definitely upgrade. But there's a big difference between requiring someone to actively hear and heed your warning and having them automatically upgrade whenever they create a newer project. Or even just prompting them to upgrade like NPM does for package vulnerabilities.

> For the users who don't upgrade, they're not likely to upgrade a third party / split off thing any more than the core language.

That's not the point. If your standard library is a package, any new project will automatically download the latest version of the standard library. If I have a vulnerability in an npm package, I simply push a new version and people will automatically download that new version when they create a new project. If there is a vulnerability in the Python standard library, users will continue to have that vulnerability until they actively patch it.

> Sucks for whom? I've been out of the Node ecosystem for several years, is there a pfs module that uses promises (either built-in or as an external lib)? If so users can use that if they want. Does it suck for the maintainer? I don't see how if those functions are small and don't really need any maintenance.

Not a canonical one. They're starting to add a new promise based API, but now we have two different APIs doing the same thing. What if we come up with a newer way to do async? Do we add yet another API?

> What would suck is if all the existing programs that wrote some fs-using code a long time ago and haven't had to touch it now suddenly need to go update it to satisfy people's fashions.

That's a bit of a straw man. With packages, you can still support multiple versions. It's just that fs would no longer be tied to Node. You could make a newer fs package, fs 2.0 or whatever, that uses promises. The older one would still be supported, of course, but it wouldn't be stuck in Node.

⬐ Jach

Wouldn't new projects also be using the latest stable released version of the language (and hence standard lib) that's shipped with their distro or wherever they're getting it from, especially in the deployment environment? Or at least I think they'd be as likely to do so as getting the newest versions of libs from the lib repository...

Unless, and maybe this is a source of disagreement, it's still the standard in Node land to specify dependencies as "whatever the latest version is, I don't care"? That approach has teeth. After you've been bitten a few times you learn to say "no more" and version pin. There are other nice benefits to version pinning too (like reproducible builds). It's pretty common in Java land, even though some people still specify "version x or higher". It's a tradeoff.

One downside is that if you don't occasionally check maven central or wherever for new versions you might miss a security issue, and if you're copying dependency specifying files over to new projects the issue will persist. This is solvable with tooling -- http://www.mojohaus.org/versions-maven-plugin/ comes to mind and I was pleasantly surprised to get an email from Github that even a toy project of mine pointed to a version of a JSON parser with a known issue.

> What if we come up with a newer way to do async? Do we add yet another API?

Sure. Especially if people were just fine using the older API. For the fs situation, putting it in its own package would be great, up until the point I have to modify years-old code that's been working just fine through upgrades until someone decided to break things... for JS that's even more intolerable since nothing in the type system forbids the function updating to accept a promise or a callback. That (as would be a new API for async) would be a non-breaking change, it only provides more and the requirements for old code are the same.

If it's possible to break up the core in a way that preserves legacy code, I'm all for it. Java's Project Jigsaw did this to an extent with its module system. Another way to help that is to separate what's standard (such as with an official ANSI or ISO standard) for the language and what a particular implementation of the standard bundles with its releases on top of the standard. I recall at one point the old IO fork of Node promised to never break "core javascript APIs", but the industry consortia that defines what that means makes me think it was more a hollow promise.

To take an example from a language that has official standards, C, across decades of versions of C compilers and standard libraries from various vendors, if my code is C89 then it compiles with compilers that support C89. People might be offended that a function like strcpy exists, but it's there, I can use it if I want, or a separate library like bstring, or maybe the compiler provides their own __builtin__strcpy_chk that I can use directly or switch between with a flag (_FORTIFY_SOURCE). This might end up being less work for the compilers, too, since they don't need a security bulletin saying "some uses of strcpy can lead to a buffer overflow, removing it in v2.1.2, update immediately to strcpy2". They can (and did) just provide a new strcpy2, made available a seamless upgrade (I believe even Node has done this with some shims for deprecated APIs) if you don't want to change the source text, and warn people about the old one.

:clap: :clap: Don't make breaking changes in public things :clap: :clap:

https://www.youtube.com/watch?v=oyLBGkS5ICk

An item about versioning is always a good time to revisit Rich Hickey's thoughts on growing software and his criticisms of semantic versioning.

https://www.youtube.com/watch?v=oyLBGkS5ICk

I prefer Rich Hickey's definition of compatibility [1]:

A function is compatible if it requires no more than it did, and provides no less than it did in the previous version. A library is compatible if it provides no fewer functions, and requires no additional context (transitive dependencies) to use it, and all of its functions are compatible as defined.

When you wish to require more, or provide less, you make a function with a new name.

[1] As linked elsewhere: https://youtu.be/oyLBGkS5ICk?t=1189

ruuda pointed out an excellent talk/rant by Rick Hickey about the difficulties with versioning: https://youtu.be/oyLBGkS5ICk?t=1792

(edit: change video link to start right at the ranty part about Semantic Versioning)

⬐ jt2190

Rich Hickey, creator of Clojure.

For an interesting take on versioning, see the Spec-ulation keynote by Rich Hickey [1]. It specifically touches upon SemVer being itself versioned.

[1]: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ joppy

It's a great talk. I can't remember whether he says it directly in the talk, but the upshot of his argument is that the "major version" part of SemVer is pretty much useless information to library consumers, and suggests that rather than using major versions, libraries should just change their names.

⬐ badsectoracula

This is an excellent talk, i found myself fully agreeing and kind of relieved that i'm not the only one thinking that you shouldn't break backwards compatibility and if you do, then rename the API. I'd only add that for libraries renaming "FOO 1" (with "1" often implied) to "FOO 2" isn't a good new name since someone else might want to pick up the slack and continue developing a "FOO 2" that is backwards compatible.

One of my favorite parts of the talk is the X.Y.Z changes in semver essentially translating (from a user perspective) to imscrewed.dontcare.dontcare :-P.

Sadly it seems that this talk has been largely ignored. I'd like everyone who provides any library, framework or anything else that other people are supposed to build on - especially on the perpetually broken Linux desktop - to watch this talk.

⬐ slowmovintarget

SemVer bit starts at https://youtu.be/oyLBGkS5ICk?t=825

Have to agree, excellent talk.

Rich Hickey made a similar (but more clear) case against semantic versioning in his Spec-ulation keynote https://www.youtube.com/watch?v=oyLBGkS5ICk

He advocates talking about "breakage" and "accretion" instead of the vague term "change".

His most controversial point, as far as I remember, was that any breaking change (i.e. a major version bump in semver) might as well be considered another product/package/lib and should better choose another name instead of pretending to be the same thing with a bumped number. For me that is taking it too far in most cases (e.g. when the intent of the software remains the same), since it would also have very disruptive effects on the community, documentation, reputation, ... of the package.

Another thought about semver: a bugfix requires only a minor version bump, but IMHO this could also be breaking if people were relying on the buggy behavior. I see the value of semver, but I guess it will always be a too-neat abstraction over something that is inherently too complex to communicate in a simple version string.

⬐ ctroein89

> Another thought about semver: a bugfix requires only a minor version bump, but IMHO this could also be breaking if people were relying on the buggy behavior. I see the value of semver, but I guess it will always be a too-neat abstraction over something that is inherently too complex to communicate in a simple version string.

I find this approach to versioning and bugs problematic, because public APIs are about the contract with the user. It's hard to work with contracts with undocumented portions. If my API says that the behavior is supposed to be one thing, and in certain cases its different, my obligation is to bring the behavior to be consistent with what my API documentation says it's supposed to do. I feel like any other position would be a slippery slope into arguing that any change in behavior needs to be considered a breaking change, and therefore there's no such thing as a patch version change.

⬐ vorpalhex

It's easy to treat all things in programming as if they're algorithms, but we should remember that there's people behind them too.

As a maintainer, it's on you to communicate effectively to your users. If you're patching buggy behavior but it's likely to break code, then a major bump may be appropriate. Likewise if your refactor is massively changing your basic contracts, then it may be wise to fork your own work and move to a new repo and name.

Semvers value is that it offers a clear enough guideline most of the time, which is about the best we can ask for.

⬐ chongli

was that any breaking change (i.e. a major version bump in semver) might as well be considered another product/package/lib and should better choose another name instead of pretending to be the same thing with a bumped number

Rich Hickey is a proponent of "hammock-driven development." He's the antithesis of the Zuckerberg "move fast and break things" culture. If you're considering a breaking change, then maybe it's a good idea to hang on to it for a while until you've got a clearer picture of what you're doing. Then maybe you can aggregate a bunch of breaking changes into a partial (or total) redesign and ship that as a new product.

Yes, sometimes you may end up with the Python 3 scenario. But I think that's the right place to have a philosophical debate, rather than between only the insiders who participate in the decision to merge the breaking change.

⬐ vincentdm

As a Clojure user I very much agree with Rich's careful way of designing software.

But in case of the Clojure language itself, I'd personally prefer a big-bang version 2.0 with significant backwards-incopatible changes to clear up some accumulated inconsistencies, over a fork into yet another language with a new name. Such a fork risks splitting the community in two, each below a critical level, and also makes the decision to upgrade more open-ended (i.e. a 2.0 release explicitly supersedes 1.x and thereby communicates that it is supposed to be better and recommended. A fork is much more open-ended and provides more arguments to part of the community to stay behind).

But there is obviously a point where the original is sufficiently unrecognizable that a new name is more fitting.

I think it's this one:

https://www.youtube.com/watch?v=oyLBGkS5ICk

HN Post: https://news.ycombinator.com/item?id=13085952

Yes, Datomic is the killer app for Clojure [^1]. Have a look at Datascript[^2] and Mozilla's Mentat[^3], which is basically an embedded Datomic in Rust.

Hickey's Spec-ulation keynote is probably his most controversial talk, but it finally swayed me toward dynamic typing for growing large systems: https://www.youtube.com/watch?v=oyLBGkS5ICk

The Clojure build ecosystem is tough. Ten years ago, I could not have wrangled Clojure with my skillset - it's a stallion. We early adopters are masochists, but we endure the pain for early advantages, like a stable JavaScript target, immutable filesets and hot-reloading way before anyone else had it.

Is it worth it? Only if it pays off. I think ClojureScript and Datomic are starting to pay off, but it's not obvious for who - certainly not very ever organisation.

React Native? I tore my hair out having to `rm -rf ./node-modules` every 2 hours to deal with breaking dependency issues.

Whenever I try to use something else (like Swift), I crawl back to Clojure for the small, consistent language. I don't think Clojure is the end-game, but a Lisp with truly immutable namespaces and data structures is probably in the future.

[^1]: In 2014 I wrote down "Why Clojure?" - http://petrustheron.com/posts/why-clojure.html [^2]: https://github.com/tonsky/datascript [^3]: https://github.com/mozilla/mentat

I don't find that too surprising, Rich tends to let things bake internally (in his head and at Cognitect) for a while before releasing anything publicly for consumption and feedback.

Most of Clojure's current development is around tooling [1] and some changes to move clojure.spec out of alpha. I believe they want to accomplish this for 1.10, but I'm not sure where I remember hearing that.

Alex Miller has mentioned that Cognitect built and continues to work on a tool that tracks all of your functions with spec definitions and generatively tests them only when the function or one of its dependencies changes. This would be extremely helpful since generative testing can be expensive to run, especially if you want to generate huge amounts of test cases. It'd be nice to say "test all of my codebase's functions 10 million times" and then have that information stored locally with the code.

In Rich's "Effective Programs" talk [2] he outlines the "10x problems" that Clojure was targeted at solving. In it there are two 10x problems that weren't solved by Clojure: resource utilization and libraries. There's not much Clojure can do to solve resource utilization as it's dependent on the host (JVM), but the library problem (especially Clojure libraries with spec definitions) is solvable (though there seems to be some disagreement on whether only allowing growth in libraries in a practical).

Pure speculation: All of their tooling projects (spec, codec, tool.deps, and the above mentioned generative test runner/tracker) are going to come together in a tool that attempts solves the library dependency problem. Rich outlines a lot of his thinking about how to accomplish this in his "Spec-ulation" talk [3]. He wants Clojure to "become the first community to make [dependencies] great". I hope that we'll see more tools and guidance from the Clojure team on this soon (and maybe improved out-of-the-box error messages)!

[1] https://github.com/clojure/tools.deps.alpha [2] https://youtu.be/2V1FtfBDsLU?t=29m53s [3] https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ puredanger

It's a journey. :) Do not expect "one grand tool" but as you've noted, all of these things are pointed in the same direction.

⬐ shaunparker

That's a better way to put it. Thanks Alex. I'm excited to see where we end up.

This is essentially the model the Go community is aiming to do with versioning standards. It's also the philosophy the linux kernel takes to e.g. syscalls.

It's something a lot of libraries could learn from. Most users want to build software for the long term and tend to avoid upgrading libraries too often.

Fwiw, this is a large part of Rich's Spec-ulation [0] talk.

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ sethev

I don't know the specifics of the Go standards but semantic versioning doesn't add anything if you don't break your consumers.

A very interesting talk by Rich Hikey (creator of Clojure) on the topic: https://m.youtube.com/watch?v=oyLBGkS5ICk

To summarize, any change that offers more (as in, more fields in the output) and/or requires less (more of the original parameters made optional) is non-breaking.

Edit: Of course, this is all under the assumption that the semantics of methods should never change. Instead of changing semantics, he proposes to just create a new method with a new name.

Rich Hickey's take on SemVer makes for a really fantastic talk.

"Change isn't a thing. It's one of two things: Growth or Breakage." Growth means the code requires less, or provides more, or has bug fixes. Breakage is the opposite; the code requires more, provides less, or does something silly like reuse a name for something completely different.

I recommend the whole talk, but the specific beef about SemVer starts here: https://youtu.be/oyLBGkS5ICk?t=1792

I the context of versions I think Rickey puts it best, actually there are no guarantees of compatibility.

Even a minor update can eventually break someone's code that relied on the bug.

"Spec-ulation – Rich Hickey"

https://www.youtube.com/watch?v=oyLBGkS5ICk

Russ actually mentions Rich Hickey's keynote talk, Spec-ulation (https://www.youtube.com/watch?v=oyLBGkS5ICk) in one of his blog posts about vgo

But it just delays the problem and introduces new ones...

Patch changes, don't care, minor changes, don't care, major changes, you're screwed [maybe].

https://www.youtube.com/watch?v=oyLBGkS5ICk (https://github.com/matthiasn/talk-transcripts/blob/master/Hi...)

I wasn't aware of greenkeeper, thanks. Can you give some examples of "Semver-based package managers"?

I'm curious what you think of the two posts I was basing mine on:

"Spec-ulation" by Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

"Volatile Software" by Steve Losh: http://stevelosh.com/blog/2012/04/volatile-software

I think there may partly be a "universes colliding" effect here, and partly just the future being non-uniformly distributed.

⬐ hobofan

Based on pornels' example, "Semver-based package managers" are most modern ones (npm, bundler, cargo) that have the concept of Semver embedded in the version constraints for dependencies.

E.g. the tilde (~) and caret (^) operators in npm[0] allow you to specify version constraints on dependencies, that allow them to be resolved/pinned to higher minor/patch versions of that package, but not to a higher major version since that will by the Semver definition contain breaking changes and those might impact your project.

[0]: https://docs.npmjs.com/misc/semver#tilde-ranges-123-12-1

⬐ akkartik

If that's true then Rich Hickey has refuted this approach to my mind. See my link in OP. I also mentioned npm and bundler in OP. This is precisely the approach I'm against. It's good locally, but deteriorates the eco-system at large when everyone follows it.

LTS would be a ton less necessary if people took backwards compat as seriously as some do.

Rich Hickey has a great talk here https://www.youtube.com/watch?v=oyLBGkS5ICk

That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained.

Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.

⬐ moomin

Backwards compatibility is exactly why clojure.set’s JIRA is full of won’t fixes, though.

It’s not just a cost for maintainers.

⬐ bpicolo

Yeah, for sure. I think the best takeaway for me from that talk was that major versioning is actually a kinda crummy part of dependency managers, because the way we structure them / code tends to allow you to only use one version of some library at a time. I've definitely been bit on many occasions by libraries doing entire restructures of the whole thing and then pushing under the same namespace.

A better strategy for namespacing would go a long way.

The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.

Rich Hickey has a wonderful talk about it.

"Spec-ulation" - https://www.youtube.com/watch?v=oyLBGkS5ICk

Which I fully agree with, basically there are no major, minor, patch level changes, in reality every change is a possible breaking changes, because we don't control what downstream is doing with our software.

⬐ tome

Oh well, he should try a language with types ...

/me runs ...

⬐ pjmlp

All my favorite languages are typed and I do agree with him.

⬐ tome

Really? How so? If you have abstract types then you can easily make internal changes without making those changes visible externally.

⬐ pjmlp

Some examples,

1 - Not all changes can be kept internal, for example adding a new case to an ADT

2 - Internal changes can still alter the semantic of the API, even if from the outside everything still looks the same from the type system point of view

⬐ tome

Yes, both are true. 2 is a particularly good example.

⬐ gmfawcett

As a simple example, suppose I swap out one container type with another one in the internal code. Both implement the same interface/traits, so the change isn't visible to users. But the time complexity of the container operations are different on the new type, and the user was depending upon the time characteristics of the old container. E.g., deletions are now O(n) instead of O(1). I've violated a user expectation without breaking type safety.

⬐ tome

Nice counterexample!

⬐ spraak

What are you trying to say? There are already a lot of explanations for why Clojure is dynamically typed and how it fills its role nicely that way.

⬐ tome

I know, I know. We've been having some interesting discussions about Clojure, Haskell and dynamic and static typing on these stories

https://news.ycombinator.com/item?id=15602658

https://news.ycombinator.com/item?id=15593069

My comment was intended to be a wry one referencing those very interesting discussions and hinting that if you have types then you can more easily make breaking changes internally that aren't visible externally.

⬐ nickbauman

Dubious. The maintenance argument is weak. https://www.youtube.com/watch?v=2V1FtfBDsLU&feature=youtu.be...

⬐ tome

I'm not sure what you mean. I thought it was self-evident that if your interface is an abstract type then you can change the implementation without breaking consumers. If you don't think it's true then please can you explain how? I thought the Clojure argument is not that types don't provide beneficial encapsulation, but that it's not worth paying the cost for them. Again if you know any different I'd be pleased to hear.

⬐ nickbauman

Apples and Oranges. We're talking about changing the interface. Did you watch the video segment?

⬐ tome

> We're talking about changing the interface.

Who's "we"? I'm talking about changing the implementation.

> Did you watch the video segment?

I watched the bit where he said if you add a constructor to a sum type you have to change everywhere it's pattern matched. True. I'd love to see Clojure's solution where you don't have to change any existing code!

EDIT: Oh wait, I think it's even worse than that. I think he's talking about product types. In which case you should use projection functions and indeed you need to make no changes to old code!

⬐ eternalban

> there are no major, minor, patch level changes, in reality every change is a possible breaking changes

It is "possible" that my blowing my nose will disturb butterflies in China.

Internal to the organization that produces the software, they are highly meaningful. Extended to user base that reads the documentation, it remains equally meaningful.

Downstream that resists information and can not be controlled is on its own. Good luck.

⬐ EGreg

I agree that platforms should stay backward compatible as much as they can. That said, if upstream changes, sometimes you've got to change too.

There should be a mechanism for releasing breaking new major versions. And it is this:

1) Release a new version for all your DIRECT dependents. Ideally they should be subscribed to your updates.

2) After they have had ample time to test it, they will release an update for THEIR direct dependents.

At no time should C update A if C depends on B and B depends on A. In fact, ideally, if C depends on B1, ..., Bn then C will NOT upgrade A until all B's have upgraded. C should push the B's to do it and supply patches.

This means version pinning for all dependencies, my friends!!

The alternative is "dependency hell" or making multiple copies of A to live inside each B.

If some B's are taking their sweet time then it may be better for some C to submit a patch to B's repo, which even if not accepted can be pulled by other C's. The question is who "owns" or "maintains" a B project if they don't react to changes of some of their dependencies. The answer, I think, should be - fork the B if it's not being actively maintained.

Edit: I think HN needs a "reason for downvote" field, perhaps seen only by the person being downvoted. As it is, many downvotes convey very little useful information.

⬐ cgdub

If upstream changes, pick a better upstream.

⬐ EGreg

So if a large platform like facebook changes, simply stop supporting facebook? Wouldn't that kind of make your library very limited?

⬐ majewsky

> there are no major, minor, patch level changes, in reality every change is a possible breaking changes

I noticed this when recently writing a spec that defines backwards-compatibility. There's surprisingly few things you can do when you want to stay backwards-compatible.

⬐ greggyb

It's much like an immutable data store. You get one operation: insert.

With full backward compatibility, you agglomerate features, and changes are implemented as optional, with the default being the old behavior.

You can add layers that depend on existing code, but you cannot make existing code depend on those new layers.

> I have a hard time being too impressed because more modern languages are doing this out of the gates

The problem with Java is that it's an already established and very popular platform and it's been so for the past decade at least. When starting from scratch, it's easy to just throw it all away and start fresh, it's easy to fix mistakes that were made in the past.

The irony though is that we still have a hard time learning from history. Just look at Go. Sometimes this industry feels like Groundhog Day, the movie.

Also, no platform or language that I know of has gotten "modules" right, with Java being one of the platforms that has gotten closest to fixing the problem actually (by means of OSGi). To see why modules are still a big problem and why we won't have a fix for the foreseeable future, I invite you to watch this keynote by Rich Hickey: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ jcadam

OSGi is a bottomless pit of pain and despair. Never again will I work on any project built on OSGi modules (well, every man has his price, but you know what I mean).

⬐ bad_user

I agree, OSGi sucks, but it's one of the few attempts at fixing the problem with compatibility breakage in dependencies.

Whenever an idea about API versioning springs into mind it is always worthwhile to watch Rich Hickey's Spec-ulation talk, https://www.youtube.com/watch?v=oyLBGkS5ICk for an alternative view of the world.

See this talk [0] about this addition to Clojure called core.spec [1]. I read the paper first, and it isn't until the end of the presentation that I understood how it was related to the feature at all. Basically, core.spec is a kind of formal verification tool designed to deal with the growth of software. It is not a type system, though it resembles one in some ways. The objective is to support comparisons between the signatures of a function at two different points and say, are these compatible? Is this a breaking change? You have to design for growth: make no assumptions about unspecified keys and always assume that later versions may return more than they used to. And so on.

If you're a fan of semver, be warned.

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

    https://news.ycombinator.com/item?id=13085952

[1] https://clojure.org/about/spec

> My point was that there is no need to anticipate such world-breaking changes with a version ID. We have the hostname for that. What you are creating is not a new version of the API, but a new system with a new brand.

I think Rich Hickey (the creator of Clojure) would agree. He said something to this effect in the "Spec-ulation" keynote at Clojure/conj 2016. https://www.youtube.com/watch?v=oyLBGkS5ICk

the very first slides talk about the deprecation and removal of old APIs. if there are any HN readers who influence go, consider Rich Hickey's argument [1] regarding deprecation and code removal.

Bottom line: he proposes that one should never take away a function/API (particularly when you are a library or programming language).

https://www.youtube.com/watch?v=oyLBGkS5ICk

edit: superficial word choice

Someone on HN posted the spec-ulation talk by Rich Hickey recently [1]. I found it really persuasive, with the fundamental point being: when you update your software in an incompatible way, you shouldn't bump the "major version" number; you should rename it.

I hope the firefox devs consider this approach. Firefox has great name recognition, but it will be a deficit if users learn that it is a name that means breakage.

1. https://www.youtube.com/watch?v=oyLBGkS5ICk

This reminds me of Rich Hickey's keynote called "Spec-ulation": https://www.youtube.com/watch?v=oyLBGkS5ICk&t=0s

He's classifying changes in library as either:

    - (backwards compatible) growth
    - breakage

He's also saying that "semantic versioning" is a recipe for breaking people's software and that if you're going to break things, then you should change the name / namespace. I agree with him.

⬐ _pmf_

It's an unfortunate side effect of the limited way some ecosystems handle multi-version dependencies; I would not blame this on semantic versioning (which clearly states which changes are to be considered breaking). With OSGi I can use one module that uses an older version and one module that uses a newer version and use these together in an application (as long as the dependencies are not exposed, which they unfortunately will be in almost all real-world code).

⬐ Chyzwar

Problem is not semantic version but .Net platform that have multiple branches of the same thing[0]. If MS was not braindead they would promote Mono to become .Net Core instead of building a new thing.

Hickey's Clojure introduce breaking changes even in minor versions[1] how this is more sensible?.

[0]https://docs.microsoft.com/en-us/dotnet/articles/standard/li... [1]https://github.com/clojure/clojure/blob/master/changes.md

⬐ bad_user

Even though its history isn't perfect, Clojure's changes are not breaking and it's one of those projects that sets a high bar for what backwards compatibility means.

Clojure code written for 1.0 (May 2009) works just fine in Clojure 1.8 and you'd find it challenging to find an item in that linked document that actually breaks compatibility.

Spec-ulation keynote by Rich Hickey at clojure/conj 2016. He discusses the worst part about semver https://www.youtube.com/watch?v=oyLBGkS5ICk

This seems inspired and perhaps even duplicated from what Rich Hickey recently said about Semantic Versioning (found here: https://www.youtube.com/watch?v=oyLBGkS5ICk)

I really enjoy how the package-manager for the Elm programming language automates semantic versioning by diffing your apis based on the type system.

I think a bit part of the real criticism of SEMVER stems from this inability to either automate it or use it correctly.

Hickey's discussion of the subject obviously has a lot more great points to it, and even when I don't agree, I can't help to enjoy listening :)

Rich Hickey's recent talk https://youtu.be/oyLBGkS5ICk?t=2790 tore into semver. Recommended.

⬐ williamle8300

He's making good points, but what alternative is he offering?

⬐ lkrubner

Create a new namespace for your new API, or your new Interface, or your new version, or whatever you want to call it. But don't delete your old namespace. If your code is someone else's dependency, you should leave the old namespace in place for those who need it.

Left unresolved is how to deal with the accumulation of old code. I assume there will eventually be automated ways of turning it into a library, so the old code no longer needs to appear in the code that you are actually working on. Some automated solution seems like the kind of thing that the Clojure crowd will come up. I think it would only take me a day to write a macro to recreate namespaces from a library. Zach Tellman has some code that does half of this (Potemkin).

⬐ nemothekid

The reasons that Rich Hickey tore into SemVer seem to be issues SemVer weren't concerned about solving and his alternatives doesn't solve the main issue SemVer was created for. The problem of "can I reasonably upgrade this library without breaking my own code, and if possible to so in an automated fashion" isn't solved by chronological versioning. If I need to upgrade my application due to a security vuln, SemVer lets me know if I can just "drop-in" the upgrade, or if I need to work more.

The problem of knowing whether 1.3 vs 3.7 was written 10 days ago or 10 years ago doesn't seem useful. Rarely has software I have written depended on how recent the library was released.

Ultimately, the talk tears into version numbers in general, but not semantic versioning - he doesn't discuss anything particular to SemVer. He could have been talking about Postgres' versioning conventions and the talk would still hold.

⬐ greenyouse

> Ultimately, the talk tears into version numbers in general, but not semantic versioning

He does talk explicitly about the trouble of making major changes in SemVer[0]. The gist of his argument was that minor changes in semver are relatively useless while major changes have a high probability of breaking your software. Major changes in semver are backwards incompatible and update the program's API in place. This leads to dowstream breakage and fear around doing upgrades.

> If I need to upgrade my application due to a security vuln, SemVer lets me know if I can just "drop-in" the upgrade, or if I need to work more.

I think the point he was trying to make was that upstream developers could change internals of a library but keep the API consistent so that downstream devs would never have to worry about scary updates. As you said with SemVer, if the security upgrade is a significant change, then you can expect breakage in the library. What he was advocating was patching issues like security vulns under the hood while keeping everything backwards compatible. Major upgrades could even add new namespaces, functions, and arguments but there's no real point to deleting or mutating old code, that just creates breakage. He wants software libraries to be immutable to take care of dependency issues and versioning to better reflect the changes made in the code.

[0]https://youtu.be/oyLBGkS5ICk?t=13m45s

https://youtu.be/oyLBGkS5ICk?t=30m6s

https://youtu.be/oyLBGkS5ICk?t=46m28s

⬐ viraptor

> As you said with SemVer, if the security upgrade is a significant change, then you can expect breakage in the library.

In practice that's very uncommon. If someone is actually doing security releases, then either they release a minimal change to supported versions, or the distributions do that for them. Actual security upgrades are normally single patches which take great care not to do any API or behaviour changes.

⬐ sethev

I thought he was advocating never introducing breaking changes.

⬐ unsoundInput

I think this talk makes his opinion on how breaking changes in libraries should be handled (in the context of the JVM ecosystem) very clear:

A) avoid breaking API changes

B) if you have to do large breaking API changes, that will probably affect alot of dependent projects, make it a new artifact / namespace / library that can live side-by-side with the old version

B is actually pretty common in large java library projects (rxjava -> rxjava 2, retrofit -> retrofit 2, dagger -> dagger ‡ all have independent maven artifacts and I think also packages) and imho this approach makes a lot of sense. It's also the more important part of this talk compared to his critique of semver.

⬐ fenomas

Isn't semver the best way to do what he's advocating, then?

I mean, it's not like people delete 1.3.0 from their packaging system when they release 2.0.0. Incrementing the major version number is semver's way of declaring a new namespace, and once declared, it lives alongside the old ones.

What is Hickey suggesting be done differently?

⬐ unsoundInput

It is about treating new versions with major API breakage as if they were completely new libraries, not as a "maybe, possibly drop-in replacement, iff we use just the right feature subset". E.g. RxJava changed their maven artifact ('io.reactivex:rxjava:1.y.z' -> 'io.reactivex.rxjava2:rxjava:2.y.z') and the package name where in which their code lives ('rx' -> 'io.reactivex'). This makes it possible for both versions to be used at the same time while transitioning, without breaking dependencies that rely on the older version and without having to bother with automatic multi-versioning in buildtools.

With that in place it is questionable what the actual advantage of semver vs a simpler monotonous versioning schema (like build number or UTC date) is.

Rich Hickey has a great talk on this: https://www.youtube.com/watch?v=oyLBGkS5ICk

Previously discussed: https://news.ycombinator.com/item?id=13085952

⬐ qwtel

Came here to say the same thing. Also related: http://hintjens.com/blog:85 "The End of Software Versions"

⬐ williamle8300

He's making good points, but what alternative is he offering?

⬐ Anderkent

@williamle8300 >He's making good points, but what alternative is he offering?

Your comment is dead, but to answer the question - the alternative proposed is to encode the contract of your library in itself, and never break it, only extend it.

If you need to change the API, you don't change behaviour or signatures of your functions while preserving the names; you just add a completely new mainspace, while letting people use the old functions if they still have reason to. You don't have to keep actively supporting the old namespaces (i.e. no new bugfixes), but just need to not break the stuff that already worked.

There's some scenarios where this is hard - for example it's a web api and supporting it is costing you $$. But for things like that devs are generally aware that what they're consuming might get shut down at some point; so doing a new api, in a new namespace, and shutting down the old one at some point is not completely out of the question.

This reminded me of "Spec-ulation" from Rich Hickey [0]

[0] https://www.youtube.com/watch?v=oyLBGkS5ICk

There was a great talk recently by Rich Hickey[0] about how change itself is breakage. Breaking early and often is catastrophically worse than python 2.7 in my opinion.

[0]: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ nickbauman

Agreed. Hickey, in that talk, claimed if you really did break backward compatibility, you should just name it something else. So in real terms Python3 should have been renamed to Pythonidae or something like that. Eventually the last folks working in the 2.7.* branch would notice that nobody else was in the room and hopefully figure out where they went.

⬐ btrask

If you look at long-running, successful projects, they don't do this.

One example is the original V8 dev team that went off to create Dart (AIUI). Nobody followed them and V8 got replacement developers. (But perhaps that was a particularly egregious case.)

⬐ derefr

I feel like the only responsible thing to do in such cases—if you truly believe that the thing you're abandoning is just a plain-old failure of engineering, and it's effectively self-flagellation to be subjected to maintaining it—is to poison the well/salt the earth when you leave, so that people are at least forced to switch to something else, if not your replacement project.

Badly-engineered buildings get condemned, and we make it illegal to use them, even if it might benefit someone to use the building in exchange for the (likely not-very-well-considered) risk of the roof falling on them.

What sort of political/social infrastructure would it take to allow someone to condemn a codebase?

⬐ kibwen

Google has begun building "time bombs" into Chrome that will start inconveniencing users once the binaries reach a certain age without being updated (I believe this is specifically related to certificate validation, but I'm not going to look up details because I don't want to be put on a watchlist for googling "chrome time bomb" :P ).

⬐ caf

Is the watchlist for googling it worse than the watchlist for writing it?

⬐ kibwen

Nah, I was just on my phone and needed an excuse for my laziness. :P

⬐ scrollaway

https://bugs.chromium.org/p/chromium/issues/detail?id=664177

⬐ cookiecaper

I disagree. Python 2->Python 3 was not really a huge change. It seems that the Python 2 userbase is finally starting to dissolve but IMO the holdouts were mostly doing it for philosophical purposes. Renaming the project for Python 3 would've resulted in a massive decrease in useful searchability, history, etc, when really only a small handful of backward incompatible changes were at stake.

It seems the better lesson is a) keep compatibility for a LONG time; there are finally some old Windows programs that Windows 10 no longer runs; and b) when you do break compatibility, don't make a big deal out of it.

Python 3 probably would've been better if it was just a feature-heavy incremental release that didn't break compatibility. 3.2 probably should've introduced a change that discouraged but still supported the old code style (maybe print a warning any time the file didn't contain a shebang-style header like #!allow_old_style, kind of a simpler reverse-version of the future module), and then maybe 3.8 could finally remove compatibility after the old method has been deprecated and difficult and annoying for 10 years or so. Making a hard delineation between compatibility, while on the surface appearing useful, seem to have a much stronger negative effect in encouraging users to stick with "their" version out of principle than it has the planned positive effect of giving users plenty of time and notice about compatibility breakages.

This seems gradual enough that most users won't notice and there won't be a hubbub, nor an insurrection among users to shun your new iteration for making them do extra work.

Dealing with people effectively, including end users, is very difficult. It takes a lot of patience.

⬐ None

None

Which is a pity. We take backwards incompatible changes too lightly. I'm working with Scala for example and love the language, but it breaks binary compatibility between major versions because Scala's features don't map that well to the JVM's bytecode and the encoding of things like traits are fragile. Well, the community is kind of coping with it by having libraries support multiple major versions at the same time. Well, it's not much, but at least the tools have evolved to support this (e.g. SBT, IDEs, etc).

That said, if you're looking for a language that has kept backwards compatibility, that's Clojure. Rich Hickey has a recent keynote that I found very interesting: https://www.youtube.com/watch?v=oyLBGkS5ICk

I don't know how to feel about it. On one hand this means Clojure will probably never fix some obvious mistakes in its standard library, without changing its name (like what they did with ClojureScript). On the other hand, as I said, it is liberating to have code written in Clojure 1.0 still working in 1.8.

This talk [1] by Rich Hickey seems more pertinent than ever. It's a contrarian stance on semver (which, almost as much as vc itself, is lauded as "the Right way" among developers). He definitely challenged my beliefs, it's likely to be worth your time too.

https://www.youtube.com/watch?v=oyLBGkS5ICk

I'm curious to hear how folks how adopt GraphQL handle server side schema changes over time. I'm all for the idea of not breaking clients almost ever [1] but sometimes you need to coordinate between client and server logic; how do you do that if the dependency of any client can literally be your entire db schema? Perhaps there is a subset of the server side schema that is designed to be GraphQL-able and is kept very stable over time?

[1] https://www.youtube.com/watch?v=oyLBGkS5ICk

Rich Hickey just did a great talk about this sort of thing at Clojure Conj: https://www.youtube.com/watch?v=oyLBGkS5ICk

⬐ DEADB17

I don't understand why he says semantic versioning does not work. In my experience (with NPM, not maven) it is very useful, adding meaning of intent by convention:

Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes.

I got the impression that the issue was maven not being able to handle multiple versions of the same package/artifact, not in the convention.

⬐ MrBuddyCasino

The JVM can't handle multiple versions of the same jar, as there is only one classpath.

A depends on B and C, which in turn depend on incompatible versions of D.

Congrats, you're screwed, unless you're running in an OSGi container.

The only reason this seldom a problem in Java land is because many popular core Java libs, including the whole frickin standard library and all the official extended specs like servlet maintain backwards compatibility, and the successful core libs do too (Guava, commons-whatever).

They do what he preaches. Bam, successful platform!

⬐ taeric

Guava is actually a bad actor in this regard. To the point that it can sometimes teach people that "coding fearlessly" is an wonderful thing.

It certainly can be. Especially in monolith code bases where you can fix everything you broke.

As a platform, though, it is very frustrating.

⬐ auganov

His core opposition is to introducing breaking changes under the same namespace. SemVer is the leading scheme that sanctifies such practice.

⬐ jjnoakes

Perhaps, but he repeatedly claimed that the minor and patch numbers conveyed no meaning, while dismissing the semver spec as a manifesto.

But if he read and understood it, he'd know those were important numbers. Maybe moreso than the major version.

Perhaps he should have argued his actual stance more, instead of the strawman stance. That put me off.

⬐ sheepmullet

From the point of view of a library consumer why should they care about the patch or minor versions at all?

Isn't later = better?

⬐ jjnoakes

Because if you start relying on something new or fixed in x.2.z of your dependency you want to make sure anyone using your code isn't using x.1.y.

⬐ sheepmullet

And doesn't automatic dependency resolution make this a non-issue for your consumer?

Edit: I.e. If you declare your own dependencies then tooling should ensure anyone who uses your code uses the same dependencies.

It doesn't work this way in Java world due to technical limitations, but it can in JS world

⬐ prodigal_erik

That limitation is healthy. If versions 1.1 and 1.2 of a class exist and I'm foolish and determined enough to use both in the same process (via multiple classloaders), Java will still ensure that I can't accidentally give a 1.1 instance to a callee expecting a 1.2 instance, or vice versa. Version mismatches at call sites fail quickly and loudly with classloading exceptions.

I think a hell of a lot of NPM packages only appear to work by accident, and over time they'll fail because of sloppiness about this.

⬐ jjnoakes

Consumers may want to use a different version.

Perhaps they want a newer one (bug fix, security fix).

Perhaps they want an older one (since another dependency was tested against an older version of the dep in question).

Semver gives you a way to decide what ranges of versions should be safe to move between in order to satisfy all of those occasionally conflicting requirements.

⬐ sheepmullet

Explicitly limiting your consumers to a specific version of another library is a breaking change. You have introduced a very specific dependency and are requiring the consumer to honour it.

By semvar rules you should be updating the major version?

⬐ jjnoakes

What?

If my library requires X version 1.2 or higher, how is it a breaking change if I don't work with version 2.0 or 1.0 or anything except 1.2 through 1.99999?

That's the whole point of software versioning, no matter what you call it (renaming things, semver, git hashes, anything). At some point you require something of someone else, and you can only use the versions of that other library which provide what you require (or more). Semver is just a way to lock those requirements into a machine-readable number scheme.

⬐ DEADB17

Wouldn't changing the MAJOR version be equivalent to changing the namespace without altering the human memorable identifier of the package?

Understanding that changes in MINOR.PATCH are backwards compatible, is the difference between NAME MAJOR.MINOR.PATCH and NEW_NAME MINOR.PATCH significant? They look to me as just two different conventions.

⬐ taeric

No. Because I don't have a way to keep the old and the new.

That is, the point is that you didn't change my use of the old. You just changed what I actually use. It may work. It may not.

This is especially egregious when I had to up versions to get some new functions, and old versions just happen to have changed.

⬐ DEADB17

I don't know about maven, but in NPM you can keep both. If you take away the limits of a particular implementation, I think that semantic versioning is a useful convention for the producer of an artifact to convey intent to the consumers.

⬐ DEADB17

Sorry @taeric, I can't reply to your post directly.

I do think that the intent of the producer is being communicated (unsafe to upgrade, safe to upgrade with new features, safe + automatic improvements).

I'm not disagreeing that "spec" adding more metadata to have better granularity and potentially reducing the amount of manual work is a good thing. But in the absence of it, "semantic versioning" is an improvement over safe and unsafe versions being indistinguishable.

⬐ taeric

No worries. In the future, you can almost always get a reply button by clicking directly on a post. (Click on the "time since post" to get to the direct link.)

I think I see the point. Yes, he is using hyperbole. However, I have found it is more accurate than not. In particular, the point that many projects feel a lot more cavalier about doing breaking changes.

⬐ taeric

That is the point. It doesn't convey intent. Outside of the "major versions could break." Which is somewhat worthless.

Consider, you are using a library that expose ten objects and functions. It is on version 1.2.8. it upgrades to 1.2.9. What do you do? You take the upgrade. Usually no questions asked.

It upgrades to 1.3.0, but only to add an eleventh function. What do you do? Probably take it, because you don't want to be behind.

It upgrades to 2.0. reason is "things have changed.". However, they kept the same function names. You think you can make the upgrade fine. Because, well they have the same names. However, you can't know, because some body thought it wise to reverse arguments of some functions. Which thankfully, is a compile time fail. What else changed, though?

⬐ sheepmullet

Like Rich mentioned from the point of view of a library consumer it's:

PATCH: Don't care MINOR: Don't care MAJOR: You're screwed

MAJOR is simply not granular enough and MINOR and PATCH are pointless.

Sometimes when I update to a new major version of a dependency it all just works. Other times I've got to spend weeks fixing up all the little problems.

Did you break one thing I didn't even use? Update the MAJOR version. Did you completely change the library requiring all consumers to rewrite Update the major version.

⬐ jjnoakes

Still don't see the big problem. If the major version is updated and it doesn't affect you, you have a 10 second job to do. If it does, you have a bigger job to do (or don't update).

What's the big deal?

⬐ sheepmullet

How do you know if it's a 10 second job or a bigger job?

⬐ jjnoakes

Read the release notes?

Read the code diff?

Try it?

⬐ sheepmullet

So lots of manual work. And you still don't see the issue?

Wouldn't it be good if there was some kind of automated way to know?

⬐ jjnoakes

Semver doesn't preclude automating anything, does it?

⬐ klibertp

But without semver you'd need to do this manual work even more often! Semver makes you do less manual labor, because you know that PATCH and MINOR don't require your attention. You don't know such a thing in many other versioning schemes.

Would it be better to eliminate even more manual labor? Yes, of course. But then is semver bad because it reduces manual labor?

⬐ prayerslayer

I get your overall point that it's better than nothing, but you'll have to admit semver makes promises that just don't hold up in reality:

> because you know that PATCH and MINOR don't require your attention

:)

In 99 % of cases, they don't. But you're never completely sure.

⬐ DEADB17

Being screwed would be the case if the consumer of the artifact is forced to upgrade. i.e.: if versions cannot coexist in a code base.

Otherwise I think the information that they convey is useful:

PATCH: improvement or correction that does not affect the consumers expectation (safe improvement)

MINOR: additional features that may be useful directly to the consumer or its transitive dependencies (safe to upgrade)

MAJOR: No longer safe to upgrade automatically. The consumer may need to investigate further or stay with the previous MAJOR.

In any case it is useful information being conveyed. The consumer decides how to act on it.

⬐ sheepmullet

> Being screwed would be the case if the consumer of the artifact is forced to upgrade. i.e.: if versions cannot coexist in a code base.

In theory multiple versions can co-exist in a codebase in js land.

In practice though the vast majority of js libs don't compose well with different versions.

At best I've found it can work as long as you consider them seperate and unrelated libraries (basically the version in js land is part of the namespace).

Edit: I definetly don't think it's as big of an issue in js land as in Java land because of transitive dependency handling.

⬐ gniquil

Haven't watched through the entire video, but the first part, analyzing dependency bottom up, and claiming version doesn't work, really reminds me of graphql. We've been doing traditional rest with /v1/..., /v2/.... This sucks for so many different ways. Graphql's solution of providing contract at the lowest level and just evolve the "graph" really feels a bit like what he was talking about in this video. And note that Facebook by not "provide less" or "require more" in their API is how they made their API still compatible with older clients. This talk is very interesting! (note I could be spewing BS here as I've not finished the video)

⬐ opvasger

exactly my thought, having dug into GraphQL recently and seen a number of talks about it.

Getting over the annoyance of looking at some initial functionality that is misplaced can be quite hard... It's really tempting to just get rid of those as you're bumping up the MAJOR.

⬐ anonfunction

Rich Hickey gave one of my favorite talks that I recommend to all programmers no matter which language they code in:

https://www.infoq.com/presentations/Simple-Made-Easy

⬐ taeric

The whole section on "just change the name" fits interestingly in with the way that the kernel is developed. They have been rather strict on this for a long time. A function released to the wild stays. Changes to it require a new function.

⬐ dmoney

Also part of the C4 methodology: https://rfc.zeromq.org/spec:22/C4/

* A patch that introduces new features to a Public Contract SHOULD do so using new names.

⬐ Rapzid

http://man7.org/linux/man-pages/man2/syscalls.2.html

⬐ tbatchelli

This is a worthwhile talk, albeit a bit too long. In this talk RH provides a framework to think about libraries, how they change over time and how we should be communicating those changes. He proposes a way forward in which the changes are communicated in a way that can be dealt with programmatically, a much better framework than versioning schemes (semver gets a beating in this talk!).

⬐ adwhit

Video is a high-bandwidth way of delivering information at a low bandwidth. But try watching tech videos at 1.5 speed, it makes them much more engaging. Also works for ponderous anime.

⬐ achikin

That's why I prefer youtube over the other services - you can speed it up. I'm not a native speaker, so 1.25 is the fastest I can understand at the moment.

⬐ asymmetric

Got any examples of ponderous anime? Lain? Akira? GITS?

⬐ Garlef

Dragonball. They sometimes charge up one attack over several episodes.

⬐ cygned

The planet will explode in five minutes.

⬐ faitswulff

Hah! Ponderous anime. I remember watching some shows at 4x speed. I'm not sure whether it's anime that introduced me to sped up video, but subtitles are a great help when watching videos at faster than 1x speed.

⬐ atemerev

I can't get any useful information from videos, it is so slowwwww. Even on 2x (the fastest I can watch), it is still 3 to 5 times slower than I read. Also, I can control my reading, skip the minutiae, or reread important parts or look up some reference — but I can't do that with videos or audiobooks or podcasts. I wonder how people even find it convenient.

Unfortunately, Rich Hickey really loves videos for some reason, so everything he says is lost to me.

(Yes, there are transcripts, but those are linear and unstructured.)

⬐ greenyouse

Hey, I usually take notes when reading or watching stuff. If you're willing to read through my semi-coherent org-mode notes, it summarizes the talk better than a full transcript.[0]

[0]http://pastebin.com/bmF0xc7y

⬐ atemerev

Thanks!

⬐ qwtel

I've revisited this post so many times, I know the number by now: http://hintjens.com/blog:85, "The End of Software Versions". I think it is related to, and similar to what Rich is saying.

⬐ j-pb

> Logic systems don't have "And nothing else will ever be true!".

Uuh. Closed world assumption? Everything that is not true is false. Most logic systems do have this. Prologs (not a logic system I know) cut operator even turns this statement into an operation.

I feel like Rich really gets it wrong this time. His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it. But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

The best example of this style going horribly wrong are the linux kernel file system modules. Different api versions all in use at the same time by the same code with no clear documentation on what to use when.

It's also ironic that the examples he uses to make his point namely, Unix APIs, Java and HTML, are horrible to work with especially because they either never developed good API's (looking at you unix poll), or they, like browsers, have become so bloated that nobody want to touch them with a ten foot pole. One of the reasons why it takes so long for browser standards to be adopted is that they have to be integrated with all the cruft that is accumulating there for almost three decades now.

"Man browsers are so reliable and bug free and it's great that the new standards like flexbox get widespread adoption quickly, but I just wish the website I made for my dog in 1992 was supported better." -no one ever.

Combinatorial complexity is not your friend.

I'd rather have people throw away stuff in a new major release, maybe give me some time to update like sqlite or python do, and then have me migrate to a new system where they have less maintenance cost and I profit from more consistency and reliability.

I think that Joe Armstrong has a better take on this. https://www.youtube.com/watch?v=lKXe3HUG2l4

Also even though I'm a fulltime Clojure dev, I would take Elms semantic versioning that is guaranteed through static type analysis anytime over spec's "we probably grew it in a consistent way" handwaving.

⬐ joe-user

> His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it. But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

It might actually be less effort to follow his method. You may need to create a new namespace or create a new function, but then you don't need to put out breaking change notices, handle user issues due to breaking changes, etc.

> "Man browsers are so reliable and bug free and it's great that the new standards like flexbox get widespread adoption quickly, but I just wish the website I made for my dog in 1992 was supported better." -no one ever.

It's not about better support for your 1992 website, it's about it still being accessible at all. Perhaps you've never had to deal with a file in a legacy format (ahem, Word) that had value to you, but was unrecognized in newer versions of software, but I can assure you that it's thoroughly frustrating.

> Also even though I'm a fulltime Clojure dev, I would take Elms semantic versioning that is guaranteed through static type analysis anytime over spec's "we probably grew it in a consistent way" handwaving.

An Elm function `foo :: int -> int` that used to increment and now acts as the identity function is merely statically-typed "we probably grew it in a consistent way" hand-waving, which may be worse than the alternative given the amount of trust people put into types.

⬐ pron

> But doesn't fly if you maintain these things out of goodwill in your anyhow limited spare time.

What percentage of the total software produced and used fits that and should discussions of general good practices address this anomaly? Obviously, if you work outside the normal industry practices, then industry best-practices don't apply to you. I don't think you should take what he says too literally to mean that every line of code you should write must obey this and there are no exceptions.

> that nobody want to touch them with a ten foot pole.

If by nobody you mean almost everyone. Those happen to be the most successful software platforms in history, and most new software outside embedded (with the big exception of Windows technologies, maybe) uses at least one of those platforms to this day.

> One of the reasons why it takes so long for browser standards to be adopted is that they have to be integrated with all the cruft that is accumulating there for almost three decades now.

So what? Look, large software gets rewritten or replaced (i.e. people will use something else) every 20 years on average. If your tools and practices are not meant to be maintained for 20 years, then one of the following must be true 1. they are not meant for large, serious software, 2. you are not aware of the realities of software production and use, or 3. you are aware, but believe that, unlike many who tried before, you will actually succeed in changing them. Given the current reality of software, backward compatibility is just more important to more people than agility.

> Combinatorial complexity is not your friend.

Every additional requirement results in increased the complexity, and backward compatibility is just one more, one that seems necessary for really successful software (especially on the server side).

> "we probably grew it in a consistent way" handwaving.

Why do you think it's handwaving? The spec is the spec, and if you conform with the spec then that's exactly what you want.

⬐ j-pb

When stating that nobody wants to touch browsers with a ten foot pole, I meant that nobody wants to contribute to browser development unless they are paid very very well.

⬐ freshhawk

> His request to support all the abominations that you have ever written and keep them compatible with recent changes, might work if you have people pay for using your library and a company behind it

This is generally his focus, on big professional software. I'm also a Clojure dev and I'm on the other side of the fence on this one as well so sometimes I'm disappointed in the enterprise focus but I knew what I was getting into and it is still worth it. Am I crazy to think that maybe other lisps would have done better if they had demanded less purity?

Same with the examples of Unix APIs, Java and HTML. Sure, they are all bloated and horrible to work with. They are also massively, insanely successful. I think they are great examples because at that scale it's impressive that they work at all.

This is part of the pragmatism that makes Clojure great, they generally stay away from trying to solve the problem of huge projects being unwieldy and ugly and painful and instead they accept it as a given and work on tools to mitigate the problem. For a lot of people backwards compatibility isn't a choice, it's a requirement set in stone. Even though it always causes everyone to pull their hair out in frustration.

One day maybe one of these other research languages or experiments will find an answer to this, and prove that it works at scale. I will celebrate more than most.

⬐ kazinator

Almost everything is mutable in Common Lisp: lexical variables, objects, global function bindings. Code compiles to native and you can deliver a single binary executable with no dependencies.

⬐ junke

> Am I crazy to think that maybe other lisps would have done better if they had demanded less purity?

People associate Common Lisp with many different things, but not purity.

⬐ DigitalJack

The primary advantage of clojure to my mind are the lovely data structures and the interaction with them.

Common Lisp and scheme can implementation them, and through reader macros interact in probably the same ways, but it would always be second or third class.

Second big deal for clojure is the ecosystem.

I'd love a native language like clojure that was driven by a language specification.

⬐ ozten

As I understand it, one of the original problems Go was designed to solve was long compile times at Google.

This talk is fascinating and points out a possible solution space for real world problems in giant codebases and build infrastructures.

What if 80% of your dependency graph can be identified as dead code paths at build time and not require you to actually take dependencies on all that dead code?

⬐ _halgari

Sadly then Go went and shot themselves in the foot. Originally (I assumed this hasn't changed), you specified dependencies via git URLs. Sounds great, but then they went and said every dependency always used the head of master. So now you're in the horrific situation of your deps suddenly changing whenever your build decides to pull the latest from the git repos.

(Note: I don't use go, but this was the situation about 3 years ago when I last looked).

⬐ NightMKoder

One of the things Rich talks about in this talk is semantic versioning is fundamentally broken. In fact any versioning scheme that allows removal is broken. In some sense this model encourages Rich's ideas better - since you can't version directly, you either make a new library or maintain backwards compatibility.

Of course the current 'go get' model has a lot of downsides, e.g. Non deterministic builds. I still think it's worth considering building on, rather than trying to fix semver. All that's really missing is a stable monotonic identifier - something as simple as git tree height may be enough.

⬐ _halgari

Absolutely, I think the "perfect world" would be adopting some of the ideas Rich proposes but adding (git url + sha) as the dependency management system.

Of course for that to work well you'd also need to protect yourself from people deleting their git hub repos, or rewriting git history.

⬐ robto

Which is a great use case for something like ipfs, I think.

⬐ anonfunction

I've always found this to be overlooked or at least a misguided design decision. IMO would be very helpful to be able to specify a commit, tag or branch to pull down.

The Go team's solution to this problem was allowing vendored dependencies. Basically you can put the dependencies inside your project in a directory called vendor which will be used instead of whatever is in your $GOPATH/src/. This allows you to pin deps to a specific version and not need to pull deps from the internet at all.

⬐ eternalban

I was surprised by this. (Rich is rigorously perceptive and characteristically peels conflations apart.) Here, he seems to be conflating linker responsibilities with (unit-module) dependency management concerns.

If deployment-unit-A depends on du-B, but system-language level dependencies have additional subtleties, then that is fine, since someone like RH could be looking into code stripping of unreachable/dead-code on linking, and I'd be completely surprised if that hasn't been already looked at extensively to date.

Yes, you would sacrifice some transfer bandwidth/time on artifact acquisition but keep conceptual models local and their implementations simpler. Or, looking at it from another point of view, OP should consider that a 'dependency' is maximally the provider of objects in context of its namespace.

⬐ greenyouse

When I was listening to the part about breakage vs accretion, I kept thinking back to the Nix pkg manager from NixOS. It does a pretty good job of separating out the artifacts by indexing them by version number and a hash of the username (i.e. artifact-name = sha($user) + $project-name + $semver). There have to be other examples of pkg managers that do this kind of thing too...

⬐ sgentle

I think there is a missing piece to this conversation, which is about the definition of namespaces. It's easy to conflate "problems with semver" and "problems with semver+the way my package manager handles versioning".

In npm, or with any package manager that allows multiple independent versions to coexist, the version becomes part of the namespace. Multiple different versions of a package can be installed at the same time and referenced from each other. It's no problem if you depend on B1.0 and C1.0, and they depend on D1.0 and D2.0 respectively; you just get all four installed.

In Rich Hickey's vision, we would move the compatibility contract into the package name, so if we want to make a new version 2 of helloworld that does not obey the contract of version 1, we call it helloworld2. In practical terms, this is no different from the above, it's just that we add the version number to the namespace through the package name rather than the package version.

Not surprisingly, this latter strategy is already used by most systems where versions aren't part of the namespace. Debian has python2 and python3. Python has urllib and urllib2 (and urllib3). Unix has mmap and mmap2. It works fine, but it's a bit of a hack.

If your version is part of the package namespace, semver works fine and solves the problem it was intended to solve. If, like in Java, every version has to share the same name, then I agree that semver doesn't really buy you that much. After all, it doesn't help you much to know that your unresolvable dependency tree is semantic.

It's not a coincidence that npm handles transitive dependencies in the way it does; it's the whole point. I'm sure it makes npm much more unwieldy to allow multiple different package versions, but having used it it is exactly one of those "solves a problem you didn't realise you had" moments.

With that said, I definitely agree that more careful API design would allow far fewer backwards-incompatible (ie semver-major) changes, and more tightly defining compatibility (like with this requires/provides concept) is a good way to do that. Many web APIs already say things like "we will return an object with at least these fields, and maybe more if we feel like it", which I think is a good example to follow.

As far as changing package names, there's also an interesting question, not about how computers deal with those package names, but how we do. After all, why helloworld and helloworld2? Why not give it another name entirely? Part of that is because you want to make the connection between these two packages explicit. But if you change your package dramatically, perhaps that connection is not doing a service to your users.

One of the biggest complaints about major rewrites is "why does it have to be the new way? I liked it the old way!" If you genuinely believe the new way is better, why not let it compete on an even playing field under its own name? (And, if you know your new package can't compete without backwards compatibility, maybe don't make a new package.)

⬐ marysimpson

Ever require the services of a hacker with discretion and top servicing, i implore you to try your very best to hire only professionals. It will increase your chances of getting your job completed. i was able to hire the services of an elite, asides the fact that i was provided a permanent solution to the service that was rendered me but gave a very efficient customer experience. gmail- pauleta.steelbreaker. M-+19283233115 (text only)

⬐ programnature

This is a very thick talk, one Rich's best ever IMHO.

The first point is how we talk about 'change' in software, to center around what things 'provide' and 'require'.

Breaking changes are changes that cause code to require more or provide less. Never do that, never need to do that. Good changes are in the realm of providing more or requiring less.

There is a detailed discussion about the different 'levels' - from functions, to packages, artifacts, and runtimes, which he views as multiple instances of the same problem. Even though we now have spec, theres a lot of work to leverage it across all those different layers to make specific statements about what is provided and required.

⬐ michaelfeathers

That's Postel's Law.

⬐ mshenfield

I found value in dissecting the different levels of change. For the sake of sanity though, we should do breaking changes. Breaking changes exist because we have limited capacity as individuals and an industry to maintain software. This is especially true for infrastructure that is supported by (limited) corporate sponsorship and volunteers. Breaking changes limit our window of focus to two or three snapshots of code, instead of having our window of focus grow without bound. Our limited capacity can still be effective as a library changes over time.

The most important point of this talk is here: "You cannot ignore [compatibility] and have something that is going to endure, and people are going to value" [0]. Breaking changes provide a benefit for library developers, but it is usually damage done to end users. As consumers we should weigh the cost of keeping up with breaking changes against the quality of a tool, and the extra capacity its developers are likely to have.

[0] https://youtu.be/oyLBGkS5ICk?t=4177

⬐ solussd

Additionally there are safer, usually reasonable, ways to deal with what would otherwise be breaking changes. Give the changed functionality a different name, create a new namespace/module without the removed functionality, or create a new library if you have introduced something fundamentally different (e.g., w.r.t. how you interact with it). That way your users can choose to refactor their code to use the change, rather than discover their expectations no longer match reality when they upgrade.

⬐ blueprint

Who says you have to maintain old code? We're talking about simply not deleting it and establishing a discrete semantic for the new version as truthfully, a new version is new content which demands a new name to accurately and precisely describe it. If it didn't it would be like saying different content doesn't produce a different hash.

⬐ wtetzner

Except that naming things is one of the hard problems. I don't see why a major version bump can't be considered a different library.

I guess you can use version numbers in the name instead, since this talk is specifically targeting maven artifacts.

⬐ blueprint

Hickey's other talk says hard is relative, and I happen to agree, especially when it comes to naming. The question is to what degree of exactness you can confirm what exists (in problems). That is a function of your degree of truthfulness. So it's "hard" only in the sense it's hard to approach 100% truthfulness. However, I have observed that one doesn't need 100%, one needs to be beyond a certain threshold of effective sufficiency. And according to human history, special, rare individuals are born who do exceed that threshold.

⬐ mshenfield

You're right, there is no obligation to maintain it. I think that misses the point though. The value in keeping the code is to allow the end user to continue to enjoy improvements in parts of the library that don't have breaking changes without upgrading those that do. You could continue to have security patches installed, for example. That value is much less when you don't do basic maintenance implement bug fixes and security patches.

⬐ blueprint

Unless I'm missing something… the answer to that problem is to (a) factor the code sufficiently to then (b) create an abstraction (interface) that backs out the concrete implementation to the specifically desired version/functionality.

⬐ michaelfeathers

Agreed. Breaking changes can lead to alienation of user base, but I think there's a danger in lulling people into expecting that kind of constancy in software. It creates dependency of another kind. Maybe the trick is to vary features at some rate, getting users used to change and bringing them along.

In retail it used to be the case that you could go to the same store a month later and see the same shirt to buy. The Sears catalog [1] presented that sort of constancy for consumers. Today there's a lot of flux. Some of it actually engineered to prevent people from delaying purchasing decisions. In software we can and do introduce breaking changes for ease of maintenance, and that can be ok as long as people are used to it. It's making the choice to have a living ecosystem.

[1] http://www.searsarchives.com/catalogs/history.htm