Hacker News Comments on
Making Software: What Really Works, and Why We Believe It
·
1
HN points
·
12
HN comments
- This course is unranked · view top recommended courses
Hacker News Stories and Comments
All the comments and stories posted to Hacker News that reference this book.Surprised nobody have mentioned Making software, https://www.amazon.com/Making-Software-Really-Works-Believe/...It takes a empirical view on the software process. Does writing tests first help you develop better code faster? Can code metrics predict the number of bugs in a piece of software? Do design patterns actually make better software?
Code review approaches are discussed in this book:https://www.amazon.com/Making-Software-Really-Works-Believe/...
IIRC the evidence was that formal code reviews were no more effective than async peer review (but both were effective at uncovering bugs).
My understanding is that it is harder for engineers to estimate how long a job takes, than to do the job. That is to say, that the complexity of doing a time estimation task is higher than the complexity of task you are estimating. I read this in the book "Making Software: What really works and Why we believe it" [0]. The assumption in the article is that an engineer can produce reliable estimates for complex work, and appropriately guide their managers, and don't think that's true.[0] https://www.amazon.co.uk/Making-Software-Really-Works-Believ...
⬐ jtc331I don’t think the article assumes an engineer can produce reliable estimates for complex work; in fact it claims the opposite:> But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown.
⬐ commandlinefan⬐ hyperpallium> I don’t think the article assumes an engineer can produce reliable estimatesWell, he sort of goes back and forth, but he includes this:
> The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks.
Which I see some variant of every time I see somebody rail against the unrealistic expectations of software estimation (which, by the way, I’ve been seeing people rail against since the late 80’s to no avail). The implication here is that if the manager had just listened to the developer and accepted his initial estimate of one month, the software would have been done in one month: the developer could estimate with precision, but the manager bumbled along and screwed everything up by trying to negotiate it down.
This is a dangerous position to take unless you’re absolutely sure about your one-month estimate: if you say one month, he says two weeks and you look him in the eye like the alpha wolf say, “no, one month, and no sooner”… you had damned well better be able to deliver in exactly one month. The reality is there’s probably no way to tell, _especially_ if other people are involved, so you’re better off shrugging your shoulders and saying, “yeah, sure, two weeks”, doing as much as you can, and preparing your story ahead of time.
I recently estimated a task would take 30 minutes.It took 40 minutes to do that estimate.
But, TBF, several important decisions were made during the process of estimating, such as what should be excluded from the task, basic organization, and some research.
BTW the task being estimated was doing time estimates for a project (which came out to be 2-3 weeks).
⬐ cwilkes⬐ skohanObviously we now need an estimate for how long it will take to make an estimate.⬐ duxupOften it seems you take the time to properly evaluate the code and changes .... often it seems you are nearly done.⬐ basilgohar⬐ commandlinefanThis is often overlooked and undervalued — upfront time spent on understanding the problem and designing before getting down to the actual work ends-up saving a lot of actual time. But most people don't have patience for that and think it is wasted time.⬐ duxupYeah I've only been coding professionally for a year now and I do SO MUCH MORE reading/researching, planning, annotating, and pseudo coding than I ever did when I started.The outcomes are so much more predictable / better quality.
> I recently estimated a task would take 30 minutes.My boss at my last job told me that he had observed that if he asked somebody “can you get this done by the end of they day”, he would get an accurate answer (either yes or no). Any further out, there was no correlation between what they said and what they actually delivered.
In that case it seems like it would be better for PM's to take the role of task estimation. Of course this only works if they are also the ones accountable for inaccurate estimates.⬐ snarf21Estimates are hard because you are literally making something that has never been made before. Yes, you have experience with sub parts and similar patterns but not the situation of this timeline, this team, this technical debt, etc.Too often the request is along the lines of "How long does it take to build a house?" not "How long will it take to build the house in these blueprints with these systems on top of a mountain that is also impervious to mudslides?". People can generally estimate the former but the latter is all about the unknown details.
> Unfortunately, the phrasing would be more like, It doesn't matter what's true, it matters what i can sell to management.Yes, that's my main beef with it. SCRUM could be based on science, but sadly, it isn't. Here's an interesting book: https://www.amazon.com/Making-Software-Really-Works-Believe/...
Linked FTA, book about software dev based on empirical studieshttps://www.amazon.com/Making-Software-Really-Works-Believe/...
⬐ hackitsLovely book (Read it cover to cover a couple of times) and a step in the right direction for the Software field.
Interesting work.It has some pretty basic flaws that are pervasive that the reader should be aware of. First off is the "productivity" measure: the last time I looked around, this was nearly impossible to quantify for software development. The author chose SLOC and development time as stand-ins for productivity. SLOC has a connection with code quality and time of development, but development time is well known to vary[1].
In particular, development time in this thesis is linked to results in a pretty classic "Psychologist's fallacy"[2]; the author generalizes their experiences as conclusive. This implies in particular that "development time" in this thesis should be thrown out for meaningful conclusions, as the sample size is 1. It is, however, an interesting experience report.
Another basic flaw is the connecting of 'modern' with 'good'. The author remarks the main disadvantage of C and Fortran are their age; this crops up here and there. Workflow tooling in Go and Rust are major focuses by the developers: C workflow tooling is usually locally brewed. This does not make C-the-language worse.
More on a meta level, my advisor in my Master's work drummed into me that I should NOT insert my opinion into the thesis until the conclusion. So I found the editorializing along the way very annoying.
And finally, and very unfortunately, the experience level of the author appears to be low in all three languages; this is significant when it comes to implementing high performance code.
---
Now, for the interesting / good parts of this experience report.
Standout interesting for me was the Rust speedup on the 48-core machine. I did not expect that, nor did I expect Go to make such a good showing here as well.
In general Go performance/memory made a surprisingly good showing (to me) for this work. I shall have to revise my opinion of it upward in the performance axis.
I am both surprised and vaguely annoyed by the Rust runtime eating so much memory (Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it).
Of course the stronger type system of Rust catching an error that C and Go didn't pick up is both (a) humorous and (b) justifies the type community's work in these areas. I look forward to Rust 1.0!
One note by the author that is worth calling out stronger is the deployment story: Go has a great one with static linking, whereas C gets sketchy and Rust is... ??. I believe Rust has a static linker option, but I havn't perused the manual in that area for some time. For serious cloud-level deployments over time, static linking is very nice, and I'm not surprised Google went that route. It's something that would be very nice to put as a Rust emission option "--crate-type staticbin".
Anyway. I look forward to larger sample sizes and, one day, a better productiv
[1] http://www.amazon.com/Making-Software-Really-Works-Believe/d... The situation is actually far worse than just varying developer time.
⬐ steveklabnik> Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it.I haven't spent time with the code yet, so I can't comment. It shouldn't be the runtime, though, as our runtime is about as big as C or C++'s. My first thought would be that Vec's growth factor may be poor in this instance, but without reading the code, who knows.
> Rust is... ??
Rust statically links everything but glibc by default. Experimental support for musl was added last week.
⬐ pnathanOh, cool! Thanks for the information!⬐ steveklabnik⬐ madezNo problem. Doing a valgrind run on the sample, let's see how it goes.⬐ pnathanWhat did you find out? When I scanned through the code, I didn't see anything jump out at me as being Wrong.I read more than once that rust has no runtime. Could you please clarify whether it has a runime, and if it does, what it consists of?⬐ steveklabnikSo, basically _every_ non-assembly programming language has a runtime. Even C and C++. But not all runtimes are created equal. Languages like C, C++, and Rust have a very minimal, no setup required runtime. Other, higher level languages often have something significantly more heavy. People will sometimes say "no runtime" to mean the class of languages which have very small runtimes.Rust's runtime is here: https://github.com/rust-lang/rust/tree/master/src/rt and https://github.com/rust-lang/rust/tree/master/src/libstd/rt
As you can see, it's very, very small. It mostly handles things like unwinding, at_exit handlers, and the like.
While TDD originally stood for writing tests first and then code for the majority of people TDD means you have tests. I have encountered countless people that say they are using TDD, but writing tests after or during coding and not before. To them test driven development means that you are done with a task when the test passes. In comparison to before where there was no tests it is easy to see how the new way of developing is related to tests so they think they are doing TDD. So I would say that "classic" TDD or test first development is dead.It also didn't help how TDD was shown in a study to not work out very well. I don't have it off hand, but it was talked about in the "Making Software" book http://www.amazon.com/exec/obidos/ASIN/0596808321/wwwicefoxn...
"Making Software" [1] has a chapter called "How Effective is Test-Driven Development?" where they do a meta review of 32 clinical studies on TDD pulled from 325 reports. They filtered out reports based on scientific rigour, completeness, overlap, subjectivity etc, and ended up with 22 reports based on 32 unique trials. "TDD effectiveness" was defined as improvement across four dimensions - internal code quality, external system quality, team productivity, and test quality - and each dimension is strictly defined in sufficient detail.They report that high-rigour studies show TDD has no clear effect on internal code quality, external system quality, or team productivity. While some studies report TDD has a positive impact, just as many report it has a negative impact or makes difference at all. The only dimension that seems to be improved is "test quality" - that is, test density and test coverage - and the researches note that the difference was not as great as they had expected.
The chapter concludes that despite mixed results, they still recommend trialling TDD for your team as it may solve some problems. However it is important to be mindful that there is yet no conclusive evidence that it works as consistently or as effectively as anecdotes from happy practitioners would suggest.
The meta-study is relatively short and can be read online [2].
[1] http://www.amazon.com/Making-Software-Really-Works-Believe/d...
[2] http://hakanerdogmus.net/weblog/wp-content/uploads/tdd-sr-bo...
⬐ steveklabnikCoincidentally, I am currently reading that book, but had not gotten to that part yet. Thanks!
it makes a huge number of claims but does not back up these claims by any empirical evidenceName a book that does! The corpus of evidence in our field is quite weak and doesn't support almost any interesting claim you might make about it. For example, there's no empirical data supporting version control [1].
There are books that purport to derive software development lessons from the research literature, but they're either not very good (because they rely on crappy studies and/or go beyond what the studies support) or not very interesting (because they merely transcribe the results of very narrow studies). An anthology of the literature was published recently [2], and it falls in the latter category. I bought it to get the state of the art in empirical research and found it contained almost nothing of use to me as a practitioner. Many of the studies were shoddy (e.g. qualitative case histories--a euphemism for "anecdote") and the better ones are so careful to limit themselves to the tiny experiments they did that you can't derive anything from them. I could mention a couple of exceptions, though, if anyone's interested.
I think the root problem is that software development is complicated, so to do rigorous formal studies on it is expensive. The amount of funding available for this work is a drop in the bucket of what it would need to be. In short, the market value of empirical evidence on software development isn't high enough for anyone to pay for it.
[1] http://neverworkintheory.org/2012/12/12/empirical-evidence-f.... and http://neverworkintheory.org/2012/12/30/why-we-need-evidence.... Notice that the HN discussion of the first post completely failed to answer its question: https://news.ycombinator.com/item?id=4931251.
[2] http://www.amazon.com/Making-Software-Really-Works-Believe/d...
The book "Making Software: What Really Works, and Why We Believe It" (http://www.amazon.co.uk/Making-Software-Really-Works-Believe...) has a section on this.Chapter 8 "Beyond lines of Code: Do we need more complexity metrics?" by Israel Herraiz and Ahmed E Hassan.
Their short answer is that, in the case they looked at, all the suggested metrics correlated with LOC, so you may as well use LOC as it's so easy to measure.
IIRC they believe it's only good to compare LOC between different employees if they are doing pretty much the exact same task however, but since LOC is correlated with code complexity, there is some measure there.
I recommend the book, as really focusing on the science of computer science.
The thing about "many good ideas and rules of thumb" is, I've got a few dozen of those of my own! Most of us do. It would be interesting if there were decisive evidence against any of them, but even when I read studies whose conclusions contradict my beliefs, the studies are so flimsy that I find it easy to keep my beliefs.There does seem to be a recent wave of software engineering literature, exemplified by http://www.amazon.com/Making-Software-Really-Works-Believe/d... (which I haven't read). Are you familiar with this more recent stuff? Does it represent new research or merely new reporting on old research? If the former, are the standards higher?
⬐ mcguireI haven't read all of Making Software yet myself, but you would be interested in one of the first few chapters. I don't remember who wrote it offhand, but as I recall, in discussing the standards of evidence needed for software engineering the author concluded, and I am paraphrasing here, that hard numbers were difficult to get and came with many, many conditions; as a result anecdotes were likely the best you could do and were perfectly acceptable. (Was that enough disclaimers?)You might be able to tell why I lost my enthusiasm for the book.
⬐ gruseomThanks. My enthusiasm mostly consists of trying to get other people to read this stuff and tell me what it says :)I think that's the argument for junking the SE literature. If it can't do any better than anecdote, well, to quote Monty Python, we've already got some they're very nice.
I highly recommend the book "Making Software: What Really Works, and Why We Believe It".John Graham-Cumming, the author of the article submitted here, has a review of this book on Amazon.com:
http://www.amazon.com/Making-Software-Really-Works-Believe/d...
"This isn't a book about evangelizing the latest development fad, it's about hard data on what does and does not work in software engineering."