HN Books @HNBooksMonth

The best books of Hacker News.

Hacker News Comments on
The Art of UNIX Programming (The Addison-Wesley Professional Computng Series)

Eric Raymond · 6 HN comments
HN Books has aggregated all Hacker News stories and comments that mention "The Art of UNIX Programming (The Addison-Wesley Professional Computng Series)" by Eric Raymond.
View on Amazon [↗]
HN Books may receive an affiliate commission when you make purchases on sites after clicking through links on this page.
Amazon Summary
The Art of UNIX Programming poses the belief that understanding the unwritten UNIX engineering tradition and mastering its design patterns will help programmers of all stripes to become better programmers. This book attempts to capture the engineering wisdom and design philosophy of the UNIX, Linux, and Open Source software development community as it has evolved over the past three decades, and as it is applied today by the most experienced programmers. Eric Raymond offers the next generation of "hackers" the unique opportunity to learn the connection between UNIX philosophy and practice through careful case studies of the very best UNIX/Linux programs.
HN Books Rankings

Hacker News Stories and Comments

All the comments and stories posted to Hacker News that reference this book.
Shouldn't scripts over 100 lines be rewritten in Python or Ruby?

http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh...

tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write)

Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...

Online for free: http://www.catb.org/esr/writings/taoup/html/

Turns out, nothing substantial had been published since the 1980s.

The best reference I've found on this is The Art of Unix Programming, which was published in 2003:

https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...

The book is also online for free:

Command Line Options: http://www.catb.org/~esr/writings/taoup/html/ch10s05.html

User Interface Design Patterns: http://www.catb.org/~esr/writings/taoup/html/ch11s06.html

Still it's nice to see this knowledge circulating!

Joel Spolsky's 2003 review of this book: https://www.joelonsoftware.com/2003/12/14/biculturalism/

veddox
An excellent book! It was one of the first software development books I read and still shapes my programming style. I probably ought to read it again though, I suspect I would understand a lot of it a lot better now...
(author here) I do use Python. In fact I wrote something like 30K lines of Python for Oil.

The question isn't shell OR Python.

My shell scripts call Python scripts, many of white I wrote myself. That is working as intended.

They also call C programs, C++ programs and R programs. And put JavaScript programs in various places.

----

I guess this is a huge misconception about shell that I have to write a blog post about.

https://news.ycombinator.com/item?id=24083764

I'm often asked this about Oil [1]: Why do you want to write programs in shell?

That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.

I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.

Shell is the best language for that, but it's also old and crufty, with poor implementations.

When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".

-----

If you want some concrete examples, look in the Oil repo. There are dozens of shell scripts that invoke custom tools in Python, R, and C++.

https://github.com/oilshell/oil/tree/master/benchmarks

For example, to generate this report on how much we speed up Python by translating it to C++: https://www.oilshell.org/release/0.8.3/benchmarks.wwz/mycpp-...

The tables are manipulated by R code, and shell/Python/CommonMark generates the HTML.

-----

Another example is that the release page is essentially a huge shell script: https://www.oilshell.org/release/0.8.3/

This page and all the linked pages are generated by: https://github.com/oilshell/oil/blob/master/devtools/release...

------

If you don't understand why shell should call Python and C, I recommend reading The Art of Unix Programming

http://www.catb.org/~esr/writings/taoup/html/

https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...

CameronNemo
>When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".

I don't think this is a good characterization at all... Shell does not have git functionality (for example) built in. That is a dependency just like it would be for a python or rust project.

chubot
What I mean is that shell speaks paths, pipes, and processes natively, and you can creatively use tools that are "just there". Example: http://www.oilshell.org/blog/2017/09/19.html

It's an analogy, not a precise statement. It could be made more precise by using Oil as the center of a container-based "semi-distro", i.e. a distro that does everything that's not hardware related.

I guess a little like the complement to what CoreOS was doing (or is?). (This is a project I've been thinking about for awhile; anyone should feel free to contact me if they've done something like that or have ideas. It's related to the dev tools problems described in the blog post.)

Here's one from the book The Art of UNIX Programming[0]: "The Unix philosophy is bottom-up and not top-down. It is pragmatic and grounded in experience."

[0] http://www.amazon.com/UNIX-Programming-Addison-Wesley-Profes...

emmelaich
Predating those two are:

UNIX as literature http://theody.net/elements.html

Life with UNIX by Don Libes http://minnie.tuhs.org/Archive/Documentation/Books/Life_with... (from the author's website)

vram22
Started reading the Don Libes book, good stuff, brings back memories of my Unix work / days; thanks for sharing.
smhenderson
I'm only part way through the 1st chapter but the Don Libes book is quite good. Having been written in 1989 it's an interesting perspective of Unix before Linux.

Thanks for sharing!

digi_owl
> "The Unix philosophy is bottom-up and not top-down. It is pragmatic and grounded in experience."

Something that seems to be lost on a solid portion of the current custodians of that legacy...

vram22
That book - TAOUP - is quite good, though written in ESR's typical style (somewhat heavy and maybe overuse of relatively complex and less-used words).

One concept or principle mentioned in the book - the X Windows concept - of "provide mechanism, not policy" [1] - is, IMO, one of the reasons for Unix's longevity and success, even as it has morphed through various versions - from the original Bell Labs ones through Sun, HP, IBM's versions, and then Linux, the BSDs, etc., and up to Android and Mac OS X and iOS. I commented somewhere recently on a blog post about this.

[1] To make it clear, the principle name is slightly misleading. It does not mean "don't provide at all". It just means, let the lower layers of a software stack (such as X Windows) provide only the mechanism, and leave it to higher layers (such as, say, KDE or GNOME), to decide the policy.

Wikipedia article:

https://en.wikipedia.org/wiki/Separation_of_mechanism_and_po...

http://www.amazon.com/Programming-Addison-Wesley-Professiona...

3.0 out of 5 stars Autohagiography with some programming tips, December 24, 2003 By A Customer

The writing style of this book tends to hurt the reading experience, as Raymond trumpets his own minor achievments in the free software community. The work feels like it needed one more rewrite before being released to the public: some related sources Raymond hadn't yet read at the time of writing, and some of his advice gets repetitive.

The exposition itself is not up to par with The Elements of Programming Style. Raymond tries to give a list of programming rules or principles to follow, but it reads more like a list of slogans that should be taken as axioms. While The Elements of Programming Style itself had a list of rules, the rules were well woven with each other, well defended, and they were used as a means of conveying a larger story. In Raymond's case, he relies upon the slogans in absence of such a story.

Thus, the book ends up more like a list of random unrelated tips. Some very profound, like his writings on threads (which he acknowleges Mark M. Miller for his help). Others are very shallow and pointless in a book that supposes to call itself about "Art." Some of the pieces appear only to function to attack Windows, and sometimes the information about Windows is embarassingly inaccurate.

One final criticism is that Raymond does not understand object-oriented programming very well and misses the point in several cases. You just need to see the popularity of Python, Java, C# (Mono), OO Perl and C++ in the Linux world to see that Raymond is off base calling OO a failed experiment. In fact, with almost any matter of opinion in the book you can feel Raymond's bias and be hit in the face with misinformation or dull false dilemmas.

However, given this book's many flaws, I rate this 3 stars instead of 2 stars because it also has valuable information from the many contributors, some of them Gods in the Unix world. These contributors often even disagree with Raymond, or point out other interesting tidbits. For these tips alone, it is worth checking out this book, though I would not recommend you buy it.

To get the true Unix programming philosophy, I recommend Software Tools, by Kernighan and Plauger. It's somewhat dated, and I recommend the Ratfor version of it, but that single book has became very influencial as I grow as a Unix programmer.

[...]

One of the problems with this book is the overly partisan tone it takes - one gets the impression that absolutely nothing Microsoft has ever done is of value, but the other major desktop PC OSes (Apple, Linux) represent different forms of perfection. (At home, I run Mac OSX, RedHat Linux and Windows, and have a reasonable sense of their relative strengths and weaknesses.)

So, be warned: Art of Unix Programming paints a one sided picture. The author is a well-known figure in the open source community, one of its fiercest advocates, and one of Microsoft's most vocal critics, so it might seem to strange to wish for less anti-Microsoft spin from this source. After all, the Raymond brand certainly carries with it an obligatory expectation of Windows-bashing, doesn't it?

One of the only Windows design decision which Raymond doesn't condemn is the (now discontinued) .ini file format. Even the thorough-going support for object-orientation in Windows is given short-shrift: after explaining the many horrors of object-oriented programming (according to Raymond), Unix-programmers are praised as "tend[ing] to share an instinctive sense of these problems." This section ([...]) is particularly illustrative of the one-sided approach that Raymond takes.

[...]

His comments about the Windows registry were a bit distressing, though -- not because they're negative, which I consider fine. Rather, it was obvious he'd never used it (comments like "there's no API for it") and it was also clear that he hadn't even bothered to research why it existed and what problems it was intended to solve. The comments were typical of what I'd expect of a Slashdot troll, but not of a bright, respectable person like ESR. I've programmed on both platforms extensively and only comment on what I have first-hand experience and knowledge of; I'd expect him to do the no less, especially as an author.

It was also curious that several times he implied unit testing == XP == agile software development. For as tuned in as he seems to be to methodolgy work, missing the forest for a single leaf is a bit embarrassing.

[...]

Hate to be the one to burst the proverbial bubble..., March 13, 2004 By Bill Joy (Hamburg, DE)

...but ESR's theory and ideology is rediculously flawed with misappropriated valuations. This is, yet another advocacy campaign for gnu/Linux, mixed in with a UNIX context. Given the purpose of the book, it's a fair assessment to label the burdening bias as filler and firewood: filler for those who really just wanted an explanation of the single-purpose, POSIX illustrated, hows and whys of UNIX programming philosophy; and firewood for those who tend to buy into slanderous hype at the whim of suggestion.

Questions: What is the title of the book? What programmatic philosophies are portrayed? How many unices are open source? Of those, how many subscribe to the same opensource mentality as gnu/Linux? Answer: zilch. Question: Then what is the relevance of such topics to the objectives of the book, as depicted in the title? Answer: fudd-ala-mode.

Aside from these intertwined distractions, and a severe Napoleonic complex against anything new or different, ESR does adequately represent the real purpose of the book, and in those efforts place value in the read.

[...]

an interesting and often annoying read..., June 1, 2004 By A Customer

I suppose any book containing so many interesting quotes from so many UNIX luminaries cannot be overlooked. (I wonder if any of them would have co-authored this) It also happens to contain a great many topics that are well-worth writing about; My only wish is that someone less in awe with the contents of his own field of vision, and with greater depth and objectivity (not to mention humility) had the opportunity to write this book.

Quality of discussion is varied as expected; Raymond is not quite the UNIX expert he thinks he is. In places, Raymond's tone encourages one to throw the book at the nearest wall and go out just to get some fresh air; He is condescending, hectoring, lecturing, and sometimes just misleading. Alas, I will still recommend it as worth reading (check your local library) with a nice grain of salt; just enough friction for thought is provided in this edition.

[...]

programming book without any code, May 11, 2013 By davez (LA, CA, United States)

If you expect to see any code, you'll be disappointed. The author gives ample examples, but mostly preaching the Unix philosophy or religion, at many times repeating the similar points. 500 pages are a little long for a book of this kind. Having read other Unix books, I've heard many of the stories retold here, but still learned a few new things. It reenforces many notions of Unix.

This is a book that I would recommend to someone new to Unix, but it is not a book that I would keep. After 10 years, the computing landscape has changed with the rise of the mobile computing and cloud computing; the Unix has evolved to support smart phones and tablets, in which the user interface has become essential.

It is actively harmful to teach students that software architecture is something that somehow arises from diagrams or that those kinds of silly pictures capture anything important about it. Powerful architectures come out of powerful ideas that in turn come from accumulated hard work of many people in different disciplines. One can learn much more from walking through the actual source code of some classic projects and from trying to understand the ideas that make them tick:

https://github.com/onetrueawk/awk - UNIX philosophy of small tools, DSLs, CS theory: state machines / regular expressions, Thompson algorithm ...

https://github.com/mirrors/emacs - Both a program and a VM for a programming language, hooks, before/after/around advices, modes, asynchronous processing with callbacks, ... Worth to think of challenges of designing interactive programs for extensibility.

https://github.com/rails/rails - Metaprogramming DSLs for creating powerful libraries, again a lesson in hooks (before_save etc.), advices (around_filter etc.), ...

https://github.com/git/git - The distributed paradigm, lots of CS theory again: hashing for ensuring consistency, DAGs everywhere, ... By the way, the sentence "yet the underlying git magic sometimes resulted in frustration with the students" is hilarious in the context of a "software architecture" course.

One of computer algebra systems - the idea of a http://en.wikipedia.org/wiki/Canonical_form

One of computer graphics engines - Linear algebra

...

There are loads of things one can learn from those projects by studying the source in some depth, but I can't think of any valuable things one could learn by just drawing pictures of the modules and connecting them with arrows. There are also several great books that explore real software design issues and not that kind of pretentious BS, they all come from acknowledged all-time master software "architects", yet all of them almost never find diagrams or "viewpoints" useful for saying the things they want to say, and they all walk you through real issues in real programs:

http://www.amazon.com/Programming-Addison-Wesley-Professiona...

http://www.amazon.com/Paradigms-Artificial-Intelligence-Prog...

http://www.amazon.com/Structure-Interpretation-Computer-Prog...

http://www.amazon.com/Unix-Programming-Environment-Prentice-...

http://www.amazon.com/Programming-Environment-Addison-Wesley...

To me, the kind of approach pictured in the post, seems like copying methods from electrical or civil engineering to appear more "serious", without giving due consideration to whether they really are helpful for anything for real-world software engineering or not. The "software engineering" class which taught those kind of diagram-drawing was about the only university class I did not ever get any use from, in fact I had enough industry experience by the point I took it that it just looked silly.

synctext
Actively harmful? Diagrams?

Sorry I have to counter-react strongly. Your post is negative without a valid point and the irony is that you agree with the post.

> One can learn much more from walking through the actual source code of some classic projects and from trying to understand what makes them tick

That is the fun part about this post! You propose to mandidate a list of classical code bases. This course lets students freely pick any Github project with activity and complexity. They get graded based on pull requests.. Contributing code and interacting with an Open Source project you picked a week ago is not "drawing pictures" and "connecting them with arrows".

stiff
Programs are powerful because of concepts. Whether or not you will be able to build a good 3d engine rests much more on the fact whether you know or able to invent a concept of a vector and have it guide all your future coding than it rests on whatever is visible from a block diagram. Both the course and the book linked to seem to focus on some abstract ideas of doubtful usefulness and never ever seem to touch upon those important things that historically have allowed us to build complex programs where other approaches have failed. Google can exist not because people invented new kinds of diagrams, but because large scale numerical linear algebra techniques were applied to the problem of search. This is the heart of Google's "architecture", not a class diagram of the crawler.

Picking arbitrary programs won't do. The point is to pick the really good designs and illustrate why they are good, and I think it boils down much more to knowing mathematics, algorithms, programming languages and having studied a lot of existing software, than to usage of any modelling techniques so far invented in formal software engineering circles.

broseph
There's no conflict between reading and understanding a codebase and seeing the value in a diagram. You can have both. Take a look at the example you chose: [0] and [1] (about a third of the way down).

They first explained PageRank (without a diagram), then explained the flow of the data from the crawler (with a diagram). You seem to be arguing that diagrams are never useful, and that just isn't true.

[0] http://infolab.stanford.edu/~backrub/google.html [1] http://infolab.stanford.edu/~backrub/over.gif

avandeursen
Here's an addition to the answers from the others -- I'm one of the authors of the post.

Thanks for your feedback.

Perhaps we did not sufficiently emphasize it in the blog post, but we do tell the students about strong existing architectures, open source as well as from industry. The books listed in further reading in the post actually contain chapters about git and emacs, and I share your enthusiasm for Bentley's little language approach.

The philosophy of the course is based on trying to apply published approaches to software architecture (such as those in Rozanski & Woods' book, but it could be others) to real systems actually maintained and used. We encourage to get in touch with the creators of those systems.

The students actually take a deep dive into a foreign code base -- I included pictures because that is easy to share in a post, but the students analyze code, report about that, execute test cases, deploy systems, etc. The pictures are indeed, as suggested by some of the others, means for communication.

stiff
What repels me is the kind of approach taken by this Rozanski & Woods book. It's like a 500 pages of authors philosophy of software, and whether all that is of any use is completely uncertain. There is not much of a widely established corpus of knowledge on software engineering/architecture, another book would treat an almost completely different set of concepts, and whatever is commonly agreed upon is trivial and most people invent in on their own when they need it. You can publish completely anything about "Software architecture" in this spirit thats sounds smart enough and it won't ever be put to any test. I personally, from experience, don't believe any practical high-level theory of software engineering of this kind exists at all, the more "philosophical" or "theoretical" it gets the less useful it becomes. It is much better to spend this time studying established disciplines where things actually get tested out one way or another, mathematics, algorithms, operating systems, ... If there is anything specific to be taught under "software architecture", it's those ideas ("patterns") that made certain programs so much better than the alternatives, but you only really learn them by digging deep in, not by studying some very abstract models.
hansbogert
From your post it seems you think software architecture and -engineering are the same.. In this course -- disclaimer: I was a student taking this course -- it's not that case, Software Architecture is not the sum of a program's lines or modules. SA is the methodology of getting involved and steering a project, this includes handling team efforts and other stakeholders. Looking and learning a lot of (implementation) patterns is of course very useful, but only a subpart of SA.

I tend to think that a good SArchitect knows that the patterns they might be using are good or well suited for the current problem, a better SArchitect knows that he's perhaps using a less well suited pattern for the problem, but can layout a plan to migrate to a new and better situation without breaking continuity of the project.

Furthermore, it can't be stated clearer that in the course there are a lot of guest lecturers who _do_ show different idea's.

porker
In addition to what the other respondents have written, could it be that you are not a visual learner, and diagrams don't 'do it' for you?
currywurst
The pictures are a result of studying the systems in question.

For example, you have given links to a couple of github repos. Now if I'm trying to create an overview of how say emacs works, I would try to identify the main sub-parts and break the system down into more manageable chunks of study. This is all what 'viewpoints' are trying to structure. None of this is evident from just the codebase unless it has been captured in some documentation, or you take the trouble to understand it yourself.

You seem to have a very low opinion of software architects, but believe me, you wouldn't want to work on a 5-year old system where huge teams of developers have let loose and no one/team was responsible to ensure a cohesive, maintainable system.

Viewpoints are fantastic for communication purposes, and if you feel that the particular viewpoints shown don't make much of a difference, I agree :). A student project is also not the ideal environment, but it is really valuable to know how to create them, considering the information needs of the audience (biz vs tech vs in-depth tech). And that is what this course is all about.

stiff
The point is that how emacs is broken down in modules is one of the least important things about emacs. I can draw a block diagram without a "software architecture" course, thank you. In software engineering, there are of minor usefulness most of the time, if of any.

There have to be persons that guide the overall development, sure, but this does not mean that those have to be "architects" living in a castle somewhere drawing those diagrams. The best "architects" are people with long experience in the trenches that know all the ins and outs, all the relevant theory, know the domain, and supervise it down to the nitty-gritty details. Linus Torvalds is a software architect for Linux, he doesn't do much coding anymore, but he isn't busy drawing diagrams without understanding the code really well either, he just is the person with the widest understanding of the whole project:

https://lkml.org/lkml/2012/10/2/547

currywurst
There are a LOT of practicing architects that really fit the mould you are describing! Linus has the big picture in his head, and perhaps if he were to draw them out, does it make him an ivory tower denizen ?!

Also, ignoring how a system is modularized is typically how you get tangled codebases by introducing unexpected dependencies. Just because it is 'less interesting' doesn't make it 'unimportant'.

HN Books is an independent project and is not operated by Y Combinator or Amazon.com.
~ yaj@
;laksdfhjdhksalkfj more things
yahnd.com ~ Privacy Policy ~
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.