HN Books @HNBooksMonth

I would like to take it a step further and ask a question that has been bothering me a while. On my time in the academy I studied the following two books (regarding C):

[1] Advanced Programming in the UNIX Environment https://www.amazon.com/Advanced-Programming-UNIX-Environment...

[2] C Programming Language https://www.amazon.com/Programming-Language-2nd-Brian-Kernig...

In combination with other classes (and books) on networking, operation systems, data structures we covered a big variance of use cases with C. My question is: How do I take this to the next level? For example I feel I never missed a concept from those classes but when I see C code posted on a thread here it's probably something completely unreadable and complex. Can anyone provide resources to advance a little? What should I study if my goal is to make good and useful modern C software?

I feel like my typing is a bit abstract but I will be happy to clarify.

PS Yes, I've heard of C++ and Rust ;P

⬐ zh3

[1] is a very good book - it's not really one for learning C though, it's more for when you know C and you need to use it in - well - an advanced unix environment. Sort of like you shouldn't be learning how to use a scalpel in an operating theatre :)

⬐ _osorin_

I got it for our operating systems class, it's not aimed to teach C from scratch. My bad for not clarifying.

⬐ belter

Start by reading Redis code, slowly take notes on paper, try to make your version changing small bits at a time.

https://github.com/redis/redis/tree/unstable/src

Entry point here at line 6816: https://github.com/redis/redis/blob/unstable/src/server.c

Also "Code Reading" will be useful. - https://www.amazon.com/Code-Reading-Open-Source-Perspective/...

"Code Reading" tends to be criticized, but I think mostly unfairly.

⬐ _osorin_

Thanks a lot for the recommendation.

The Linux Programming Interface: https://man7.org/tlpi/

Advanced Programming in the UNIX Environment, 3rd Edition: https://www.amazon.com/gp/product/0321637739

neat. Great way to explore the systems.

For me out of university, it was a co-workers books (some of which I ended up buying) that help me understand the role of UNIX better (I used the alpha machines at university, but was thrown into HPUX/Solaris at work..)

I think this was one: https://www.amazon.com/Advanced-Programming-UNIX-Environment...

Of course a little out of date now, but a lot of the general concepts are the same.

Unix Power Tools helped me a lot as well. https://www.oreilly.com/library/view/unix-power-tools/059600...

How did others learn this stuff?

Unix System Programming. A course largely based around https://www.amazon.com/Advanced-Programming-UNIX-Environment...

An interesting side-effect of this course - I got _really_ good at using a console/command prompt and handling text with vim and pipes and filters and other text manipulation. I think this has helped me me productive at my jobs way more than I thought it would :)

Advanced Programming in the Unix Environment - Stevens, Rago [0]

Unix Network Programming - Stevens [1]

[0] https://www.amazon.com/Advanced-Programming-UNIX-Environment...

[1] https://www.amazon.com/Unix-Network-Programming-Sockets-Netw...

I used the first edition, but I think all you need is this:

https://www.amazon.com/Advanced-Programming-UNIX-Environment...

⬐ Firerouge

Or try this one if you'd like to build your own from scratch

https://www.amazon.com/Design-Implementation-MTX-Operating-S...

If you want to become a professional and not just a dabbler I would recommend reading some of the following books I have in my bookshelf:

[0] RHCSA & RHCE Training and Exam Preparation Guide by Asghar Ghori. This book will help insure you know your stuff as your system engineer/administrator wise.

[1] A Practical Guide to Linux Commands, Editor and Shell Programming Third Edition. This book will cover the majority of what you would need and want to know when connecting to a remote linux system over ssh.

If you want to get under the hood and become an expert, the following books should help get you started:

[2] Advanced Programming in the UNIX Environment

[3] The Linux Programming Interface: A Linux and UNIX System Programming Handbook

[4] Linux Kernel Development 3rd Edition

To get a nice general overview and get up and going quickly:

[5] How Linux works: What every superuser should know

[6] The Linux Command Line

[7] Python Crash Course

[8] Automate the boring stuff with Python. This is a great book to help you think about how to automate most of the repetitive things you will end up doing on a regular basis.

[0] https://www.amazon.com/RHCSA-RHCE-Red-Enterprise-Linux/dp/14...

[1] https://www.amazon.com/Practical-Guide-Commands-Editors-Prog...

[2] https://www.amazon.com/Advanced-Programming-UNIX-Environment...

[3] https://www.amazon.com/Linux-Programming-Interface-System-Ha...

[4] https://www.amazon.com/Linux-Kernel-Development-Robert-Love/...

[5] https://www.amazon.com/How-Linux-Works-Superuser-Should/dp/1...

[6] https://www.amazon.com/Linux-Command-Line-Complete-Introduct...

[7] https://www.amazon.com/Python-Crash-Course-Hands-Project-Bas...

[8] https://www.amazon.com/Automate-Boring-Stuff-Python-Programm...

⬐ grepthisab

This looks like a gold mine! Thanks so much!

⬐ iDemonix

The RHCSA/RHCE stuff is good, I passed the RHCSA a couple years ago and it's a solid set of foundations.

There's a good chat (devopschat.slack.com) that's useful for learning stuff like this.

Its a bit tricky I think.

> A secure coding standard form CERT should focus entirely on describing conventions and program properties that do not already follow from the standard as a matter of correctness.

from CERT 1.7 "The wiki also contains two platform-specific annexes at the time of this writing; one annex for POSIX and one for Windows. These annexes have been omitted from this standard because they are not part of the core standard."

So while the CERT does use some examples from system interfaces its not a standard for programming the system interfaces for POSIX or Windows. It looks like there trying to limit the standard to ISO C. The examples you gave fall into the system interface category. POSIX is huge and the same for Windows, much bigger then ISO C.

I think in order to explain conventions for a system interface you really need a longer form publication like a book. So you can take 50 pages to describe an interface and how to use it and show examples etc.

The best way that I have found to figure this stuff out is the standard way. You get a copy of all the relevant standards as a foundation, ISO, POSIX, Window and stuff like CERT. Then you you get some of the system programming books (listed below). Then you find get some good reference code that show best practice. usually code from the operating system or utilities. Lastly read all the compiler docs and tool docs to set up the best code analysis framework you can.

These are a few system programming books that I use.

(best intro book) GNU/Linux Application Programming https://www.amazon.com/GNU-Linux-Application-Programming/dp/...

UNIX Systems Programming https://www.amazon.com/UNIX-Systems-Programming-Communicatio...

Advanced Programming in the UNIX Environment https://www.amazon.com/Advanced-Programming-UNIX-Environment...

Windows System Programming https://www.amazon.com/Programming-Paperback-Addison-Wesley-...

The Linux Programming Interface http://www.man7.org/tlpi/

edit: I'm not sure your skill level, you may have seen all of those but I posted them regardless. There is a lot of security and convention in those books.

Two alternative, and better, choices, if you're willing to spend a few dollars (and hopefully expense it to your manager):

The Linux Programming Interface: http://www.amazon.com/Linux-Programming-Interface-System-Han...

Advanced Programming in the UNIX Environment: http://www.amazon.com/Advanced-Programming-UNIX-Environment-...

A not-too-distant third choice, Linux System Programming: http://www.amazon.com/Linux-System-Programming-Talking-Direc...

⬐ ntumlin

How strongly do you recommend these books? Unfortunately I'm a college student without much cash to toss around, but I can scrape together enough for a book if it'll make me a better programmer in the long run.

⬐ ereyes01

If you want to learn what's one layer beneath the hood of most higher-level language environments, they are well worth the investment. And IMO this stuff is worth learning. One day, you will be glad to know it when you use strace to successfully debug why your application doesn't work, while the rest of your coworkers tear up more code in befuddlement :-)

⬐ ndesaulniers

LPI is the way to go.

I just started doing kernel devlopment professionally, and ironically the content of LPI is relatively higher level.

Glad I read it before starting though.

⬐ stplsd

In my opinion these are best two books on the subject. Stevens' book is classic and have been updated quite recently (3rd edition came out 3 years ago), LPI is very extensive and have some specific Linux info not present in Stevens' book and is clearly influenced by Stevens book.

⬐ b3h3moth

Stevens is dead many years ago. The second and the third editions were written by Rago. He is a good author but he is not Stevens.

As someone who is moving into more of a devops role from a pure development role. Here is my learning list so far.

1. The TCP/IP Guide: A Comprehensive, Illustrated Internet Protocols Reference. http://www.amazon.com/The-TCP-Guide-Comprehensive-Illustrate...

2. Advanced Programming in the UNIX Environment http://www.amazon.com/Programming-Environment-Addison-Wesley...

3. A systems programming language- I choose golang.

4. GDB/makefiles

5. SSH, The Secure Shell: The Definitive Guide: The Definitive Guide http://www.amazon.com/SSH-Secure-Shell-Definitive-Guide-eboo...

⬐ LinuxDevOps

If you like the Stevens, his TCP/IP Illustrated is kind of a 'bible'.

Also, not sure what you're getting into but usually for sysadmin for scripting you need to know one of: Bash|python|Perl|Ruby

Here are few examples of great 1000+ pages books with lots of code. Though reading them is much more effort than reading a self-help book: http://www.amazon.com/Programming-Environment-Addison-Wesley... http://www.amazon.com/TCP-IP-Illustrated-Implementation-Vol/... http://www.amazon.com/Physically-Based-Rendering-Second-Edit...

⬐ michaelwww

O'Reilly, C# 5.0 In A Nutshell http://shop.oreilly.com/product/0636920023951.do

It is actively harmful to teach students that software architecture is something that somehow arises from diagrams or that those kinds of silly pictures capture anything important about it. Powerful architectures come out of powerful ideas that in turn come from accumulated hard work of many people in different disciplines. One can learn much more from walking through the actual source code of some classic projects and from trying to understand the ideas that make them tick:

https://github.com/onetrueawk/awk - UNIX philosophy of small tools, DSLs, CS theory: state machines / regular expressions, Thompson algorithm ...

https://github.com/mirrors/emacs - Both a program and a VM for a programming language, hooks, before/after/around advices, modes, asynchronous processing with callbacks, ... Worth to think of challenges of designing interactive programs for extensibility.

https://github.com/rails/rails - Metaprogramming DSLs for creating powerful libraries, again a lesson in hooks (before_save etc.), advices (around_filter etc.), ...

https://github.com/git/git - The distributed paradigm, lots of CS theory again: hashing for ensuring consistency, DAGs everywhere, ... By the way, the sentence "yet the underlying git magic sometimes resulted in frustration with the students" is hilarious in the context of a "software architecture" course.

One of computer algebra systems - the idea of a http://en.wikipedia.org/wiki/Canonical_form

One of computer graphics engines - Linear algebra

...

There are loads of things one can learn from those projects by studying the source in some depth, but I can't think of any valuable things one could learn by just drawing pictures of the modules and connecting them with arrows. There are also several great books that explore real software design issues and not that kind of pretentious BS, they all come from acknowledged all-time master software "architects", yet all of them almost never find diagrams or "viewpoints" useful for saying the things they want to say, and they all walk you through real issues in real programs:

http://www.amazon.com/Programming-Addison-Wesley-Professiona...

http://www.amazon.com/Paradigms-Artificial-Intelligence-Prog...

http://www.amazon.com/Structure-Interpretation-Computer-Prog...

http://www.amazon.com/Unix-Programming-Environment-Prentice-...

http://www.amazon.com/Programming-Environment-Addison-Wesley...

To me, the kind of approach pictured in the post, seems like copying methods from electrical or civil engineering to appear more "serious", without giving due consideration to whether they really are helpful for anything for real-world software engineering or not. The "software engineering" class which taught those kind of diagram-drawing was about the only university class I did not ever get any use from, in fact I had enough industry experience by the point I took it that it just looked silly.

⬐ synctext

Actively harmful? Diagrams?

Sorry I have to counter-react strongly. Your post is negative without a valid point and the irony is that you agree with the post.

> One can learn much more from walking through the actual source code of some classic projects and from trying to understand what makes them tick

That is the fun part about this post! You propose to mandidate a list of classical code bases. This course lets students freely pick any Github project with activity and complexity. They get graded based on pull requests.. Contributing code and interacting with an Open Source project you picked a week ago is not "drawing pictures" and "connecting them with arrows".

⬐ stiff

Programs are powerful because of concepts. Whether or not you will be able to build a good 3d engine rests much more on the fact whether you know or able to invent a concept of a vector and have it guide all your future coding than it rests on whatever is visible from a block diagram. Both the course and the book linked to seem to focus on some abstract ideas of doubtful usefulness and never ever seem to touch upon those important things that historically have allowed us to build complex programs where other approaches have failed. Google can exist not because people invented new kinds of diagrams, but because large scale numerical linear algebra techniques were applied to the problem of search. This is the heart of Google's "architecture", not a class diagram of the crawler.

Picking arbitrary programs won't do. The point is to pick the really good designs and illustrate why they are good, and I think it boils down much more to knowing mathematics, algorithms, programming languages and having studied a lot of existing software, than to usage of any modelling techniques so far invented in formal software engineering circles.

⬐ broseph

There's no conflict between reading and understanding a codebase and seeing the value in a diagram. You can have both. Take a look at the example you chose: [0] and [1] (about a third of the way down).

They first explained PageRank (without a diagram), then explained the flow of the data from the crawler (with a diagram). You seem to be arguing that diagrams are never useful, and that just isn't true.

[0] http://infolab.stanford.edu/~backrub/google.html [1] http://infolab.stanford.edu/~backrub/over.gif

⬐ avandeursen

Here's an addition to the answers from the others -- I'm one of the authors of the post.

Thanks for your feedback.

Perhaps we did not sufficiently emphasize it in the blog post, but we do tell the students about strong existing architectures, open source as well as from industry. The books listed in further reading in the post actually contain chapters about git and emacs, and I share your enthusiasm for Bentley's little language approach.

The philosophy of the course is based on trying to apply published approaches to software architecture (such as those in Rozanski & Woods' book, but it could be others) to real systems actually maintained and used. We encourage to get in touch with the creators of those systems.

The students actually take a deep dive into a foreign code base -- I included pictures because that is easy to share in a post, but the students analyze code, report about that, execute test cases, deploy systems, etc. The pictures are indeed, as suggested by some of the others, means for communication.

⬐ stiff

What repels me is the kind of approach taken by this Rozanski & Woods book. It's like a 500 pages of authors philosophy of software, and whether all that is of any use is completely uncertain. There is not much of a widely established corpus of knowledge on software engineering/architecture, another book would treat an almost completely different set of concepts, and whatever is commonly agreed upon is trivial and most people invent in on their own when they need it. You can publish completely anything about "Software architecture" in this spirit thats sounds smart enough and it won't ever be put to any test. I personally, from experience, don't believe any practical high-level theory of software engineering of this kind exists at all, the more "philosophical" or "theoretical" it gets the less useful it becomes. It is much better to spend this time studying established disciplines where things actually get tested out one way or another, mathematics, algorithms, operating systems, ... If there is anything specific to be taught under "software architecture", it's those ideas ("patterns") that made certain programs so much better than the alternatives, but you only really learn them by digging deep in, not by studying some very abstract models.

⬐ hansbogert

From your post it seems you think software architecture and -engineering are the same.. In this course -- disclaimer: I was a student taking this course -- it's not that case, Software Architecture is not the sum of a program's lines or modules. SA is the methodology of getting involved and steering a project, this includes handling team efforts and other stakeholders. Looking and learning a lot of (implementation) patterns is of course very useful, but only a subpart of SA.

I tend to think that a good SArchitect knows that the patterns they might be using are good or well suited for the current problem, a better SArchitect knows that he's perhaps using a less well suited pattern for the problem, but can layout a plan to migrate to a new and better situation without breaking continuity of the project.

Furthermore, it can't be stated clearer that in the course there are a lot of guest lecturers who _do_ show different idea's.

⬐ porker

In addition to what the other respondents have written, could it be that you are not a visual learner, and diagrams don't 'do it' for you?

⬐ currywurst

The pictures are a result of studying the systems in question.

For example, you have given links to a couple of github repos. Now if I'm trying to create an overview of how say emacs works, I would try to identify the main sub-parts and break the system down into more manageable chunks of study. This is all what 'viewpoints' are trying to structure. None of this is evident from just the codebase unless it has been captured in some documentation, or you take the trouble to understand it yourself.

You seem to have a very low opinion of software architects, but believe me, you wouldn't want to work on a 5-year old system where huge teams of developers have let loose and no one/team was responsible to ensure a cohesive, maintainable system.

Viewpoints are fantastic for communication purposes, and if you feel that the particular viewpoints shown don't make much of a difference, I agree :). A student project is also not the ideal environment, but it is really valuable to know how to create them, considering the information needs of the audience (biz vs tech vs in-depth tech). And that is what this course is all about.

⬐ stiff

The point is that how emacs is broken down in modules is one of the least important things about emacs. I can draw a block diagram without a "software architecture" course, thank you. In software engineering, there are of minor usefulness most of the time, if of any.

There have to be persons that guide the overall development, sure, but this does not mean that those have to be "architects" living in a castle somewhere drawing those diagrams. The best "architects" are people with long experience in the trenches that know all the ins and outs, all the relevant theory, know the domain, and supervise it down to the nitty-gritty details. Linus Torvalds is a software architect for Linux, he doesn't do much coding anymore, but he isn't busy drawing diagrams without understanding the code really well either, he just is the person with the widest understanding of the whole project:

https://lkml.org/lkml/2012/10/2/547

⬐ currywurst

There are a LOT of practicing architects that really fit the mould you are describing! Linus has the big picture in his head, and perhaps if he were to draw them out, does it make him an ivory tower denizen ?!

Also, ignoring how a system is modularized is typically how you get tangled codebases by introducing unexpected dependencies. Just because it is 'less interesting' doesn't make it 'unimportant'.