HN Books @HNBooksMonth

I don't have any one complete book that I can recommend, and I don't even really have a great reading list for this. But I'll make an attempt to share what I think is useful as a starting point.

1. Systems Operations is first and foremost about understanding systems, in all of their complexity, which means understanding the internals of your OS primarily.

2. Performance and networking, in particular, are super important areas to focus on understanding when it comes to learning the topic to help with software development.

3. A lot of it is about understanding concepts in abstract and being able to extrapolate to other situations and apply these concepts, so there's actually quite a lot of useful information that can be learned on one OS and still applied to another OS (or on one game engine and applied to another, et al).

Here's a few books I think are worth reading, not in any particular order of prevalence, but loosely categorized

Databases:

High Performance MySQL: https://www.amazon.com/gp/product/1449314287/

SQL Queries for Mere Mortals: https://www.amazon.com/gp/product/0321992474/

The Art of SQL: https://www.amazon.com/gp/product/0596008945/

Networking:

TCP/IP Illustrated: https://www.amazon.com/exec/obidos/ISBN=0201633469/wrichards... (updates on author's site at http://www.kohala.com/start/tcpipiv1.html)

The TCP/IP Guide: https://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-P...

UNIX Network Programming: https://www.amazon.com/dp/0131411551

Beej's Guide to Network Programming: http://beej.us/guide/bgnet/

Operating Systems:

Operating Systems Concepts: https://www.amazon.com/Operating-System-Concepts-Abraham-Sil... (various editions, I have the 7th edition... I recommend you find the latest)

Modern Operating Systems: https://www.amazon.com/Modern-Operating-Systems-Andrew-Tanen... (the "Tanenbaum Book")

Operating Systems Design and Implementation: https://www.amazon.com/Operating-Systems-Design-Implementat-... (the other one, the "MINIX Book")

Windows Internals:

Part 1: https://www.amazon.com/Windows-Internals-Part-architecture-m...

Part 2: https://www.amazon.com/Windows-Internals-Part-2-7th/dp/01354... (I had the pleasure of being taught from this book by Mark Russinovich and David Solomon at a previous employer, was an amazing class and these books are incredible resources even applied outside of Windows, we used 5th edition, I linked 7th, which has the 2nd part pending publication).

MacOS Internals:

Part 1: https://www.amazon.com/MacOS-iOS-Internals-User-Mode/dp/0991...

Part 2: https://www.amazon.com/MacOS-iOS-Internals-II-Kernel/dp/0991...

Part 3: https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp...

Linux Kernel Programming:

Part 1: https://www.amazon.com/Linux-Kernel-Development-Cookbook-pro...

Part 2: https://www.amazon.com/Linux-Kernel-Programming-Part-Synchro...

The Linux Programming Interface: https://www.amazon.com/Linux-Programming-Interface-System-Ha...

General Systems Administration:

Essential Systems Administration: https://www.amazon.com/gp/product/0596003439/

UNIX and Linux Systems Administration Handbook: https://www.amazon.com/UNIX-Linux-System-Administration-Hand...

The Linux Command Line and Shell Scripting Bible: https://www.amazon.com/Linux-Command-Shell-Scripting-Bible/d...

UNIX Shell Programming: https://www.amazon.com/Unix-Shell-Programming-Stephen-Kochan...

BASH Hackers Wiki: https://wiki.bash-hackers.org/

TLDP Advanced BASH Scripting Guide: https://tldp.org/LDP/abs/html/

The Debian Administrator's Handbook: https://debian-handbook.info/browse/stable/

TLDP Linux System Administrator's Guide: https://tldp.org/LDP/sag/html/index.html

Performance & Benchmarking:

Systems Performance: https://www.amazon.com/Systems-Performance-Brendan-Gregg-dp-... (this is Brendan Gregg's book where you learn about the magic of dtrace)

BPF Performance Tools: https://www.amazon.com/Performance-Tools-Addison-Wesley-Prof... (the newer Brendan Gregg book about BPF, stellar)

The Art of Computer Systems Performance Analysis: https://www.cse.wustl.edu/~jain/books/perfbook.htm (no longer available from Amazon, but is available direct from publisher. This is basically the one book you should read about creating and structuring benchmarks or performance tests)

I guess that's a "reading list", but this is just a small part of what you need to know to excel in systems operations.

I would say for the typical software developer writing web applications, the most important thing to know is how databases work and how networking works, since these are going to be the primary items affecting your application performance. But there's obviously topics not included in this list that are also worth understanding, such as browser/DOM internals, how caching and CDNs work, and web-specific optimizations that can be achievable with HTTP/2 or QUIC.

For the average software developer writing desktop applications, I'd say make sure you /really/ understand OS internals... at the base everything you do on a computer system is based on what the OS provides to you. Even though you are abstracted (possibly many layers) away from this, being able to peel back the layers and understand what's /really/ happening is essential to writing high-quality application code that is performant and secure, as well as making you a champ at debugging issues.

If you're trying to get into systems operations as a field, this is just a brush over the top surface and there's a lot deeper diving required.

⬐ planet-and-halo

Appreciate the recs. Thanks very much for taking the time to write a detailed response.

The best start would probably be both parts of Windows Internals:

https://www.amazon.com/Windows-Internals-Part-architecture-m...

https://www.amazon.com/Windows-Internals-Part-Developer-Refe...

These aren't about programming per se, but if you want to do systems programming it helps to have a detailed understanding of the system. :)

After that, specific reading probably depends on the exact task you want to perform, but MS has good documentation and tutorials in many areas. Writing drivers, for example:

https://docs.microsoft.com/en-us/windows-hardware/drivers/ge...

⬐ muricula

To add on, the windows internals books have exercises with the sysinternals tools. If sitting down and plowing through a doorstop of a book isn't your cup of tea, try picking a chapter or two which sound interesting, skimming them, and doing the exercises.

Oh come on!

https://www.amazon.co.uk/Windows-Internals-Part-architecture...

Certainly all systems have undocumented stuff that are left undocumented because the users are not supposed to use it. If they would, they couldn't change it.

⬐ bediger4000

System calls seems like the very definition of what should be documented for users. Particularly if you want someone to write compilers for your operating system.

⬐ kyberias

How exactly does the Windows design here prevent writing compilers? The API is stable, it just isn't on the syscall level.

Please educate yourself about the Windows architecture.

⬐ yyyk

The stable program-kernel interface is not the kernel syscalls, but library API. That's exactly the same as every other non-Linux OS out there including other Unixes and other Free OSs.

⬐ bediger4000

Pretty sure that isn't true of FreeBSD, NetBSD or OpenBSD. It most certainly wasn't true of any Unix.

They all have files that end up as /usr/include/sys/syscall.h and define well-known numerical constants for each (documented!) system call.

https://github.com/openbsd/src/blob/master/sys/sys/syscall.h

https://github.com/lattera/freebsd/blob/master/sys/sys/sysca...

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/sys/syscall.h?re...

Oh, look! They even share the exact same numerical values for the old school system calls. #1 is exit(), #2 is fork(), so on and so forth.

Gosh, on my linux laptop, here in /usr/include/asm/unistd_64.h is pretty much the same numbers, but objectively hidden a little bit more than in the direct Unix descendants. I interpret this as "less documented than in Unixes".

Wow, even Minix source code has the same numbers: https://minixnitc.github.io/posix.html

I wonder why that is? Maybe it's because the system call numbers are in fact documented, part of the POSIX API, and descend from Bell Labs Unix source code. Here's V7's list of system call numbers as proof:

https://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/include...

⬐ yyyk

Documenting syscalls does not mean a guarantee of stability. I can't recall any other Unixes except Linux which has committed to maintaining syscalls, which means this interface cannot be relied on.

e.g.

OpenBSD deletes "relic from the past" syscall:

https://cvsweb.openbsd.org/src/sys/sys/syscall.h?rev=1.199&c...

Solaris documenting deleted syscalls (note however the libc interface is guaruaneed):

https://docs.oracle.com/cd/E23824_01/html/E22973/gkzlf.html

MacOS X is so unstable that even go developers had to go through libSystem:

https://golang.org/doc/go1.12

⬐ Volt

> MacOS X is so unstable that even go developers had to go through libSystem

Just to be clear to anyone else, the Mac OS X syscall interface is guaranteed to be unstable by Apple. Going through libSystem is the only way.

⬐ derpherpsson

A book... That cost money.

Compare that with the man pages of some decent BSD. Or even Linux..

But yes. I have contemplated buying one of those windows internals. I will probably buy that book just because of your comment. It's not that expensive.

The series of books known as "Windows System Internals"[0]

Raymond Chen of "Old New Thing" fame and all his friends[1]

The creator of process monitor occasionally writes nice blog posts about troubleshooting[2]

A lot of historical NT stuff is still valid, or at least mildly relevant today.

[0]https://www.amazon.com/Windows-Internals-Part-architecture-m...

[1]https://blogs.msdn.microsoft.com/oldnewthing/

[2]https://blogs.technet.microsoft.com/markrussinovich/2005/10/...

For those who are interested, the next version of Windows Internals is available for pre-order from Amazon :)

https://www.amazon.com/Windows-Internals-Part-architecture-m...

Oh, and @EvanAnderson - you might want to take a look at the Windows Sybsystem for Linux (WSL) - which allows you to run unmodified ELF64 Linux binaries directly on Windows:

https://blogs.msdn.microsoft.com/commandline/2016/06/02/lear...

Windows Internals 7th Edition Book 1 is available already[0], and there will be 2 more books[1]. Russinovich and Solomon have stepped down a while ago, according to that forum thread, but the new book lists them as authors because Catlin and Hanrahan worked over the already existing stuff.

[0] http://www.amazon.com/Windows-Internals-Book-User-Edition/dp...

[1] http://forum.sysinternals.com/when-is-the-7th-edition-book-f...