HN Books @HNBooksMonth

I don't have any one complete book that I can recommend, and I don't even really have a great reading list for this. But I'll make an attempt to share what I think is useful as a starting point.

1. Systems Operations is first and foremost about understanding systems, in all of their complexity, which means understanding the internals of your OS primarily.

2. Performance and networking, in particular, are super important areas to focus on understanding when it comes to learning the topic to help with software development.

3. A lot of it is about understanding concepts in abstract and being able to extrapolate to other situations and apply these concepts, so there's actually quite a lot of useful information that can be learned on one OS and still applied to another OS (or on one game engine and applied to another, et al).

Here's a few books I think are worth reading, not in any particular order of prevalence, but loosely categorized

Databases:

High Performance MySQL: https://www.amazon.com/gp/product/1449314287/

SQL Queries for Mere Mortals: https://www.amazon.com/gp/product/0321992474/

The Art of SQL: https://www.amazon.com/gp/product/0596008945/

Networking:

TCP/IP Illustrated: https://www.amazon.com/exec/obidos/ISBN=0201633469/wrichards... (updates on author's site at http://www.kohala.com/start/tcpipiv1.html)

The TCP/IP Guide: https://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-P...

UNIX Network Programming: https://www.amazon.com/dp/0131411551

Beej's Guide to Network Programming: http://beej.us/guide/bgnet/

Operating Systems:

Operating Systems Concepts: https://www.amazon.com/Operating-System-Concepts-Abraham-Sil... (various editions, I have the 7th edition... I recommend you find the latest)

Modern Operating Systems: https://www.amazon.com/Modern-Operating-Systems-Andrew-Tanen... (the "Tanenbaum Book")

Operating Systems Design and Implementation: https://www.amazon.com/Operating-Systems-Design-Implementat-... (the other one, the "MINIX Book")

Windows Internals:

Part 1: https://www.amazon.com/Windows-Internals-Part-architecture-m...

Part 2: https://www.amazon.com/Windows-Internals-Part-2-7th/dp/01354... (I had the pleasure of being taught from this book by Mark Russinovich and David Solomon at a previous employer, was an amazing class and these books are incredible resources even applied outside of Windows, we used 5th edition, I linked 7th, which has the 2nd part pending publication).

MacOS Internals:

Part 1: https://www.amazon.com/MacOS-iOS-Internals-User-Mode/dp/0991...

Part 2: https://www.amazon.com/MacOS-iOS-Internals-II-Kernel/dp/0991...

Part 3: https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp...

Linux Kernel Programming:

Part 1: https://www.amazon.com/Linux-Kernel-Development-Cookbook-pro...

Part 2: https://www.amazon.com/Linux-Kernel-Programming-Part-Synchro...

The Linux Programming Interface: https://www.amazon.com/Linux-Programming-Interface-System-Ha...

General Systems Administration:

Essential Systems Administration: https://www.amazon.com/gp/product/0596003439/

UNIX and Linux Systems Administration Handbook: https://www.amazon.com/UNIX-Linux-System-Administration-Hand...

The Linux Command Line and Shell Scripting Bible: https://www.amazon.com/Linux-Command-Shell-Scripting-Bible/d...

UNIX Shell Programming: https://www.amazon.com/Unix-Shell-Programming-Stephen-Kochan...

BASH Hackers Wiki: https://wiki.bash-hackers.org/

TLDP Advanced BASH Scripting Guide: https://tldp.org/LDP/abs/html/

The Debian Administrator's Handbook: https://debian-handbook.info/browse/stable/

TLDP Linux System Administrator's Guide: https://tldp.org/LDP/sag/html/index.html

Performance & Benchmarking:

Systems Performance: https://www.amazon.com/Systems-Performance-Brendan-Gregg-dp-... (this is Brendan Gregg's book where you learn about the magic of dtrace)

BPF Performance Tools: https://www.amazon.com/Performance-Tools-Addison-Wesley-Prof... (the newer Brendan Gregg book about BPF, stellar)

The Art of Computer Systems Performance Analysis: https://www.cse.wustl.edu/~jain/books/perfbook.htm (no longer available from Amazon, but is available direct from publisher. This is basically the one book you should read about creating and structuring benchmarks or performance tests)

I guess that's a "reading list", but this is just a small part of what you need to know to excel in systems operations.

I would say for the typical software developer writing web applications, the most important thing to know is how databases work and how networking works, since these are going to be the primary items affecting your application performance. But there's obviously topics not included in this list that are also worth understanding, such as browser/DOM internals, how caching and CDNs work, and web-specific optimizations that can be achievable with HTTP/2 or QUIC.

For the average software developer writing desktop applications, I'd say make sure you /really/ understand OS internals... at the base everything you do on a computer system is based on what the OS provides to you. Even though you are abstracted (possibly many layers) away from this, being able to peel back the layers and understand what's /really/ happening is essential to writing high-quality application code that is performant and secure, as well as making you a champ at debugging issues.

If you're trying to get into systems operations as a field, this is just a brush over the top surface and there's a lot deeper diving required.

⬐ planet-and-halo

Appreciate the recs. Thanks very much for taking the time to write a detailed response.

Advanced Programming in the Unix Environment - Stevens, Rago [0]

Unix Network Programming - Stevens [1]

[0] https://www.amazon.com/Advanced-Programming-UNIX-Environment...

[1] https://www.amazon.com/Unix-Network-Programming-Sockets-Netw...

I once did some research on networking and related stuff - it was not my primary topic, but I remember a few things.

During my reading, I found one of the best (as in readable) books was Doug Comer's "Internetworking With TCP/IP vol. 1" - an excellent theoretical reference. [1] However, skip the other volumes from Doug Comer (I think there are 3 volumes).

For writing practical applications, Richard Stevens' "Unix Network Programming" [2] is usually recommended, I didn't find it an easy read though. Perhaps others can pitch in.

For both the books suggested, getting a used old copy for cheap is a good idea because the core information was already there even in the first editions.

Finally, read up on PlanetLab [3]. Its a fascinating project - a small scale internet built on top of a subset of nodes contributed by universities and research organizations across the globe - that people can contribute to, and if you actually manage to get into the developer's list and make a contribution, you can quite honestly claim to have pushed the state of the art forward.

And lastly, be prepared to spend a good amount of time - I don't think it will be a fast or easy process. For whatever reason, I have found that the community around this work to be a little small, especially in comparison to how much it permeates pretty much everyone's life.

[1] https://www.amazon.com/Internetworking-TCP-Vol-1-Principles-...

[2] https://www.amazon.com/Unix-Network-Programming-Sockets-Netw...

[3] http://svn.planet-lab.org/

OK, if you don't have any real experience in low-level embedded coding (relevant to device drivers), RTOS or OS design in general, file systems, data structures, algorithms, interfaces, etc. And, if you have "hobby level" experience with Assembler, C and C++. And, if your intent is to write a desktop OS, from the ground up, without making use of existing technologies, drivers, file systems, memory management, POSIX, etc. Here's a list of books that could be considered required reading before you can really start to write specifications and code. Pick twenty of these and that might be a good start.

In no particular order:

1- http://www.amazon.com/C-Programming-Language-2nd-Edition/dp/...

2- http://www.amazon.com/The-Answer-Book-Solutions-Programming/...

3- http://www.amazon.com/The-Standard-Library-P-J-Plauger/dp/01...

4- http://www.amazon.com/C-Traps-Pitfalls-Andrew-Koenig/dp/0201...

5- http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp...

6- http://www.amazon.com/Data-Structures-In-Noel-Kalicharan/dp/...

7- http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/...

8- http://www.amazon.com/Mastering-Algorithms-C-Kyle-Loudon/dp/...

9- http://www.amazon.com/Code-Complete-Practical-Handbook-Const...

10- http://www.amazon.com/Design-Patterns-Elements-Reusable-Obje...

11- http://www.amazon.com/The-Mythical-Man-Month-Engineering-Ann...

12- http://www.amazon.com/The-Programming-Language-4th-Edition/d...

13- http://www.amazon.com/The-Standard-Library-Tutorial-Referenc...

14- http://www.amazon.com/API-Design-C-Martin-Reddy/dp/012385003...

15- http://www.amazon.com/The-Linux-Programming-Interface-Handbo...

16- http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

17- http://www.amazon.com/System-Programming-Unix-Adam-Hoover/dp...

18- http://www.amazon.com/Memory-Programming-Concept-Frantisek-F...

19- http://www.amazon.com/Memory-Management-Implementations-Prog...

20- http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl...

21- http://www.amazon.com/PCI-System-Architecture-4th-Edition/dp...

22- http://www.amazon.com/Universal-Serial-System-Architecture-E...

23- http://www.amazon.com/Introduction-PCI-Express-Hardware-Deve...

24- http://www.amazon.com/Serial-Storage-Architecture-Applicatio...

25- http://www.amazon.com/SATA-Storage-Technology-Serial-ATA/dp/...

26- http://www.amazon.com/Beyond-BIOS-Developing-Extensible-Inte...

27- http://www.amazon.com/Professional-Assembly-Language-Program...

28- http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...

29- http://www.amazon.com/Version-Control-Git-collaborative-deve...

30- http://www.amazon.com/Embedded-Software-Primer-David-Simon/d...

31- http://www.amazon.com/Programming-Embedded-Systems-C/dp/1565...

32- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

33- http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...

34- http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...

35- http://www.amazon.com/Design-Operating-System-Prentice-Hall-...

36- http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...

37- http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-...

38- http://www.amazon.com/TCP-IP-Illustrated-Vol-Implementation/...

39- http://www.amazon.com/TCP-Illustrated-Vol-Transactions-Proto...

40- http://www.amazon.com/User-Interface-Design-Programmers-Spol...

41- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

42- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

43- http://www.amazon.com/Programming-POSIX-Threads-David-Butenh...

44- http://www.intel.com/p/en_US/embedded/hwsw/software/hd-gma#d...

45- http://www.intel.com/content/www/us/en/processors/architectu...

46- http://www.intel.com/p/en_US/embedded/hwsw/hardware/core-b75...

47- http://www.hdmi.org/index.aspx

48- http://en.wikipedia.org/wiki/Digital_Visual_Interface

49- http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-...

50- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

51- http://www.amazon.com/Python-Programming-Introduction-Comput...

52- http://www.amazon.com/Practical-System-Design-Dominic-Giampa...

53- http://www.amazon.com/File-Systems-Structures-Thomas-Harbron...

54- ...well, I'll stop here.

Of course, the equivalent knowledge can be obtained by trial-and-error, which would take longer and might result in costly errors and imperfect design. The greater danger here is that a sole developer, without the feedback and interaction of even a small group of capable and experienced programmers could simply burn a lot of time repeating the mistakes made by those who have already trenched that territory.

If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.

⬐ tagada

The guy don't want to write a perfectly designed OS, neither he wants to design a RTOS by the way ... He didn't even specify whether his OS would be multitask. If not, no need for any scheduling at all, huh ... he just "writes his own OS". It's a fun experience, very rich and very teaching. But at least now, we all can admire the wideness of all your "unlimited knowledge" especially in terms of titles of books. Afterall, no needs for creating a new OS, if you get all your inspiration from things that already exists. You will end with Yet Another Nix ... Not sure he wants a YanOS, though.

Without talking about filesystems, drivers, memory managment or existing norms and standards (which it seems he wants to avoid ... which is not that stupid ... all having been designed 40 years ago ... who knows maybe he'll have a revolutionary idea, beginning from scratch), going from scratch with no exact idea of where you go could be a good thing. Definitely. You solve the problem only when you face them. Step by step, sequentially. Very virile, man.

I would advice the guy to start obviously with the boot loader. Having a look at the code of GRUB (v1), LILO, or better yet, on graphical ones (BURG, GAG or this good one XOSL which is completely under GUI - mouse pointer, up to 1600x1200 screens, and many supported filesystems, written in ASM ... it could actually be a hacking basis for a basic OS on itself) could be a great source of inspiration, and beginning to play directly with the graphical environment.

You could also have a look on these things http://viralpatel.net/taj/tutorial/hello_world_bootloader.ph... and of course on Kolibri OS http://kolibrios.org/en/?p=Screenshots

⬐ tagada

Blah, blah, blah ...

Better advice would be: Try, make mistakes, learn from them, continue.

Way, way, way less discouraging, pessimistic and above all pedantic.

⬐ derefr

> If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.

I've always wondered if there could be created some way to skip this step in [research] OS prototyping, by creating a shared library (exokernel?) of just drivers, while leaving the "design decisions" of the OS (system calls, memory management, scheduling, filesystems, &c.--you know, the things people get into OS development to play with) to the developer.

People already sort of do this by targeting an emulator like VirtualBox to begin with--by doing so, you only (initially) need one driver for each feature you want to add, and the emulator takes care of portability. But this approach can't be scaled up to a hypervisor (Xen) or KVM, because those expect their guest operating systems to also have relevant drivers for (at least some of) the hardware.

I'm wondering at this point if you could, say, fork Linux to strip it down to "just the drivers" to start such a project (possibly even continuing to merge in driver-related commits from upstream) or if this would be a meaningless proposition--how reliant are various drivers of an OS on OS kernel-level daemons that themselves rely on the particular implementation of OS process management, OS IPC, etc.? Could you code for the Linux driver-base without your project growing strongly isomorphic structures such as init, acpid, etc.?

Because, if you could--if the driver-base could just rely on a clean, standardized, exported C API from the rest of the kernel, then perhaps (and this is the starry-eyed dream of mine) we could move "hardware support development" to a separate project from "kernel development", and projects like HURD and Plan9 could "get off the ground" in terms of driver support.

⬐ robomartin

A lot depends on the platform. If the OS is for a WinTel motherboard it is one thing. If, however, the idea is to bypass driver development for a wide range of platforms it gets complicated.

In my experience one of the most painful aspects of bringing up an OS on a new platform is exactly this issue of drivers as well as file systems. A little google-ing quickly reveals that these are some of the areas where one might have to spend big bucks in the embedded world in order to license such modules as FFS (Flash File System) with wear leveling and other features as well as USB and networking stacks. Rolling your own as a solo developer or even a small team could very well fit into the definition of insanity. I have done a good chunk of a special purpose high-performance FFS. It was an all-absorbing project for months and, realistically, in the end, it did not match all of the capabilities of what could be had commercially.

This is where it is easy to justify moving into a more advanced platform in order to be able to leverage Embedded Linux. Here you get to benefit and leverage the work of tens of thousands of developers devoted to scratching very specific itches.

The down-side, of course, is that if what you need isn't implemented in the boad support package for the processor you happen to be working with, well, you are screwed. The idea that you can just write it yourself because it's Linux is only applicable if you or your team are well-versed in Linux dev at a low enough level. If that is not the case you are back to square one. If you have to go that route you have to hire an additional developer that knows this stuff inside out. That could mean $100K per year. So now your are, once again, back at square one: hiring a dev might actually be more exoensive than licensing a commercial OS with support, drivers, etc.

I was faced with exactly that conundrum a few years ago. We ended-up going with Windows CE (as ugly as that may sound). There are many reasons for that but the most compelling one may have been that we could identify an OEM board with the right I/O, features, form factor, price and full support for all of the drivers and subsystems we needed. In other words, we could focus on developing the actual product rather than having to dig deeper and deeper into low-level issues.

It'd be great if low level drivers could be universal and platform independent to the degree that they could be used as you suggest. Obviously VM-based platforms like Java can offer something like that so long as someone has done the low-level work for you. All that means is that you don't have to deal with the drivers.

To go a little further, part of the problem is that no standard interface exists to talk to chips. In other words, configuring and running a DVI transmitter, a serial port and a Bluetooth I/F are vastly different even when you might be doing some of the same things. Setting up data rates, clocks, etc. can be day and night from chip to chip.

I haven't really given it much thought. My knee-jerk reaction is that it would be very hard to crate a unified, discoverable, platform-independent mechanism to program chips. The closest one could possibly approach this idea would be if chip makers were expected to provide drivers written to a common interface. Well, not likely or practical.

Not an easy problem.

⬐ derefr

> It'd be great if low level drivers could be universal and platform independent to the degree that they could be used as you suggest. Obviously VM-based platforms like Java can offer something like that so long as someone has done the low-level work for you. All that means is that you don't have to deal with the drivers.

Another thought: if not just a package of drivers, then how stripped down (for the purpose of raw efficiency) could you make an operating system intended only to run an emulator (IA32, JVM, BEAM, whatever) for "your" operating system? Presumably you could strip away scheduling, memory security, etc. since the application VM could be handling those if it wanted to. Is there already a major project to do this for Linux?

This post only contains non-aff links

While I'm certainly good at C, and have been the guy people asked the C and C toolchain questions at a couple places I've worked, I'd say you're in for more than 1 book to grok the topics you've expressed interest in, and haven't met the single book that does all you want.

1: If you're a python programmer, I do suggest staying with pure C and AWAY from C++. You have little use case for it if you can mix python and C and it is literally pain incarnate while you do not know every little bit of it and/or work with people who do not know every little bit of it.

2: Secondly, a single book on C will not get you where you're going. You will need several.

3: I'd stick with K&R for reference, but then go to the following two books for threading and all the other stuff like networking:

You need these two books to get down C and it's ecosystem (make, autotools, etc): Advanced Unix Programming by Rochkind (Really good explanation of signals especially): http://www.amazon.com/Advanced-UNIX-Programming-Marc-Rochkin...

Gnu/Linux Application Programming by M. Tim Jones (Recovers some of the same stuff as AUP, but does better with the toolchain): http://www.amazon.com/GNU-Linux-Application-Programming/dp/1...

If you want to go more advanced, this is likely the correct book: Advanced Programming in the Unix Environment: http://www.amazon.com/Advanced-Programming-UNIX-Environment-...

Although if you're diving deep into networking, this is likely the correct book: Unix Networking Programming: Vol I by Stevens:

http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...