HN Books @HNBooksMonth

It amazes me sometimes how much of a dying breed systems engineers are. When I was coming up as a SWE I worked for a series of systems engineers, so I learned software from a systems engineering perspective and it's been invaluable as distributed systems have gotten bigger and more OS-like.

If you're wanting to dive a little deeper than this guide touches check out:

- https://man7.org/tlpi/ - Really good for understanding how to build applications within a Linux ecosystem.

- https://www.amazon.com/Linux-Kernel-Development-Robert-Love/... - Really good for understanding why Linux is the way it is. The Kernel this book was written on is awful old, but the principles shine through.

⬐ alex_sf

We're just calling ourselves SREs now. It pays better.

⬐ kodah

I'm a bit conflicted with that title. I consider myself a SWE with an emphasis on systems, but more specifically Linux. From what I've seen a lot of the SRE market is (mostly) the same old Ops scene with new badges.

⬐ GauntletWizard

There's a ton of that, just like Devops was a "hot market" for a while and then was diluted. We'll have a new name in a few years, or we'll find some other way of distinguising ourselves... I am not one to promote unions, but at some point we do need a systems Engineering, ENGINEERING, Iron Ring, because our tools are increasingly infrastructural in the same way bridges are.

> Let's not even talk about pre-fork parallelism which is probably the simplest way to use multiple cores when dealing with independent tasks but hardly anybody seems to use these days. I wonder if most devs even realize that it's an option.

Speaking from experience: they don't. I recently picked up a Linux kernel development book [1] and after four or five chapters I was able to eliminate several bottlenecks and eliminate latency edge cases in personal projects with a greater understanding of the scheduler algorithm and process priority/affinity/yields.

I think the problem is that the vast majority of developers work on top of cross-platform runtimes that hide any detail that can't be easily implemented for all operating systems. It's been more than a decade since I've used Unix sockets directly in any language and I don't how to set SO_REUSEPORT in Rust. I'd probably have to implement it using `TCPListener::as_raw_fd()` and `unsafe libc::setsockopt`? It's just easier stay sane within the bumpers.

[1] https://www.amazon.com/Linux-Kernel-Development-Robert-Love/...

I read this book few years ago:

Linux Kernel Development by Robert Love. It was not too hard read for newbie like me. I was just dabbling in understanding few things about OS in general.

https://www.amazon.com/Linux-Kernel-Development-Robert-Love/... https://rlove.org/

Speaking of the Linux kernel related books, I'd probably mention, Understanding The Linux Kernel [0] and Linux Kernel Development [1]. The LDDv4 is not going to happen any time soon; it seems there are no plans for a new edition at all.

  [0] https://www.amazon.com/Understanding-Linux-Kernel-Third-Daniel/dp/0596005652
  [1] https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468

-ss

If you want to become a professional and not just a dabbler I would recommend reading some of the following books I have in my bookshelf:

[0] RHCSA & RHCE Training and Exam Preparation Guide by Asghar Ghori. This book will help insure you know your stuff as your system engineer/administrator wise.

[1] A Practical Guide to Linux Commands, Editor and Shell Programming Third Edition. This book will cover the majority of what you would need and want to know when connecting to a remote linux system over ssh.

If you want to get under the hood and become an expert, the following books should help get you started:

[2] Advanced Programming in the UNIX Environment

[3] The Linux Programming Interface: A Linux and UNIX System Programming Handbook

[4] Linux Kernel Development 3rd Edition

To get a nice general overview and get up and going quickly:

[5] How Linux works: What every superuser should know

[6] The Linux Command Line

[7] Python Crash Course

[8] Automate the boring stuff with Python. This is a great book to help you think about how to automate most of the repetitive things you will end up doing on a regular basis.

[0] https://www.amazon.com/RHCSA-RHCE-Red-Enterprise-Linux/dp/14...

[1] https://www.amazon.com/Practical-Guide-Commands-Editors-Prog...

[2] https://www.amazon.com/Advanced-Programming-UNIX-Environment...

[3] https://www.amazon.com/Linux-Programming-Interface-System-Ha...

[4] https://www.amazon.com/Linux-Kernel-Development-Robert-Love/...

[5] https://www.amazon.com/How-Linux-Works-Superuser-Should/dp/1...

[6] https://www.amazon.com/Linux-Command-Line-Complete-Introduct...

[7] https://www.amazon.com/Python-Crash-Course-Hands-Project-Bas...

[8] https://www.amazon.com/Automate-Boring-Stuff-Python-Programm...

⬐ grepthisab

This looks like a gold mine! Thanks so much!

⬐ iDemonix

The RHCSA/RHCE stuff is good, I passed the RHCSA a couple years ago and it's a solid set of foundations.

There's a good chat (devopschat.slack.com) that's useful for learning stuff like this.

Hit the books? That's what I do, I'm reading this one for fun now: https://www.amazon.com/Linux-Kernel-Development-Robert-Love/...

It's awesome. I have a library stacked full of software and CS books. Maybe try to get your bachelors at night? I got my master's at night while working.

This is a craft. If you want to be an expert, train like an expert. Realize 3 months of a bootcamp isn't going to cut it, not at least until you've been working for 4, 6, 8 years.

If you can't tell me what inter process communication is, what it's used for, pipes, signals, etc. then I think you have some pretty big gaps in knowledge that preclude you from being an expert at this point.

https://www.amazon.com/Linux-Kernel-Development-Robert-Love/... is very good, as is https://www.amazon.com/Mac-OS-iOS-Internals-Apples/dp/111805...

⬐ JdeBP

Amit Singh's book on MacOS is quite interesting, too. But gtirloni did ask about Linux, for which neither MacOS book is really appropriate.

http://www.amazon.com/dp/0672329468/

I'd also add Love's Linux Kernel Development[2], published in 2010. Great resource overall for someone wanting to go developing the internals of the kernel.

[2] http://www.amazon.com/dp/0672329468

⬐ incision

Yes.

I actually have that book as well. I don't know how I forget to mention it. As I recall, it was bit less dense than the other two.

Linux Kernel Development by Robert Love is an exceptional book.

http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...

The large pages feature (huge pages) still works the same way but the transparent huge pages patch has been integrated recently: http://www.slideshare.net/raghusiddarth/transparent-hugepage...

I personally wouldn't mess with the VM settings the way the author suggests. I don't remember doing that eight years ago anyway even with less RAM on a desktop. On a server you want the working set to fit in RAM anyway and the page cache is already pretty smart.

The best resource is a general kernel book: http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...

in terms of the interaction between memory, the cpu, and virtual memory What every Programmer should know about memory should be required reading: http://www.akkadia.org/drepper/cpumemory.pdf

OK, if you don't have any real experience in low-level embedded coding (relevant to device drivers), RTOS or OS design in general, file systems, data structures, algorithms, interfaces, etc. And, if you have "hobby level" experience with Assembler, C and C++. And, if your intent is to write a desktop OS, from the ground up, without making use of existing technologies, drivers, file systems, memory management, POSIX, etc. Here's a list of books that could be considered required reading before you can really start to write specifications and code. Pick twenty of these and that might be a good start.

In no particular order:

1- http://www.amazon.com/C-Programming-Language-2nd-Edition/dp/...

2- http://www.amazon.com/The-Answer-Book-Solutions-Programming/...

3- http://www.amazon.com/The-Standard-Library-P-J-Plauger/dp/01...

4- http://www.amazon.com/C-Traps-Pitfalls-Andrew-Koenig/dp/0201...

5- http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp...

6- http://www.amazon.com/Data-Structures-In-Noel-Kalicharan/dp/...

7- http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/...

8- http://www.amazon.com/Mastering-Algorithms-C-Kyle-Loudon/dp/...

9- http://www.amazon.com/Code-Complete-Practical-Handbook-Const...

10- http://www.amazon.com/Design-Patterns-Elements-Reusable-Obje...

11- http://www.amazon.com/The-Mythical-Man-Month-Engineering-Ann...

12- http://www.amazon.com/The-Programming-Language-4th-Edition/d...

13- http://www.amazon.com/The-Standard-Library-Tutorial-Referenc...

14- http://www.amazon.com/API-Design-C-Martin-Reddy/dp/012385003...

15- http://www.amazon.com/The-Linux-Programming-Interface-Handbo...

16- http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

17- http://www.amazon.com/System-Programming-Unix-Adam-Hoover/dp...

18- http://www.amazon.com/Memory-Programming-Concept-Frantisek-F...

19- http://www.amazon.com/Memory-Management-Implementations-Prog...

20- http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl...

21- http://www.amazon.com/PCI-System-Architecture-4th-Edition/dp...

22- http://www.amazon.com/Universal-Serial-System-Architecture-E...

23- http://www.amazon.com/Introduction-PCI-Express-Hardware-Deve...

24- http://www.amazon.com/Serial-Storage-Architecture-Applicatio...

25- http://www.amazon.com/SATA-Storage-Technology-Serial-ATA/dp/...

26- http://www.amazon.com/Beyond-BIOS-Developing-Extensible-Inte...

27- http://www.amazon.com/Professional-Assembly-Language-Program...

28- http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...

29- http://www.amazon.com/Version-Control-Git-collaborative-deve...

30- http://www.amazon.com/Embedded-Software-Primer-David-Simon/d...

31- http://www.amazon.com/Programming-Embedded-Systems-C/dp/1565...

32- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

33- http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...

34- http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...

35- http://www.amazon.com/Design-Operating-System-Prentice-Hall-...

36- http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...

37- http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-...

38- http://www.amazon.com/TCP-IP-Illustrated-Vol-Implementation/...

39- http://www.amazon.com/TCP-Illustrated-Vol-Transactions-Proto...

40- http://www.amazon.com/User-Interface-Design-Programmers-Spol...

41- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

42- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

43- http://www.amazon.com/Programming-POSIX-Threads-David-Butenh...

44- http://www.intel.com/p/en_US/embedded/hwsw/software/hd-gma#d...

45- http://www.intel.com/content/www/us/en/processors/architectu...

46- http://www.intel.com/p/en_US/embedded/hwsw/hardware/core-b75...

47- http://www.hdmi.org/index.aspx

48- http://en.wikipedia.org/wiki/Digital_Visual_Interface

49- http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-...

50- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

51- http://www.amazon.com/Python-Programming-Introduction-Comput...

52- http://www.amazon.com/Practical-System-Design-Dominic-Giampa...

53- http://www.amazon.com/File-Systems-Structures-Thomas-Harbron...

54- ...well, I'll stop here.

Of course, the equivalent knowledge can be obtained by trial-and-error, which would take longer and might result in costly errors and imperfect design. The greater danger here is that a sole developer, without the feedback and interaction of even a small group of capable and experienced programmers could simply burn a lot of time repeating the mistakes made by those who have already trenched that territory.

If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.

⬐ tagada

The guy don't want to write a perfectly designed OS, neither he wants to design a RTOS by the way ... He didn't even specify whether his OS would be multitask. If not, no need for any scheduling at all, huh ... he just "writes his own OS". It's a fun experience, very rich and very teaching. But at least now, we all can admire the wideness of all your "unlimited knowledge" especially in terms of titles of books. Afterall, no needs for creating a new OS, if you get all your inspiration from things that already exists. You will end with Yet Another Nix ... Not sure he wants a YanOS, though.

Without talking about filesystems, drivers, memory managment or existing norms and standards (which it seems he wants to avoid ... which is not that stupid ... all having been designed 40 years ago ... who knows maybe he'll have a revolutionary idea, beginning from scratch), going from scratch with no exact idea of where you go could be a good thing. Definitely. You solve the problem only when you face them. Step by step, sequentially. Very virile, man.

I would advice the guy to start obviously with the boot loader. Having a look at the code of GRUB (v1), LILO, or better yet, on graphical ones (BURG, GAG or this good one XOSL which is completely under GUI - mouse pointer, up to 1600x1200 screens, and many supported filesystems, written in ASM ... it could actually be a hacking basis for a basic OS on itself) could be a great source of inspiration, and beginning to play directly with the graphical environment.

You could also have a look on these things http://viralpatel.net/taj/tutorial/hello_world_bootloader.ph... and of course on Kolibri OS http://kolibrios.org/en/?p=Screenshots

⬐ tagada

Blah, blah, blah ...

Better advice would be: Try, make mistakes, learn from them, continue.

Way, way, way less discouraging, pessimistic and above all pedantic.

⬐ derefr

> If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.

I've always wondered if there could be created some way to skip this step in [research] OS prototyping, by creating a shared library (exokernel?) of just drivers, while leaving the "design decisions" of the OS (system calls, memory management, scheduling, filesystems, &c.--you know, the things people get into OS development to play with) to the developer.

People already sort of do this by targeting an emulator like VirtualBox to begin with--by doing so, you only (initially) need one driver for each feature you want to add, and the emulator takes care of portability. But this approach can't be scaled up to a hypervisor (Xen) or KVM, because those expect their guest operating systems to also have relevant drivers for (at least some of) the hardware.

I'm wondering at this point if you could, say, fork Linux to strip it down to "just the drivers" to start such a project (possibly even continuing to merge in driver-related commits from upstream) or if this would be a meaningless proposition--how reliant are various drivers of an OS on OS kernel-level daemons that themselves rely on the particular implementation of OS process management, OS IPC, etc.? Could you code for the Linux driver-base without your project growing strongly isomorphic structures such as init, acpid, etc.?

Because, if you could--if the driver-base could just rely on a clean, standardized, exported C API from the rest of the kernel, then perhaps (and this is the starry-eyed dream of mine) we could move "hardware support development" to a separate project from "kernel development", and projects like HURD and Plan9 could "get off the ground" in terms of driver support.

⬐ robomartin

A lot depends on the platform. If the OS is for a WinTel motherboard it is one thing. If, however, the idea is to bypass driver development for a wide range of platforms it gets complicated.

In my experience one of the most painful aspects of bringing up an OS on a new platform is exactly this issue of drivers as well as file systems. A little google-ing quickly reveals that these are some of the areas where one might have to spend big bucks in the embedded world in order to license such modules as FFS (Flash File System) with wear leveling and other features as well as USB and networking stacks. Rolling your own as a solo developer or even a small team could very well fit into the definition of insanity. I have done a good chunk of a special purpose high-performance FFS. It was an all-absorbing project for months and, realistically, in the end, it did not match all of the capabilities of what could be had commercially.

This is where it is easy to justify moving into a more advanced platform in order to be able to leverage Embedded Linux. Here you get to benefit and leverage the work of tens of thousands of developers devoted to scratching very specific itches.

The down-side, of course, is that if what you need isn't implemented in the boad support package for the processor you happen to be working with, well, you are screwed. The idea that you can just write it yourself because it's Linux is only applicable if you or your team are well-versed in Linux dev at a low enough level. If that is not the case you are back to square one. If you have to go that route you have to hire an additional developer that knows this stuff inside out. That could mean $100K per year. So now your are, once again, back at square one: hiring a dev might actually be more exoensive than licensing a commercial OS with support, drivers, etc.

I was faced with exactly that conundrum a few years ago. We ended-up going with Windows CE (as ugly as that may sound). There are many reasons for that but the most compelling one may have been that we could identify an OEM board with the right I/O, features, form factor, price and full support for all of the drivers and subsystems we needed. In other words, we could focus on developing the actual product rather than having to dig deeper and deeper into low-level issues.

It'd be great if low level drivers could be universal and platform independent to the degree that they could be used as you suggest. Obviously VM-based platforms like Java can offer something like that so long as someone has done the low-level work for you. All that means is that you don't have to deal with the drivers.

To go a little further, part of the problem is that no standard interface exists to talk to chips. In other words, configuring and running a DVI transmitter, a serial port and a Bluetooth I/F are vastly different even when you might be doing some of the same things. Setting up data rates, clocks, etc. can be day and night from chip to chip.

I haven't really given it much thought. My knee-jerk reaction is that it would be very hard to crate a unified, discoverable, platform-independent mechanism to program chips. The closest one could possibly approach this idea would be if chip makers were expected to provide drivers written to a common interface. Well, not likely or practical.

Not an easy problem.

⬐ derefr

> It'd be great if low level drivers could be universal and platform independent to the degree that they could be used as you suggest. Obviously VM-based platforms like Java can offer something like that so long as someone has done the low-level work for you. All that means is that you don't have to deal with the drivers.

Another thought: if not just a package of drivers, then how stripped down (for the purpose of raw efficiency) could you make an operating system intended only to run an emulator (IA32, JVM, BEAM, whatever) for "your" operating system? Presumably you could strip away scheduling, memory security, etc. since the application VM could be handling those if it wanted to. Is there already a major project to do this for Linux?

The most thorough treatment is Bovet & Cesati (944 pages): http://www.amazon.com/Understanding-Linux-Kernel-Third-Editi...

A good "gentle introduction" book is the Love book (440 pages): http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...

⬐ pwaring

Isn't Bovet & Cesati a bit out of date now? I used to have a copy, but it was published in 2005, which means it was probably written in 2004.

Also, it's not kernel-specific, but this book covers a lot of system programming concepts (expensive though):

http://www.amazon.co.uk/Linux-Programming-Interface-System-H...

I found Robert Love's Linux Kernel Development to be quite useful.

http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...

⬐ thejteam

I have this book and think it's great. It covers Linux 2.6, which happens to be the version of the kernel used in CentOS6.2 which is what I use at work. What are the major differences between 2.6 and the 3 line. Specifically, what is the major difference that caused the numbering scheme to go from 2 to 3?

⬐ kenny_r

Basically, Linus just felt that the minor release numbers were getting too big. He dedided to not call that release 2.6.40 and call it 3.0.

⬐ pingswept

The jump to 3 was made arbitrarily after 2.6.39. Here's the announcement last summer from Linus: https://lwn.net/Articles/452531/

As a start, read and understand http://www.amazon.com/Understanding-Linux-Kernel-Third-Editi... (that's a very technical and in-depth book) and http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d... (that's a more gentle overview book) then dive in with http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl... and write a very simple file system.

Learn C as necessary.

The FreeBSD kernel book is worth a look too: http://www.amazon.com/Design-Implementation-FreeBSD-Operatin...

I have been keeping a list of books I used to augment my CS Masters Degree courses on various topics, here are the relevant ones I have found useful for the topics you have listed:

--Computer Organization--:

Computer Systems: A Programmer's Perspective http://www.amazon.com/Computer-Systems-Programmers-Randal-Br...

I liked this much better than Computer Organization and Design by Patterson and Hennessy which everyone has encountered at some point. The developer-centric view was very cool.

--Computer Security--:

Kernel Exploitation: Attacking the Core http://www.amazon.com/Guide-Kernel-Exploitation-Attacking-Co...

Most 'hacking' books are goofy. This one is very good and doubles nicely as a hackers operating systems text.

Web Application Hackers Handbook http://www.amazon.com/Web-Application-Hackers-Handbook-Disco...

Very nice overview for web concerns.

--Operating Systems-:

Operating System Design and Implementation http://www.amazon.com/Operating-Systems-Design-Implementatio...

I don't agree with Tanenbaum's views on micro vs. monolithic kernels but this book is a great mix of theory and implementation.

Linux Kernel Devleopment http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...

I used this to get a feel for the monolithic implementations of topics covered by Tanenbaum.

--Networking--:

TCP/IP Illustrated Series. More than you would ever want to know.