Pearson/Addison-Wesley Professional’s The Design and Implementation of the FreeBSD Operating System is a long awaited update to a successful and authorative guide to the FreeBSD kernel. The second edition covers all major improvements between FreeBSD version 5 and 11 and, according to the publisher, it has been extensively rewritten for one-third of its content, while another one-third is completely new.
The book’s second edition is significantly larger than the first, weighing in at more than 900 pages, and covers new FreeBSD features such as virtualization, sandboxing, NFSv4 and ZFS support, and more.
The first chapter in the book provides a recapitulation of FreeBSD’s history, while the second provides a succinct overview of all the major components of the FreeBSD kernel. The rest of the chapters in the book follow chapter two’s structure to lead the reader into great detail about each topic.
The book takes an approach to FreeBSD from inside out, starting with kernel services, then moving to process and memory management, I/O and devices, filesystems, IPC and network protocols, and finally system startup and shutdown. The book provides dense, technical information in a clear way, with lots of pseudo-code, diagrams, and tables to illustrate the main points. Each chapter starts with a general introduction to the topic at hand, then goes into a detailed explanation.
While going sufficiently low-level on FreeBSD, the book still provides enough insight on operating system design and concepts to make the book appeal not only to Free BSD developers but also to anyone looking for a thorough, detailed discussion of a modern operating system.
InfoQ has spoken with Marshall Kirk McKusick, one of the book’s authors.
InfoQ: The Design and Implementation of the FreeBSD Operating System describes FreeBSD “under the hood” and provides a wealth of advanced material. Which is the target reader for your book? What can one expect to learn about FreeBSD by reading this book?
Marshall: Our book is of direct use to the professionals who work with FreeBSD systems. Applications developers can learn how to effectively and efficiently interface to the system; systems programmers can learn how to extend, enhance, and interface to the system; system administrators without direct experience with the FreeBSD kernel can learn how to maintain, tune, and configure the system; and individuals involved in technical and sales support can learn the capabilities and limitations of the system.
Our book provides a broad overview of how the FreeBSD kernel implements its basic services. It will be most useful to those who need to learn how these services are provided. Readers who will benefit from this book include operating-system implementors, system programmers, UNIX application developers, administrators, and curious users. This book is directed to users who have had at least a year of experience using a UNIX-like system. Knowledge of the C programming language is helpful, but not essential. Readers should have an understanding of fundamental algorithms (searching, sorting, and hashing) and data structures (lists, queues, and arrays).
Our book provides a firm background in the FreeBSD kernel. The book starts with an introduction to the kernel and the services that it provides including the locking used for concurrency control. Next, the details of processes are described including process management tasks such as scheduling and signals. The security framework and policies are outlined including Capsicum sandboxing and the FreeBSD jail facility that allows the creation of isolated virtual hosts running on the same system. The description of processes concludes with an extensive description of the management of kernel and process memory. The book then turns to an I/O view of the kernel by introducing the I/O framework and detailing the infrastructure for doing I/O multiplexing and supporting both local and remote filesystems. It then describes how it configures and operates both character devices and bulk data transfer devices such as disks and how it manages virtual devices needed support Xen and bhyve virtualization. The design and implementation of three filesystems is covered: the Fast filesystem (FFS), the Zettabyte filesystem (ZFS), and the Network filesystem (NFS). Next the interprocess communication (IPC/socket) interface is detailed followed by coverage of the layering and implementation of the networking including routing, forwarding, and security of the TCP/IP protocols. It concludes with coverage of the kernel startup procedure. The book emphasizes code organization, data structure navigation, and algorithms rather than reviewing the kernel code itself. It does not cover the machine specific parts of the system such as the implementation of device drivers.
InfoQ: How would you position FreeBSD among other available open source Unix-like OSes?
Marshall: FreeBSD is the most widely distributed of the three main BSD distributions (the other two being OpenBSD and NetBSD). FreeBSD is widely used to support the core infrastructure of many companies worldwide including NetFlix, WhatsApp, Yahoo!, Juniper Networks, Network Appliance, EMC / Isilon, and Apple’s Darwin, the base operating system of Mac OS X. Because it can be built with a small footprint, it is also seeing increased use in embedded applications.
The primary open-source alternative to FreeBSD is Linux. The licensing terms of FreeBSD do not require the distribution of changes and enhancements to the system which makes the FreeBSD license friendly to businesses as well as individual users. The licensing terms of Linux require that all changes and enhancements to the kernel be made available in source form at minimal cost. Thus, companies that need to control the distribution of their intellectual property increasingly are building their products using FreeBSD.
InfoQ: FreeBSD has been the most popular BSD system out there for quite some time now. What do you think are the main factors that have led the project to this success?
Marshall: From its founding in 1993, the FreeBSD project has had a goal of making its distributions as easily accessible and installable to as wide a group of companies and individuals as possible.
Another feature of FreeBSD that has lead to its success is that unlike many other open-source projects that have a permanent set of folks in charge (for example Linus Torvolds and his trusted lieutenants in Linux), FreeBSD’s system of governance is structured to be self-organizing, to allow motivated folks to rise to the top roles, and is set up to gracefully handle the inevitable turnover of its developers.
At the outer ring, there are 5000 to 6000 developers, each of whom works on some part of the system such as maintaining the FreeBSD kernel, continuing development of the 1000 core FreeBSD utilities, writing FreeBSD documentation, and updating other open-source software in the FreeBSD ports collection. Developers are able to access the source-code repository, but they are not permitted to change it. Instead, they must work with a committer or file a problem report to get their changes added to the system.
At the next ring in there are currently 300 to 400 committers. Like the developers, most of them specialize in some part of the system. Unlike the developers, they are permitted to make changes to those parts of the source-code repository in which they have been authorized to work. All nontrivial changes should be reviewed by one or more other committers before being checked into the source tree. Most committers are doing work of their own as well as reviewing and committing the work of several developers.
Nomination for advancement from developer to committer is done by the existing committers. Most commonly a developer will be nominated by the committer with whom they have been working. The nomination, along with a description and evaluation of past work and an initial scope of new work, is sent to the core team for approval.
At the center of the project is the core team. The core team is composed of nine people who are elected every 2 years. The candidates for the core team come from the committers and the committers elect the core team. The core team acts as the final gatekeepers of the source code. They monitor what is being committed and resolve conflicts if two or more committers cannot agree on how to solve a particular problem. The core team also approves the advancement of developers to committers and (in rare circumstances) temporarily or permanently evicts someone from the committer group. The usual reason for departure from the committer group is inactivity (making no changes to the system for more than a year).
InfoQ: What makes FreeBSD unique, or, if you prefer, what is the reason to choose FreeBSD over the other BSD systems?
Marshall: Unlike the other BSDs and most open-source projects in general, FreeBSD has maintained each of its major releases for a minimum of five years providing bug fixes and security patches over the entire support period. This extended support period has allowed companies to build products based on these releases without concern that they will lose support for the release on which their product is built.
InfoQ: What are the most compelling feature that FreeBSD has added in the last few years?
Marshall: A compelling feature added to FreeBSD recently is the Capsicum interface to allow fine-grain sandboxing of code of unknown provenance. Capsicum allows very tight bounding of applications with security implications to ensure that they are not able to access, modify, or steal any unapproved information.
Another compelling recent addition to FreeBSD is the importation of the ZFS filesystem from Open Solaris. Unlike Linux which does not allow the importation of ZFS due to licensing conflicts, FreeBSD has fully integrated ZFS into the kernel and is able to deliver ZFS’s full feature set and performance.
InfoQ: In a 2012 interview, Dru Lavigne, Director at the FreeBSD Foundation, noted that “The FreeBSD project has been very fortunate in its ability to attract both funding for security research and for collaboration with academia in the field of security.” What have been the implications of this for FreeBSD?
Marshall: The Capsicum work noted above started as a research project at Cambridge University (UK). By virtue of the researchers using FreeBSD as their development platform, it was straightforward to adapt that research into the production interface, ancillary libraries, and support programs provided in FreeBSD today. As Dru noted, the FreeBSD Foundation was instrumental in raising money to pay for the conversion of the research prototype to production code, and in finding, funding, and managing the folks that did the conversion work.
InfoQ: Do you think that FreeBSD can make a good reference to learn about Operating Systems? If yes, what advantages as a learning tool does it provide?
Marshall: All three authors are working to expand the use of FreeBSD as a teaching tool. We are developing lesson plans, lecture notes, and laboratory experiments for use in teaching advanced undergraduate and first or second year graduate courses. As these courses are developed we are announcing them and making them available on the www.teachbsd.com web site. We are just getting started on this project, but hope to have a rich set of materials within a year or two.
The FreeBSD Foundation has recently embarked on a project to bring FreeBSD into high school computer science curriculums. While this project is just getting underway, the hope is to increase the interest of high school students in computer technology in general and FreeBSD in particular.
InfoQ: Where is FreeBSD headed? What can we expect from future versions?
Marshall: The project’s roadmap is driven by developer input at FreeBSD developer summits held at least twice a year in conjunction with the BSDCan (in Ottawa Canada) and the EuroBSD (rotating through EU countries) conferences. It is also driven by the FreeBSD user community that meets 3–4 times per year at FreeBSD vendor summits.
The traditional FreeBSD releases have been targetted at servers and embedded systems. The PC-BSD distribution is targetted at desktop users. It takes the current FreeBSD distribution and pulls together a set of packages from the ports collection (a desktop, browser, mail reader, etc) to make an easy-to-install desktop (or laptop) system.
This Q&A is based on the book, ‘The Design and Implementation of the FreeBSD Operating System’, 2nd Ed. authored by Marshall Kirk McKusick, George V. Neville-Neil and Robert N.M. Watson, published by Pearson/Addison-Wesley Professional, Sept. 2014, ISBN 978–0–321–96897–5. For more info please visit the publisher site.
In September 2015, the publisher will also release McKusick's new video training: "Introduction to the FreeBSD Open-Source Operating System LiveLessons".
About the Book Authors
Marshall Kirk McKusick writes, consults, and teaches classes on UNIX- and BSD-related subjects. While at the University of California, Berkeley, he implemented the 4.2BSD fast filesystem. He was research computer scientist at the Berkeley Computer Systems Research Group (CSRG), overseeing development and release of 4.3BSD and 4.4BSD. He is a FreeBSD Foundation board member and a long-time FreeBSD committer. Twice president of the Usenix Association, he is also a member of ACM, IEEE, and AAAS.
George V. Neville-Neil hacks, writes, teaches, and consults on security, networking, and operating systems. A FreeBSD Foundation board member, he served on the FreeBSD Core Team for four years. Since 2004, he has written the “Kode Vicious” column for Queue and Communications of the ACM. He is vice chair of ACM’s Practitioner Board and a member of Usenix Association, ACM, IEEE, and AAAS.
Robert N.M. Watson is a University Lecturer in systems, security, and architecture in the Security Research Group at the University of Cambridge Computer Laboratory. He supervises advanced research in computer architecture, compilers, program analysis, operating systems, networking, and security. A FreeBSD Foundation board member, he served on the Core Team for ten years and has been a committer for fifteen years. He is a member of Usenix Association and ACM.