How to upstream code to open source projects

Code commonly flows downstream, from an open-source project into an organization’s own products. Upstreaming is the process of reversing that flow—contributing code back to an open-source project. The value proposition of upstreaming includes harnessing the strength of an open-source community to examine code, find and fix problems, and add their own features that make the code more valuable to everyone using it.

As someone who has been deeply involved with open-source projects for many years—I’ve committed code to the open source FreeBSD operating system project for over a decade now, served on that project’s core team for two terms, contributed to open source ZFS, and co-written two books about ZFS—I’ve seen countless organizations meet the challenges and reap the substantial benefits of upstreaming. In short, contributed code that becomes part of a mainline open-source project receives shared maintenance, active development, and extension, with other members across the community often adding value that goes well beyond the initial contribution.

In fact, upstreaming is how most major new features get introduced to open-source projects like FreeBSD and ZFS, among many others. Let’s look at the specific benefits of upstreaming and the best practices for directly unlocking its advantages.

Benefits of upstreaming

A commitment to upstreaming results in higher-quality code, a simpler development process, reduced maintenance burdens, and greater project sustainability.

Higher-quality code

Developing code with the intention of upstreaming it acts as a forcing function for quality. For development teams and leaders who must constantly resist the mindsets of “just get code out the door” and “it’s fine if the code is ugly because no one will see it,” a commitment to upstreaming and undergoing the upstream review process provides an unassailable guardrail.

At the same time, when code begins development intended for internal eyes and internal usage, then needs to have generality bolted on later, the result is usually less satisfying features (and greater maintenance burdens) going forward. Adhering to upstream coding practices and style requirements throughout development results in much higher-quality code—code that’s built to provide value not just within an organization but to everyone.

Easier development

One of the biggest challenges of using open-source technologies emerges when organizations make the error of developing for the open-source software version they’re currently using (which can be several years old) rather than for the latest version. That inefficient (if not wholly ineffective) practice means rebasing to catch up with all of the other software changes that may impact your new feature, and facing all of the technical debt that lagging behind creates.

Developing for the latest main version of open-source software and then backporting as necessary ensures that the upstreaming process does not get bogged down by changes that are happening upstream during development, and means far less work in the end as you are not trying to rebase against a rapidly moving target.

Less maintenance

Again, organizations that build new features on an older local branch of open-source software and then attempt to upstream to a newer version tend to struggle because they require subject-matter expertise on all of the areas of the software that have changed over that period. Expertise availability is a key upstreaming challenge. For example, a development team that isn’t yet closely familiar with the FreeBSD operating system will find it challenging to address all of the surface area touched by the upstream changes they have to rebase against to catch their code up to mainline before they can upstream their patch changes.

Committing to upstreaming from the start allows for a far more concentrated focus on what your patch is actually doing, which is likely in your subject-matter wheelhouse. Once changes are integrated upstream and part of the mainline open-source software, maintenance is no longer your problem alone. It is a communal effort.

Project sustainability

With upstream contributions maintained by the community, new features attract more users and more contributions. That virtuous cycle grows open-source projects, communities, and effective software functionality to your greater benefit.

Best practices for upstreaming a patch

Based on my experience upstreaming patches to open-source projects, the process outlined below provides an overview of how to successfully upstream a feature.

Step 1: Create the candidate patch

Importantly, you must make sure your patch is generally useful and valuable to other users of the project you’re contributing to. It’s not a great practice to upstream code that’s overly specific to your application and of little value to anyone else. That’s just throwing code over the wall and hoping somebody else will maintain it. Extending a feature to offer general value can take some extra work, but that usefulness is essential to the quality of your contribution and the community response.

This leads to another key upstreaming challenge: advocacy. Making your contributed feature a success requires community excitement. They need to understand why they should adopt it and how they can benefit from it. Knowing the community well and engaging early (even before development) to understand others’ needs goes a long way toward overcoming this challenge. The more your upstreamed code is used, the more likely it will thrive and expand (versus being removed or deprecated down the road). Engagement is often the single most important factor that determines where community maintenance effort is applied.

Step 2: Submit the patch for review by other upstream contributors

Provide sufficient documentation for others to make use of the code, along with tests that ensure your code won’t break other features, and that any regression in the functionality of your feature caused by other changes will be detected. Naturally, also include a clear description of your patch, one that explains its purpose and its value for the reviewers.

Other upstream contributors then review your patch and provide feedback, which you must address appropriately.

Step 3: Reviewers integrate the patch into the upstream branch

The patch may also be merged to the stable branch or branches of the open source software. Future releases of the upstream software will include your changes. Advocating for backports to the stable branches and helping to maintain them will help to ensure that those backports are available for your products that use those stable branches and that you have an easy upgrade path in the future.

Adopt an ‘upstream first’ mentality

Upstreaming forces your organization to create code that’s not just good enough for internal use, but good enough to serve an entire open-source community and earn their lasting support. By designing features that effectively serve community needs as well as your own, the upstreaming and review process becomes that much easier, and the features your organization relies upon can grow and flourish far beyond what your team could do alone. That’s the power of open source.

Allan Jude is the FreeBSD Engineering Manager of Klara Inc., which provides enterprise services and support for open source infrastructures. He has been a FreeBSD Committer since 2014.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.