Somehow OpenSearch has succeeded

OpenSearch shouldn’t exist. The open source alternative to Elasticsearch started off as Amazon Web Services’ (AWS) answer to getting outflanked by Elastic’s change in Elasticsearch’s license, which was in turn sparked by AWS building a successful Elasticsearch service but contributing little back. In 2019 when AWS launched its then Open Distro for Elasticsearch, I thought its reasons rang hollow and, frankly, sounded sanctimonious. This was, after all, a company that used more open source than it contributed. Two years later, AWS opted to fork Elasticsearch to create OpenSearch, committing to a “long-term investment” in OpenSearch.

I worked at AWS at the time. Privately, I didn’t think it would work.

Rather, I didn’t feel that AWS really understood just how much work was involved in running a successful open source project, and the company would fail to invest the time and resources necessary to make OpenSearch a viable competitor to Elasticsearch. I was wrong. Although OpenSearch has a long way to go before it can credibly claim to have replaced Elasticsearch in the minds and workloads of developers, it has rocketed up the search engine popularity charts, with an increasingly diverse contributor population. In turn, the OpenSearch experience is adding a new tool to AWS’ arsenal of open source strengths.

Who made who?

Even as someone sympathetic to AWS, I really disliked the company’s 2019 “Keeping open source open” blog post, in which it talked up its reasons for creating a more open alternative to Elasticsearch. First, the company cited its “active participation in open source communities” but offered scant evidence. Yes, the company had been involved in projects like Linux and had committers to Apache Lucene, used by Elastic, but generally speaking, AWS did more to build managed services that made consuming open source easy rather than contribute to the projects directly. AWS wasn’t the freeloader that critics claimed, but neither was it in the strongest position to lament about “open source maintainers [that muddy] the waters between the open source community and the proprietary code they create to monetize the open source.”

Pot, meet kettle.

After that post in 2019, the tug-of-war over Elasticsearch licensing kept escalating until 2021, when AWS opted to break ranks entirely with Elastic’s handling of Elasticsearch and fork it. At the time I ran AWS’ open source strategy and marketing team and, while I understood the reasoning behind the move and supported it, I didn’t think the team driving this decision—or the company as a whole—really understood just how much work was needed to succeed. I still think that’s true. However, I had underestimated just how far the company and that team would go to live up to the Amazon Leadership Principle of “Customer Obsession.”

Two years and well over 100 million downloads later, I’m happy to report just how wrong I was. As part of the AWS OpenSearch team, David Tippett and Eli Fisher laid out a few key indicators of OpenSearch’s success as they gave their 2022 year in review. They topped more than 100 million downloads and gathered 8,760 pull requests from 496 contributors, a number of whom don’t work for AWS. Not stated were other success factors, such as Adobe’s earlier decision to replace Elasticsearch with OpenSearch in its Adobe Commerce suite, or its increasingly open governance with third-party maintainers for the project. Nor did they tout its lightning-fast ascent up the DB-Engines database popularity rankings, hitting the Top 50 databases for the first time.

OpenSearch, in short, is a bonafide open source success story. More surprisingly, it’s an AWS open source success story. For many who have been committed to the “AWS strip mines open source” narrative, such success stories aren’t supposed to exist. Reality bites.

Old dog, new tricks

As I recently wrote, there’s a quiet open source revolution underway at AWS. Even as AWS pushed past Microsoft and Oracle to become the world’s largest database vendor by revenue, according to Gartner, the company has also become a major contributor to an array of community-led and company-led database projects, including OpenSearch. AWS is now a top 4 contributor to PostgreSQL, the largest external contributor to MariaDB (a popular fork of MySQL), and more. I haven’t run the numbers recently, but I suspect AWS is now ranked in the top 3 contributors to open source projects, as measured by the number of employees actively contributing. It’s hard to express just how profound a sea change this is.

It may also be cause for concern for some companies. As much as some people liked to grouse that AWS was taking and not giving back, these same people may be dismayed by a company that knows how to do more than write blog posts about “keeping open source open.” If AWS were to turn forking projects into standard operating procedure, that might get uncomfortable.

Fortunately, not only has AWS learned how to build more open source, it has also learned how to partner with open source companies. My personal experience with AWS is that the company wasn’t much better at partnering in 2019 than it was at contributing to open source (not as bad as many complained, but also not as good as AWS itself claimed). That has changed. It’s also true that partnering and contributing to existing projects will always prove to be a better financial proposition for AWS balance sheets than forking projects. It’s better to contribute some resources versus contributing most resources.

Back to OpenSearch. AWS still has a long way to go, and its OpenSearch success doesn’t seem to be blunting Elastic’s income statement. It turns out that there can and should be many winners in open source. But it’s nice to see that OpenSearch is one of them. In the process, it has made AWS a better company to work for and buy from.