Raising the Bar with Amazon OpenSearch Serverless

As an engineer, the 2023 drama BlackBerry hit close to home. One of the most memorable scenes is when the meticulously detail-oriented Co-Founder and CTO of BlackBerry, Mike Lazaridis, pushes back against the hard-driving transplant CEO, Jim Balsillie:

[Balsille]: “Are you familiar with the saying, 'Perfect is the enemy of good?'”

[Lazaridis]: “Good enough is the enemy of humanity.”

This quip applies not just to the design of the absurdly popular pre-iPhone mobile device, but also to cloud architectures. In this blog post, I explain how Alcion’s migration from Amazon OpenSearch managed domains to Amazon OpenSearch Serverless is one example of fighting the good fight against “good enough.”  

Amazon OpenSearch Service Managed Domain – Good Enough

When I designed the first iteration of Alcion’s search functionality in November 2022, Amazon OpenSearch Service managed domain was the leading solution which offered powerful search capability at enterprise scale – it's based on OpenSearch, a fork of the well-known ElasticSearch open-source project and comes packaged as an Amazon managed-service offering. I ran my design proposal by our platform engineering team and got the green-light, but they were lacking in enthusiasm. As one of the team members said, “OpenSearch managed domain is the best practice here, but you’re not going to like it.”
‍
Indeed, the name “OpenSearch managed domain” was somewhat misleading. Sure, Amazon takes on the complexity of provisioning the underlying infrastructure, but operators were still on the hook for estimating usage patterns and translating those into a specific configuration for the managed domain. This configuration could get as complex as needing to set up index state management (ISM) policies, but even the more basic knobs, like count and size of EBS volumes, caused serious headaches. Alcion couldn’t run the risk of under-provisioning OpenSearch resources because this would have exposed us to reliability issues, but over-provisioning resources meant that our OpenSearch managed domains would chew through cloud compute credits at a rate only to be outdone by OpenAI.

As I sat at my desk in February 2023, debugging an OpenSearch CPU utilization alarm and losing sleep over scaling concerns for Alcion’s public launch, I realized that I was paying the price for settling on the “good-enough” approach. Mike Lazaridis wouldn’t have approved.

Amazon OpenSearch Serverless – Raising the Bar

So, when Amazon announced the general availability of their new Amazon OpenSearch Serverless offering, I knew we needed to seriously consider being an early adopter. Over the span of the next two months, I dove head-first into everything OpenSearch Serverless. I enjoyed the benefits of some awesome new functionality (auto-scaling!) while spending some (many) late nights discovering the nuances of the service and seeing how far it could be pushed.  

Throughout this process, I raised so many Amazon support tickets that the customer success representative established a direct line of communication to the Amazon OpenSearch Serverless engineering team. This relationship resulted in several key learnings and, once OpenSearch Serverless had been rolled out to all Alcion’s customers, I even had the privilege of co-authoring a blog post with the Amazon team!

That blog post goes into a number of design considerations for building on Amazon OpenSearch Serverless. It has something for everybody – architecture diagrams for the system design guru, cost estimates for the belt-tightening VP, and code snippets for new-grad (let’s be honest, senior-level) engineers. I highly recommend checking it out and seeing more details about how OpenSearch Serverless helped Alcion raise the bar for our search platform.

Microsoft 365 Data Protection – Raising the Bar

Alcion offers more than just a cutting-edge search architecture – we're an AI driven data protection platform for Microsoft 365 helps IT admins protect the digital assets which are business critical for their companies and employees. If you’re interested in checking out Alcion, you can try Alcion for free! The trial runs for 14 days, and no credit card is required. We also share instructions on how you can get a free Microsoft 365 test/sandbox domain in five minutes and use it to trial Alcion.

I hope you give Alcion an opportunity to raise the bar for your data protection solution.