Tenant isolation that survives the customer who queries ten times harder than everyone else
KubernetesOpenSearch DashboardsTerraformShard count dropped by roughly an order of magnitude after small tenants were pooled, and cluster-state operations became fast again. Search queue rejections during peak hours largely disappeared, and…
Overview
A B2B SaaS provider's index-per-tenant model had stopped scaling as their customer count grew. AceMQ designed a tenant architecture that keeps isolation guarantees while cutting the shard count the cluster has to track.
Challenge
With one index per tenant and default shard settings, most indices held very little data while the cluster carried the metadata cost of all of them. Cluster state grew large enough to slow every allocation decision. At the same time, a small number of large tenants could monopolize the search thread pool and cause queue rejections for everyone else, so the isolation the index-per-tenant model was supposed to provide was not actually holding under load.
Environment
OpenSearch on Kubernetes, thousands of tenants ranging from a few thousand documents to hundreds of millions.
Approach
We designed a tiered model: small tenants share pooled indices with a routing key and document-level security, while large tenants keep dedicated indices sized to their data. Alongside that, we added search-level governance so no single tenant can exhaust shared thread pools, and defined the promotion path a tenant follows as it grows.
Solution
- 1Tiered index strategy separating pooled shared indices for small tenants from dedicated indices for large ones
- 2Custom routing and document-level security to keep tenant isolation intact within shared indices
- 3Defined promotion criteria and a tested reindex path for moving a tenant from pooled to dedicated as it grows
- 4Search thread pool sizing, queue limits, and per-tenant concurrency controls so one heavy tenant cannot starve the rest
- 5Query governance including timeouts, result-size caps, and rejection handling surfaced back to the application tier
- 6Cluster and index templates managed as code with tenant onboarding automated end to end
Outcome
Shard count dropped by roughly an order of magnitude after small tenants were pooled, and cluster-state operations became fast again. Search queue rejections during peak hours largely disappeared, and onboarding a new tenant is now an automated step rather than a manual index creation.
Technologies
KubernetesOpenSearch DashboardsTerraformRelated Use Cases
Elasticsearch to OpenSearch Migration Assessment
Assessment of the technical and licensing implications of moving a large Elasticsearch estate to OpenSearch, including client and plugin compatibility.
OpenSearch Snapshot Restore Failure Remediation
Emergency recovery of an OpenSearch domain where snapshot restores were failing partway through and leaving indices in a red state.
Need OpenSearch Architecture Guidance?
AceMQ's senior OpenSearch engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.