How to Optimize Search-Based Systems with ElasticSearch and MongoDB

The Owens Valley Radio Observatory.

Introduction

In today's digital landscape, search functionality is a cornerstone for many web applications. Whether it's an e-commerce platform, a content management system, or a social media network, the ability to quickly and accurately retrieve information is essential. This blog post will explore the architecture and implementation for search based Web App with ElasticSearch and MongoDb.

Search and Analytics Engines

When discussing search and analytics engines, it's crucial to understand the two primary types of queries you might encounter: structured and unstructured.

Structured Queries

Consider a requirement where you need to fetch documents based on a date range. For instance: “Based on the value set by the user for Date Field X and Date Field Y, return the first 10 documents between the date X and Y.” This can be translated into a SQL or NoSQL query, such as:

SELECT TOP(10) * FROM Documents WHERE X < Date AND Date < Y

This is a structured query, performed on fields with a defined structure, and is straightforward to implement using traditional database systems.

Unstructured Queries

Now, consider a different requirement: “Based on the value set by the user in Textbox Field Z, return the top 10 documents most relevant to the property description in the document.” This introduces several challenges:

• Full-text Search: Analyzing document descriptions to find occurrences of words in Field Z.

• Relevance Scoring: Calculating the relevance of occurrences and considering synonyms.

Structured queries are insufficient here. Instead, we need to query unstructured data, which can be effectively handled using tools like ElasticSearch or OpenSearch.

ElasticSearch and OpenSearch

ElasticSearch

According to the official ElasticSearch website:

“ElasticSearch is the distributed search and analytics engine at the heart of the Elastic Stack. Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in ElasticSearch.”

OpenSearch

Per the OpenSearch official website:

“OpenSearch is a distributed, community-driven, Apache 2.0-licensed, 100% open-source search and analytics suite used for a broad set of use cases like real-time application monitoring, log analytics, and website search.”

OpenSearch is a fork of ElasticSearch, sharing many capabilities and characteristics. Both are powerful for searching information but have high costs for update operations.

Combining ElasticSearch and MongoDB

To illustrate, let's consider a book e-commerce store requiring an efficient search mechanism using ElasticSearch. A typical book data model might include fields like Title, Authors, Category, ISBN, Description, Price, Quantity, and SKU.

From this set we can identify two kinds of fields, the fields for a structured search and an structured search.

• Structured Search: Price, Category, Authors

• Unstructured Search: Title, Description

• Irrelevant for Search: Quantity, SKU

ElasticSearch can store a subset of these fields focused on search functionality, while MongoDB can persist the complete dataset.

Handling Updates

Updating information in ElasticSearch is costly. When an update request is received, ElasticSearch doesn't modify the existing document directly. Instead, it retrieves the document, applies the changes, and then reindexes it as a new document. To mitigate this, especially in systems with high update volumes, we can use a dual-store approach:

1. Read Operations: Directed to ElasticSearch for fast retrieval.

2. Write Operations: Sent to MongoDB and enqueued in a write operations queue.

3. Write Worker: Processes the queue and updates ElasticSearch in bulk.

This architecture optimizes read performance while managing write costs effectively.

Conclusion

Integrating ElasticSearch with MongoDB allows for a robust search-based web application, balancing the need for efficient full-text search with manageable update operations. By leveraging the strengths of both databases, we can create a system that meets user expectations for speed and relevance in search results while maintaining data consistency and performance.