Exploring OpenSearch and Its Latest Features

The demand for scalable, efficient, and flexible search solutions is constantly increasing as modern applications rely on enormous amounts of data. One of the most popular solutions for searching, analyzing, and visualizing data has been Elasticsearch. However, with the advent of OpenSearch—a community-driven, open-source fork of Elasticsearch—organizations now have access to a powerful search engine with innovative features and a transparent governance model.

This blog post will explore OpenSearch in detail, highlighting its key functionalities, the community behind it, and, most importantly, the latest features that are pushing the boundaries of search capabilities.

1. Introduction to OpenSearch

OpenSearch is a powerful search engine and analytics suite designed for real-time log analytics, application monitoring, security information and event management (SIEM), and more. Born out of a fork from Elasticsearch 7.10, OpenSearch maintains the open-source spirit while enabling advanced search features and scalability.

OpenSearch provides the following key capabilities:

Full-text search
Structured search and filtering
Data ingestion pipelines
Real-time search analytics
Extensive plugins for observability, security, and performance

Given its broad scope and robust community backing, OpenSearch has quickly become a prominent player in the world of search engines.

2. The Governance and Open Source Philosophy

OpenSearch emerged in 2021 as a response to licensing changes made to Elasticsearch by Elastic N.V. This move by Amazon Web Services (AWS), along with other contributors, ensured that a fully open-source alternative to Elasticsearch would continue to thrive. The Apache 2.0 License governs OpenSearch, ensuring it remains free for community collaboration and wide-scale usage.

OpenSearch’s development is driven by community input, with transparency and inclusivity in its core philosophy. Developers and organizations can participate in feature development, submit bug reports, and contribute plugins. This decentralized model allows OpenSearch to grow organically, addressing real-world needs in the search space.

How OpenSearch Differs from Elasticsearch

While OpenSearch shares many features with its predecessor Elasticsearch, it diverges by promoting long-term open-source development. It is also compatible with existing Elasticsearch clients and APIs (as of version 7.10), making it a natural transition for organizations that previously used Elasticsearch.

Key differences:

Fully open-source: Licensed under Apache 2.0, OpenSearch is free and open to modification.
Active community contributions: OpenSearch maintains active forums and discussions to allow developers to contribute.
Compatibility with Elasticsearch APIs: OpenSearch retains compatibility with Elasticsearch 7.10, making migration easier.

3. Setting Up OpenSearch: A Basic Guide

Before we dive into the newest features, let’s walk through the process of setting up OpenSearch. Whether you’re deploying on-premise or leveraging cloud infrastructure, OpenSearch can scale to fit your needs.

Prerequisites

Java Runtime Environment (JRE): OpenSearch requires Java 11.
Memory: Minimum 4GB of RAM for optimal performance.
Storage: Adequate disk space for data and logs (depends on use case).

Installation Steps

Download OpenSearch
- Navigate to the official OpenSearch downloads page.
- Choose the appropriate version for your operating system (Linux, Windows, Docker, etc.).
Extract and Configure OpenSearchbashCopy codetar -zxvf opensearch-1.0.0.tar.gz cd opensearch-1.0.0
Configure Cluster Settings Open the config/opensearch.yml file to set the basic cluster configuration:y
Start the OpenSearch Node

Plain Text

cluster.name: opensearch-cluster 
node.name: opensearch-node-1 
network.host: 0.0.0.0 
discovery.seed_hosts: ["127.0.0.1"]

Shell

./bin/opensearch

Once your node is running, you can interact with it through the API or the built-in OpenSearch Dashboards, which can be installed separately.

Installing OpenSearch Dashboards

Dashboards are a visualization layer for OpenSearch, providing a user-friendly interface for analyzing data. You can install it similarly by downloading it from the official site and configuring it to point to your OpenSearch cluster.

Shell

tar -zxvf opensearch-dashboards-1.0.0.tar.gz
cd opensearch-dashboards-1.0.0<br>./bin/opensearch-dashboards

Now that you have OpenSearch and OpenSearch Dashboards up and running, let’s move on to exploring the core and latest features.

4. Exploring Core Features of OpenSearch

At its heart, OpenSearch is a powerful distributed search engine designed for horizontal scalability and real-time analytics. Core features include:

Search and Query DSL: OpenSearch supports a powerful domain-specific language (DSL) for querying structured and unstructured data, ranging from basic full-text search to complex filters and aggregations.
Indexing and Sharding: Data is organized into indexes and split across shards, allowing OpenSearch to scale linearly and distribute search and indexing workloads across multiple nodes.
Ingestion Pipelines: You can configure pipelines for ingesting and processing data before it is indexed.
Alerting and Monitoring: OpenSearch includes built-in alerting capabilities to notify users when specific conditions (e.g., log anomalies or threshold breaches) are met.

5. The Top 10 Latest Features in OpenSearch

OpenSearch has undergone rapid development since its launch, with new features being added regularly. Below, we highlight the top 10 latest features that are transforming how users interact with data.

5.1. Improved Search Relevance

Search relevance is the cornerstone of any search engine, and OpenSearch has introduced several enhancements to improve how it ranks and retrieves documents based on a user’s query.

Key Improvements:

Learning-to-Rank (LTR) Models: OpenSearch integrates with machine learning models that can re-rank search results based on signals.
Custom Scoring Scripts: You can now define custom scripts to manipulate document scores based on various factors like recency, popularity, or user profile.

Code Sample: Custom Scoring Script

Shell

POST /my-index/_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "content": "OpenSearch"
        }
      },
      "boost_mode": "multiply",
      "functions": [
        {
          "script_score": {
            "script": {
              "source": "doc['popularity'].value * 1.5 + _score"
            }
          }
        }
      ]
    }
  }
}

5.2. Fine-Grained Role-Based Access Control (RBAC)

OpenSearch now includes enhanced RBAC to control access at a more granular level. This feature allows administrators to define roles that dictate who can read, write, or manage data within the cluster.

Key Features:

Field-level security: Limit access to specific fields within a document.
Document-level security: Restrict access based on the content of documents.

Example Configuration:

In the OpenSearch Dashboards, navigate to Security → Roles and define custom roles with fine-grained permissions. Below is an example of a role that restricts access to only a subset of documents.

JSON

{
"role": {
    "index_permissions": [
        {
            "index_patterns": ["my-index"],
            "document_level_security_query": "{\"term\": {\"user_id\": \"1\"}}",
            "allowed_actions": ["read"]
        }
    ]
}
}

5.3. Data Streams

Data Streams in OpenSearch enable time-series indexing for large volumes of data, such as logs, metrics, or IoT device data. This feature allows you to seamlessly index data into multiple backing indices over time, while simplifying data management and querying.

Key Features:

Automatic creation of new backing indices as time advances.
Simplified querying across all backing indices through the data stream alias.

Creating a Data Stream:

Shell

PUT /_data_stream/logs-metrics

Indexing into a Data Stream:

JSON

POST /logs-metrics/_doc
{
"@timestamp": "2023-09-01T12:00:00Z",
"level": "info",
  "message": "OpenSearch log event"
}

OpenSearch will automatically create backing indices based on the time range of the documents being indexed.

5.4. Index State Management (ISM) Enhancements

Index State Management (ISM) is a critical feature for automating the lifecycle of indexes. With the latest version, OpenSearch has added several new capabilities to ISM policies, including conditional transitions based on index size, document count, or time since creation.

Key Features:

Automatic index rollover: Transition indexes based on their age or size.
Multiple conditions for transitions: Combine multiple criteria (size, document count, etc.) for more flexibility.

Example ISM Policy:

JSON

PUT _plugins/_ism/policies/log-rollover-policy
{
  "policy": {
    "description": "Manage log indexes based on size",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "50gb",
              "min_age": "7d"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_size": "50gb",
              "min_age": "7d"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "delete": {
              "min_age": "30d"
            }
          }
        ]
      }
    ]
  }
}

5.5. Cross-Cluster Search

Cross-cluster search allows you to search across multiple OpenSearch clusters, enabling organizations to manage separate clusters for different workloads while maintaining a unified search experience.

Key Features:

Cluster aliases: Use cluster aliases to represent remote clusters.
Unified search: Search across all clusters as if they were a single entity.

Example: Cross-Cluster Search Query

JSON

GET /my-index/_search
{
  "query": {
    "match": {
      "message": "OpenSearch"
    }
  }
}

With cross-cluster search enabled, this query will run across all associated clusters.

5.6. Multi-Tenancy in Dashboards

Multi-tenancy enables the isolation of data and visualizations in OpenSearch Dashboards based on the tenant (e.g., department or team). This feature is essential for organizations that want to allow different users to have access to their own dashboards while still sharing the same OpenSearch cluster.

Key Features:

Tenant-specific data isolation: Ensure each tenant has access only to its data and dashboards.
Shared dashboards: Create shared dashboards for collaboration.

Creating Tenants in Dashboards:

Navigate to Security → Tenants in the OpenSearch Dashboards.
Define custom tenants and assign them to roles or users.

5.7. SQL Support for OpenSearch

OpenSearch now natively supports SQL queries, allowing users familiar with relational databases to leverage their SQL skills for querying search data.

Key Features:

Familiar SQL syntax: Use SELECT, WHERE, JOIN, and other SQL keywords to query OpenSearch.
Aggregation support: Perform aggregations, such as COUNT, AVG, SUM, directly in SQL queries.

Example SQL Query:

JSON

POST _plugins/_sql
{
  "query": "SELECT COUNT(*) FROM my-index WHERE age > 30"
}

SQL support provides a bridge for users who are more comfortable with SQL than the DSL query language.

5.8. Security Anomaly Detection

Anomaly detection in OpenSearch has been expanded to include security-centric use cases. This feature allows for real-time detection of anomalous behavior, such as unauthorized access attempts, data exfiltration, or other suspicious activities.

Key Features:

Unsupervised learning models: Use ML models to detect outliers without training.
Real-time monitoring: Continuously monitor data for anomalies.

Setting Up Anomaly Detection:

In OpenSearch Dashboards, go to Anomaly Detection.
Create an anomaly detector for a specific data stream (e.g., security logs).

5.9. Real-Time Monitoring and Alerts

OpenSearch has enhanced its alerting capabilities, allowing for more complex alert conditions and real-time notifications based on incoming data. This is particularly useful for monitoring infrastructure, security, and business metrics.

Key Features:

Multi-condition alerts: Combine multiple criteria (e.g., error count, latency, user activity) into a single alert.
Real-time notifications: Send alerts via email, webhooks, or messaging platforms.

Creating an Alert:

JSON

PUT _plugins/_alerting/monitors/monitor-1
{
  "name": "Log Error Monitor",
  "enabled": true,
  "triggers": [
    {
      "name": "Error Trigger",
      "condition": {
        "script": {
          "source": "ctx.results[0].hits.total > 10"
        }
      },
      "actions": [
        {
          "name": "Send Email",
          "destination_id": "email-destination",
          "message_template": {
            "subject": "Log Error Alert",
            "body": "Errors have exceeded threshold in the past 5 minutes."
          }
        }
      ]
    }
  ]
}

5.10. OpenSearch Dashboards Integration

The latest updates in OpenSearch Dashboards focus on deeper integration with the search engine, enhanced visualizations, and a better user experience for managing data and clusters.

Key Features:

Enhanced visualizations: New options for creating charts, maps, and graphs.
Index management integration: Manage indices directly from the Dashboards interface.
Custom plugins: Build and deploy custom visualizations and tools.

6. Code Samples: How to Use These Features

Here are some additional code samples demonstrating how to use the features mentioned above.

1. Creating a Learning-to-Rank Model for Search Relevance

JSON

PUT /_ltr/_model/my-model
{
  "model": {
    "type": "ranklib",
    "definition": {
      "source": "…"
    }
  }
}

POST /_ltr/_featureset/my-features
{
  "features": [
    {
      "name": "title",
      "params": ["query"],
      "template_language": "mustache",
      "template": {
        "match": {
          "title": "{{query}}"
        }
      }
    }
  ]
}

Shell

pip install opensearch-py

Python

from opensearchpy import OpenSearch
import json

# Step 1: Connect to OpenSearch
host = "https://localhost:9200"  # Replace with your OpenSearch host
auth = ("admin", "admin")  # Replace with your OpenSearch username/password
client = OpenSearch(
    hosts=[host],
    http_auth=auth,
    verify_certs=False
)

# Step 2: Define a feature set
feature_set_name = "my_ltr_features"
features = [
    {
        "name": "bm25",
        "params": ["keywords"],
        "template_language": "mustache",
        "template": {
            "match": {
                "title": "{{keywords}}"
            }
        }
    },
    {
        "name": "field_length",
        "params": ["field"],
        "template_language": "mustache",
        "template": {
            "script_score": {
                "script": {
                    "source": "doc['{{field}}'].size()"
                }
            }
        }
    }
]

# Step 3: Upload the feature set
client.transport.perform_request(
    method="POST",
    url=f"_ltr/_featureset/{feature_set_name}",
    body={"featureset": {"features": features}}
)

print(f"Feature set '{feature_set_name}' created.")

# Step 4: Train and upload a model
model_name = "my_ltr_model"
model_definition = """
<model>
…your trained model in XGBoost or LightGBM format…
</model>
"""

client.transport.perform_request(
    method="PUT",
    url=f"_ltr/_model/{model_name}",
    body={
        "model": {
            "name": model_name,
            "model": {
                "type": "model/xgboost+json",
                "definition": model_definition
            }
        }
    }
)

print(f"Model '{model_name}' uploaded.")

# Step 5: Use the LTR model for re-ranking
query = {
    "query": {
        "bool": {
            "must": [
                {"match": {"title": "open source"}}
            ]
        }
    },
    "rescore": {
        "window_size": 100,
        "query": {
            "rescore_query": {
                "sltr": {
                    "params": {"keywords": "open source"},
                    "model": model_name
                }
            },
            "query_weight": 1.0,
            "rescore_query_weight": 2.0
        }
    }
}

response = client.search(index="my-index", body=query)
print("Re-ranked search results:", json.dumps(response, indent=2))

2. SQL Query for Aggregations

JSON

POST _plugins/_sql
{
  "query": "SELECT category, COUNT(*) FROM products GROUP BY category"
}

3. Cross-Cluster Search

GET /<remote_cluster>:index/_search
{
  "query": {
    "match": {
      "message": "Error"
    }
  }
}

Python

from opensearchpy import OpenSearch
import json

# Step 1: Connect to OpenSearch
host = "https://localhost:9200"  # Replace with your OpenSearch host
auth = ("admin", "admin")  # Replace with your OpenSearch username/password
client = OpenSearch(
    hosts=[host],
    http_auth=auth,
    verify_certs=False
)

# Define the index name
index_name = "your-index-name"

# Step 2: Define the query and aggregations
query_body = {
    "size": 0,  # Don't return documents, just aggregations
    "query": {
        "match_all": {}
    },
    "aggs": {
        # 1. Terms Aggregation
        "popular_terms": {
            "terms": {
                "field": "category.keyword",  # Replace with your field
                "size": 10  # Top 10 terms
            }
        },
        # 2. Date Histogram Aggregation
        "sales_over_time": {
            "date_histogram": {
                "field": "order_date",  # Replace with your date field
                "calendar_interval": "month"
            }
        },
        # 3. Statistical Aggregation
        "price_stats": {
            "stats": {
                "field": "price"  # Replace with your numeric field
            }
        },
        # 4. Average Aggregation
        "average_price": {
            "avg": {
                "field": "price"  # Replace with your numeric field
            }
        },
        # 5. Max Aggregation
        "max_price": {
            "max": {
                "field": "price"  # Replace with your numeric field
            }
        },
        # 6. Min Aggregation
        "min_price": {
            "min": {
                "field": "price"  # Replace with your numeric field
            }
        },
        # 7. Sum Aggregation
        "total_revenue": {
            "sum": {
                "field": "price"  # Replace with your numeric field
            }
        },
        # 8. Range Aggregation
        "price_ranges": {
            "range": {
                "field": "price",
                "ranges": [
                    {"to": 50},
                    {"from": 50, "to": 100},
                    {"from": 100}
                ]
            }
        },
        # 9. Nested Aggregation
        "nested_example": {
            "nested": {
                "path": "nested_field"  # Replace with your nested field path
            },
            "aggs": {
                "nested_terms": {
                    "terms": {
                        "field": "nested_field.subfield.keyword"  # Replace with subfield
                    }
                }
            }
        }
    }
}

# Step 3: Execute the query
response = client.search(index=index_name, body=query_body)

# Step 4: Print the response
print("Aggregations result:")
print(json.dumps(response["aggregations"], indent=2))

3. Cross-Cluster Search

JSON

GET /<remote_cluster>:index/_search
{
  "query": {
    "match": {
      "message": "Error"
    }
  }
}

7. Use Cases: Leveraging OpenSearch in Real-World Applications

1. Log Analytics for Enterprise IT

OpenSearch’s ability to ingest and analyze vast amounts of log data makes it an excellent fit for enterprise IT environments. From monitoring system logs to identifying anomalous behaviors, OpenSearch provides real-time insights into infrastructure performance.

2. E-commerce Search

With powerful relevance scoring and SQL support, OpenSearch is ideal for e-commerce platforms seeking to improve the relevance of their search results. Learning-to-rank models, combined with index state management, ensure that users always see the most relevant results.

3. Security Information and Event Management (SIEM)

The enhanced security features in OpenSearch make it a natural choice for building a SIEM platform. With real-time monitoring, alerting, and anomaly detection, OpenSearch allows security teams to track potential threats and respond immediately.

8. Conclusion

OpenSearch is evolving rapidly, delivering robust search capabilities and a community-driven model for development. With new features like data streams, fine-grained RBAC, cross-cluster search, and enhanced security monitoring, OpenSearch is set to continue its growth and prominence in the search engine landscape.

Whether you’re looking to deploy a scalable search solution for e-commerce, log analytics, or security, OpenSearch offers the tools, flexibility, and innovation to meet modern needs.

Let us know your thoughts about these latest features, and feel free to try them out in your OpenSearch environment today!

Exploring OpenSearch and Its Latest Features

1. Introduction to OpenSearch

2. The Governance and Open Source Philosophy

How OpenSearch Differs from Elasticsearch

3. Setting Up OpenSearch: A Basic Guide

Prerequisites

Installation Steps

Installing OpenSearch Dashboards

4. Exploring Core Features of OpenSearch

5. The Top 10 Latest Features in OpenSearch

5.1. Improved Search Relevance

Key Improvements:

Code Sample: Custom Scoring Script

5.2. Fine-Grained Role-Based Access Control (RBAC)

Key Features:

Example Configuration:

5.3. Data Streams

Key Features:

Creating a Data Stream:

Indexing into a Data Stream:

5.4. Index State Management (ISM) Enhancements

Key Features:

Example ISM Policy:

5.5. Cross-Cluster Search

Key Features:

Example: Cross-Cluster Search Query

5.6. Multi-Tenancy in Dashboards

Key Features:

Creating Tenants in Dashboards:

5.7. SQL Support for OpenSearch

Key Features:

Example SQL Query:

5.8. Security Anomaly Detection

Key Features:

Setting Up Anomaly Detection:

5.9. Real-Time Monitoring and Alerts

Key Features:

Creating an Alert:

5.10. OpenSearch Dashboards Integration

Key Features:

6. Code Samples: How to Use These Features

1. Creating a Learning-to-Rank Model for Search Relevance

2. SQL Query for Aggregations

3. Cross-Cluster Search

7. Use Cases: Leveraging OpenSearch in Real-World Applications

1. Log Analytics for Enterprise IT

2. E-commerce Search

3. Security Information and Event Management (SIEM)

8. Conclusion

Comments

Leave a Reply Cancel reply