Introduction

This tutorial provides guidance on sending logs from Fluent Bit installed on a self-managed MongoDB on an Ubuntu Droplet to DigitalOcean’s managed OpenSearch database.

OpenSearch is an open-source suite for search and analytics, derived from Elasticsearch and Kibana. Its features include real-time data ingestion, sophisticated querying, and strong security, making it a suitable choice for applications in e-commerce, IT monitoring, and finance. OpenSearch is constantly updated through community contributions, offering features such as SQL query support, machine learning, and alerting.

Fluent Bit serves as a lightweight, open-source log processor and forwarder, designed to collect, process, and transmit log data. It excels in performance, making it perfect for environments with limited resources, such as containers and edge computing. Fluent Bit is widely used for real-time logging and monitoring due to its capacity to efficiently aggregate and send logs to various destinations.

Prerequisites

Before initiating, confirm you have:

• A DigitalOcean Cloud account that provides access to Managed OpenSearch.
• Basic familiarity with OpenSearch and Fluent Bit.
• Access to the OpenSearch Dashboard.

Installing Fluent Bit

Fluent Bit can be installed on various platforms, and for this tutorial, we will focus on an Ubuntu platform. Execute the following command in your Droplet’s Terminal.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

For detailed information, refer to the official documentation.

Configuring Fluent Bit

The default location for the Fluent Bit configuration file is /etc/fluent-bit/fluent-bit.conf. Modifications will be needed in this file to enable logging to OpenSearch.

Fluent Bit Inputs

Fluent Bit includes a variety of input plugins that extract log and event data from different sources. For gathering log data from log files, the tail Input plugin is utilized.

Modify the fluent-bit.conf file with the following entries:

[INPUT]
   name tail
   Path /var/log/mongodb/mongod.log

To learn more about Input plugins, visit Fluent Bit’s official Manual.

Fluent Bit Outputs

Alongside input plugins, Fluent Bit also provides an output plugin for directing collected logs to various destinations. In this case, we will use the Opensearch Output plugin to send logs to OpenSearch.

[OUTPUT]
   Name  opensearch
   Match *
   Host opensearch hostname
   port 25060
   HTTP_User doadmin #DO Managed OpenSeach username
   HTTP_Passwd XXXX #Managed OpenSeach Password
   Index mongodb
   tls On
   Suppress_Type_Name On

For more details on Output plugins, explore Fluent Bit’s official Manual.

After completing the configurations, proceed to start the Fluent Bit Service.

systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

Access MongoDB Logs in OpenSearch Dashboard

• Log into your OpenSearch Dashboard through your DigitalOcean Cloud account.
• Select the hamburger icon and click on Dashboard Management from the Management section.
• Select Index patterns and then click on Create Index pattern.
• Name the index pattern to match the one specified in the fluent-bit.conf file.
• When the index pattern is ready, you can view the MongoDB logs in the Discover tab.

Conclusion

Congratulations! You have successfully configured Fluent Bit to collect and transmit logs from a self-hosted MongoDB to a DigitalOcean Managed OpenSearch database. This guide covered Fluent Bit configuration and the creation of an index pattern in OpenSearch, enabling efficient monitoring and analysis of your MongoDB logs.

Ready to deploy your AI model? Start your one-click HUGS journey on DigitalOcean today and experience seamless, scalable AI infrastructure.

Sr. Technical Writer at DigitalOcean | Medium Top Writers (AI & ChatGPT) | 2M+ monthly views & 34K Subscribers | Former Cloud Consultant at AMEX | Former SRE (DevOps) at NUTANIX

Join us! Follow our straightforward steps to deploy Hugging Face HUGS on DigitalOcean GPU Droplets and bring your AI models to fruition.