External Services

MozDef uses multiple external open source services to store data. These services can be setup on multiple hosts, allowing for a more distrubuted environment.

Elasticsearch

Elasticsearch is the main data storage of MozDef. It’s used to store alerts and event documents, which can then be searched through in a fast and efficient manner. Each day’s events is stored in a separate index (events-20190124 for example),

Note

MozDef currently only supports Elasticsearch version 6.8

Elasticsearch requires java, so let’s install it:

yum install -y java-1.8.0-openjdk

Import public signing key of yum repo:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create yum repo file:

vim /etc/yum.repos.d/elasticsearch.repo

With the following contents:

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Install elasticsearch:

yum install -y elasticsearch

Start Service:

systemctl start elasticsearch
systemctl enable elasticsearch

It may take a few seconds for Elasticsearch to start up, but we can look at the log file to verify when it’s ready:

tail -f /var/log/elasticsearch/elasticsearch.log

Once the services seems to have finished starting up, we can verify using curl:

curl http://localhost:9200

You should see some information in JSON about the Elasticsearch endpoint (version, build date, etc). This means Elasticsearch is all setup, and ready to go!

Kibana

Kibana is a webapp to visualize and search your Elasticsearch cluster data.

Import public signing key of yum repo:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create yum repo file:

vim /etc/yum.repos.d/kibana.repo

With the following contents:

[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Install kibana:

yum install -y kibana

Kibana should work just fine out of the box, but we should take a look at what settings are available:

cat /etc/kibana/kibana.yml

Some of the settings you’ll want to configure are:

• server.name (your server’s hostname)
• elasticsearch.url (the url to your elasticsearch instance and port)
• logging.dest ( /path/to/kibana.log so you can easily troubleshoot any issues)

Then you can start the service:

systemctl start kibana
systemctl enable kibana

Now that Kibana and Elasticsearch are setup, we can populate the MozDef indices and Kibana settings:

su mozdef
source /opt/mozdef/envs/python/bin/activate
cd /opt/mozdef/envs/mozdef/scripts/setup
python initial_setup.py http://localhost:9200 http://localhost:5601

RabbitMQ

RabbitMQ is used on workers to have queues of events waiting to be inserted into the Elasticsearch cluster (storage).

RabbitMQ requires EPEL repos so we need to first install that:

yum -y install epel-release

Download and install Rabbitmq:

wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm
rpm --import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
yum install -y rabbitmq-server-3.6.1-1.noarch.rpm

COPY docker/compose/rabbitmq/files/rabbitmq.config /etc/rabbitmq/ COPY docker/compose/rabbitmq/files/enabled_plugins /etc/rabbitmq/

Create rabbitmq configuration file:

vim /etc/rabbitmq/rabbitmq.config

With the following contents:

[
  {rabbit,
    [
      {tcp_listeners, [5672]},
      {loopback_users, []}
    ]
  },
  {rabbitmq_management,
    [
      {listener,
        [
          {port, 15672},
          {ip, "127.0.0.1"}
        ]
      }
    ]
  }
].

Enable management plugin:

vim /etc/rabbitmq/enabled_plugins

With the following contents:

[rabbitmq_management].

Start Service:

systemctl start rabbitmq-server
systemctl enable rabbitmq-server

MongoDB

Mongodb is the backend database used by Meteor (the web UI).

Note

It’s preferred to run this service on the same host that the Web UI will be running on, so you don’t need to expose this service externally.

Create yum repo file:

vim /etc/yum.repos.d/mongodb.repo

With the following contents:

[mongodb-org-3.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc

Then you can install mongodb:

yum install -y mongodb-org

Overwrite config file:

cp /opt/mozdef/envs/mozdef/config/mongod.conf /etc/mongod.conf

Start Service:

systemctl start mongod
systemctl enable mongod

Nginx

Nginx is used as a proxy to forward requests to the loginput service.

Install nginx:

yum install -y nginx

Copy mozdef nginx conf:

cp /opt/mozdef/envs/mozdef/config/nginx.conf /etc/nginx/nginx.conf

Ensure nginx is started and running:

systemctl start nginx
systemctl enable nginx