Usage

Web Interface

MozDef uses the Meteor framework for the web interface and bottle.py for the REST API. For authentication, MozDef supports local account creation. Meteor (the underlying UI framework) supports many authentication options including google, github, twitter, facebook, oath, native accounts, etc.

Events visualizations

Since the backend of MozDef is Elastic Search, you get all the goodness of Kibana with little configuration. The MozDef UI is focused on incident handling and adding security-specific visualizations of SIEM data to help you weed through the noise.

Alerts

Alerts are implemented as Elastic Search searches. MozDef provides a plugin interface to allow open access to event data for enrichment, hooks into other systems, etc.

Incident handling

Sending logs to MozDef

Events/Logs are accepted as json over http(s) with the POST or PUT methods or over rabbit-mq. Most modern log shippers support json output. MozDef is tested with support for:

We have some configuration snippets

What should I log?

If your program doesn’t log anything it doesn’t exist. If it logs everything that happens it becomes like the proverbial boy who cried wolf. There is a fine line between logging too little and too much but here is some guidance on key events that should be logged and in what detail.

Event Example Rationale
Authentication Events Failed/Success logins Authentication is always an important event to log as it establishes traceability for later events and allows correlation of user actions across systems.
Authorization Events Failed attempts to insert/update/delete a record or access a section of an application. Once a user is authenticated they usually obtain certain permissions. Logging when a user’s permissions do not allow them to perform a function helps troubleshooting and can also be helpful when investigating security events.
Account Lifecycle Account creation/deletion/update Adding, removing or changing accounts are often the first steps an attacker performs when entering a system.
Password/Key Events Password changed, expired, reset. Key expired, changed, reset. If your application takes on the responsibility of storing a user’s password (instead of using a centralized source) it is important to note changes to a users credentials or crypto keys.
Account Activations Account lock, unlock, disable, enable If your application locks out users after failed login attempts or allows for accounts to be inactivated, logging these events can assist in troubleshooting access issues.
Application Exceptions Invalid input, fatal errors, known bad things

If your application catches errors like invalid input attempts on web forms, failures of key components, etc creating a log record when these events occur can help in troubleshooting and tracking security patterns across applications. Full stack traces should be avoided however as the signal to noise ratio is often overwhelming.

It is also preferable to send a single event rather than a multitude of events if it is possible for your application to correlate a significant exception.

For example, some systems are notorious for sending a connection event with source IP, then sending an authentication event with a session ID then later sending an event for invalid input that doesn’t include source IP or session ID or username. Correctly correlating these events across time is much more difficult than just logging all pieces of information if it is available.

JSON format

This section describes the structure JSON objects to be sent to MozDef. Using this standard ensures developers, admins, etc are configuring their application or system to be easily integrated into MozDef.

Background

Mozilla used CEF as a logging standard for compatibility with Arcsight and for standardization across systems. While CEF is an admirable standard, MozDef prefers JSON logging for the following reasons:

  • Every development language can create a JSON structure.
  • JSON is easily parsed by computers/programs which are the primary consumer of logs.
  • CEF is primarily used by Arcsight and rarely seen outside that platform and doesn’t offer the extensibility of JSON.
  • A wide variety of log shippers (heka, logstash, fluentd, nxlog, beaver) are readily available to meet almost any need to transport logs as JSON.
  • JSON is already the standard for cloud platforms like amazon’s cloudtrail logging.

Description

As there is no common RFC-style standard for json logs, we prefer the following structure adapted from a combination of the graylog GELF and logstash specifications.

Note all fields are lowercase to avoid one program sending sourceIP, another sending sourceIp, another sending SourceIPAddress, etc. Since the backend for MozDef is elasticsearch and fields are case-sensitive this will allow for easy compatibility and reduce potential confusion for those attempting to use the data. MozDef will perform some translation of fields to a common schema but this is intended to allow the use of heka, nxlog, beaver and retain compatible logs.

Mandatory Fields

Field Purpose Sample Value
category General category/type of event matching the ‘what should I log’ section below authentication, authorization, account creation, shutdown, atartup, account deletion, account unlock, zeek
details Additional, event-specific fields that you would like included with the event. Please completely spell out a field rather an abbreviate: i.e. sourceipaddress instead of srcip. <see below>
hostname The fully qualified domain name of the host sending the message server1.example.com
processid The PID of the process sending the log 1234
processname The name of the process sending the log myprogram.py
severity RFC5424 severity level of the event in all caps: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY INFO
source Source of the event (file name, system name, component name) /var/log/syslog/2014.01.02.log
summary Short human-readable version of the event suitable for IRC, SMS, etc. john login attempts over threshold, account locked
tags An array or list of any tags you would like applied to the event

vpn, audit

nsm,zeek,intel

timestamp Full date plus time timestamp of the event in ISO format including the timezone offset 2014-01-30T19:24:43+06:00
utctimestamp Full UTC date plus time timestamp of the event in ISO format including the timezone offset 2014-01-30T13:24:43+00:00
receivedtimestamp Full UTC date plus time timestamp in ISO format when mozdef parses the event. This is set by mozdef upon receipt of the event 2014-01-30T13:24:43+00:00

Details substructure (mandatory if such data is sent, otherwise optional)

Field Purpose Sample Value
destinationipaddress Destination IP of a network flow 8.8.8.8
destinationport Destination port of a network flow 80
sourceipaddress Source IP of a network flow 8.8.8.8
sourceport Source port of a network flow 42297
sourceuri Source URI such as a referer https://www.mozilla.org/
destinationuri Destination URI as in “wget this URI” https://www.mozilla.org/
error Action resulted in an error or failure true/false
success Transaction failed/ or succeeded true/false
username Username, email, login, etc. kang@mozilla.com
useragent Program agent string curl/1.76 (Windows; 5.1)

Examples

{
    "timestamp": "2014-02-14T11:48:19.035762739-05:00",
    "hostname": "somemachine.in.your.company.com",
    "processname": "/path/to/your/program.exe",
    "processid": 3380,
    "severity": "INFO",
    "summary": "joe login failed",
    "category": "authentication",
    "source": "ldap",
    "tags": [
        "ldap",
        "adminAccess",
        "failure"
    ],
    "details": {
        "username": "joe",
        "task": "access to admin page /admin_secret_radioactiv",
        "result": "10 authentication failures in a row",
        "success": false
    }
}

Simple test

If you want to just post some sample json to Mozdef do something like

curl -v --header "Content-Type: application/json"   --request POST   --data '{"tags": ["test"],"summary": "just a test"}'   http://localhost:8080/events

where http://localhost:8080 is whatever is running the ‘loginput’ service. The ‘data’ curl option is what gets posted as json to MozDef. If your post is successful, you should then be able to find the event in elastic search/kibana.

Writing alerts

Alerts allow you to create notifications based on events stored in elasticsearch. You would usually try to aggregate and correlate events that are the most severe and on which you have response capability. Alerts are stored in the alerts folder.

There are two types of alerts:

  • simple alerts that consider events on at a time
    • For example you may want to get an alert everytime a single LDAP modification is detected.
  • aggregation alerts that allow you to aggregate events on the field of your choice
    • For example you may want to alert when more than 3 login attempts failed for the same username.

You’ll find documented examples in the alerts folder.

Once you’ve written your alert, you need to configure it in celery to be launched periodically. If you have a AlertBruteforceSsh class in a alerts/bruteforce_ssh.py file for example, in alerts/lib/config you can configure the task to run every minute:

ALERTS = {
        'bruteforce_ssh.AlertBruteforceSsh': crontab(minute='*/1'),
}