JSON format

This section describes the structure JSON objects to be sent to MozDef. Using this standard ensures developers, admins, etc are configuring their application or system to be easily integrated into MozDef.

Background

Mozilla used CEF as a logging standard for compatibility with Arcsight and for standardization across systems. While CEF is an admirable standard, MozDef prefers JSON logging for the following reasons:

• Every development language can create a JSON structure.
• JSON is easily parsed by computers/programs which are the primary consumer of logs.
• CEF is primarily used by Arcsight and rarely seen outside that platform and doesn’t offer the extensibility of JSON.
• A wide variety of log shippers (heka, logstash, fluentd, nxlog, beaver) are readily available to meet almost any need to transport logs as JSON.
• JSON is already the standard for cloud platforms like amazon’s cloudtrail logging.

Description

As there is no common RFC-style standard for json logs, we prefer the following structure adapted from a combination of the graylog GELF and logstash specifications.

Note all fields are lowercase to avoid one program sending sourceIP, another sending sourceIp, another sending SourceIPAddress, etc. Since the backend for MozDef is elasticsearch and fields are case-sensitive this will allow for easy compatibility and reduce potential confusion for those attempting to use the data. MozDef will perform some translation of fields to a common schema but this is intended to allow the use of heka, nxlog, beaver and retain compatible logs.

Mandatory Fields

Field	Purpose	Sample Value
category	General category/type of event matching the ‘what should I log’ section below	authentication, authorization, account creation, shutdown, atartup, account deletion, account unlock, zeek
details	Additional, event-specific fields that you would like included with the event. Please completely spell out a field rather an abbreviate: i.e. sourceipaddress instead of srcip.	<see below>
hostname	The fully qualified domain name of the host sending the message	server1.example.com
processid	The PID of the process sending the log	1234
processname	The name of the process sending the log	myprogram.py
severity	RFC5424 severity level of the event in all caps: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY	INFO
source	Source of the event (file name, system name, component name)	/var/log/syslog/2014.01.02.log
summary	Short human-readable version of the event suitable for IRC, SMS, etc.	john login attempts over threshold, account locked
tags	An array or list of any tags you would like applied to the event	vpn, audit nsm,zeek,intel
timestamp	Full date plus time timestamp of the event in ISO format including the timezone offset	2014-01-30T19:24:43+06:00
utctimestamp	Full UTC date plus time timestamp of the event in ISO format including the timezone offset	2014-01-30T13:24:43+00:00
receivedtimestamp	Full UTC date plus time timestamp in ISO format when mozdef parses the event. This is set by mozdef upon receipt of the event	2014-01-30T13:24:43+00:00

Details substructure (mandatory if such data is sent, otherwise optional)

Field	Purpose	Sample Value
destinationipaddress	Destination IP of a network flow	8.8.8.8
destinationport	Destination port of a network flow	80
sourceipaddress	Source IP of a network flow	8.8.8.8
sourceport	Source port of a network flow	42297
sourceuri	Source URI such as a referer	https://www.mozilla.org/
destinationuri	Destination URI as in “wget this URI”	https://www.mozilla.org/
error	Action resulted in an error or failure	true/false
success	Transaction failed/ or succeeded	true/false
username	Username, email, login, etc.	kang@mozilla.com
useragent	Program agent string	curl/1.76 (Windows; 5.1)

Examples

{
    "timestamp": "2014-02-14T11:48:19.035762739-05:00",
    "hostname": "somemachine.in.your.company.com",
    "processname": "/path/to/your/program.exe",
    "processid": 3380,
    "severity": "INFO",
    "summary": "joe login failed",
    "category": "authentication",
    "source": "ldap",
    "tags": [
        "ldap",
        "adminAccess",
            "failure"
    ],
    "details": {
        "username": "joe",
        "task": "access to admin page /admin_secret_radioactiv",
        "result": "10 authentication failures in a row",
        "success": false
    }
}