JSON format¶
This section describes the structure JSON objects to be sent to MozDef. Using this standard ensures developers, admins, etc are configuring their application or system to be easily integrated into MozDef.
Background¶
Mozilla used CEF as a logging standard for compatibility with Arcsight and for standardization across systems. While CEF is an admirable standard, MozDef prefers JSON logging for the following reasons:
- Every development language can create a JSON structure.
- JSON is easily parsed by computers/programs which are the primary consumer of logs.
- CEF is primarily used by Arcsight and rarely seen outside that platform and doesn’t offer the extensibility of JSON.
- A wide variety of log shippers (heka, logstash, fluentd, nxlog, beaver) are readily available to meet almost any need to transport logs as JSON.
- JSON is already the standard for cloud platforms like amazon’s cloudtrail logging.
Description¶
As there is no common RFC-style standard for json logs, we prefer the following structure adapted from a combination of the graylog GELF and logstash specifications.
Note all fields are lowercase to avoid one program sending sourceIP, another sending sourceIp, another sending SourceIPAddress, etc. Since the backend for MozDef is elasticsearch and fields are case-sensitive this will allow for easy compatibility and reduce potential confusion for those attempting to use the data. MozDef will perform some translation of fields to a common schema but this is intended to allow the use of heka, nxlog, beaver and retain compatible logs.
Mandatory Fields¶
Field | Purpose | Sample Value |
---|---|---|
category | General category/type of event matching the ‘what should I log’ section below | authentication, authorization, account creation, shutdown, atartup, account deletion, account unlock, zeek |
details | Additional, event-specific fields that you would like included with the event. Please completely spell out a field rather an abbreviate: i.e. sourceipaddress instead of srcip. | <see below> |
hostname | The fully qualified domain name of the host sending the message | server1.example.com |
processid | The PID of the process sending the log | 1234 |
processname | The name of the process sending the log | myprogram.py |
severity | RFC5424 severity level of the event in all caps: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, EMERGENCY | INFO |
source | Source of the event (file name, system name, component name) | /var/log/syslog/2014.01.02.log |
summary | Short human-readable version of the event suitable for IRC, SMS, etc. | john login attempts over threshold, account locked |
tags | An array or list of any tags you would like applied to the event | vpn, audit nsm,zeek,intel |
timestamp | Full date plus time timestamp of the event in ISO format including the timezone offset | 2014-01-30T19:24:43+06:00 |
utctimestamp | Full UTC date plus time timestamp of the event in ISO format including the timezone offset | 2014-01-30T13:24:43+00:00 |
receivedtimestamp | Full UTC date plus time timestamp in ISO format when mozdef parses the event. This is set by mozdef upon receipt of the event | 2014-01-30T13:24:43+00:00 |
Details substructure (mandatory if such data is sent, otherwise optional)¶
Field | Purpose | Sample Value |
---|---|---|
destinationipaddress | Destination IP of a network flow | 8.8.8.8 |
destinationport | Destination port of a network flow | 80 |
sourceipaddress | Source IP of a network flow | 8.8.8.8 |
sourceport | Source port of a network flow | 42297 |
sourceuri | Source URI such as a referer | https://www.mozilla.org/ |
destinationuri | Destination URI as in “wget this URI” | https://www.mozilla.org/ |
error | Action resulted in an error or failure | true/false |
success | Transaction failed/ or succeeded | true/false |
username | Username, email, login, etc. | kang@mozilla.com |
useragent | Program agent string | curl/1.76 (Windows; 5.1) |
Examples¶
{
"timestamp": "2014-02-14T11:48:19.035762739-05:00",
"hostname": "somemachine.in.your.company.com",
"processname": "/path/to/your/program.exe",
"processid": 3380,
"severity": "INFO",
"summary": "joe login failed",
"category": "authentication",
"source": "ldap",
"tags": [
"ldap",
"adminAccess",
"failure"
],
"details": {
"username": "joe",
"task": "access to admin page /admin_secret_radioactiv",
"result": "10 authentication failures in a row",
"success": false
}
}