Concept of operations

Event Management

From an event management point of view MozDef relies on Elastic Search for:

  • event storage
  • event archiving
  • event indexing
  • event searching

This means if you use MozDef for your log management you can use the features of Elastic Search to store millions of events, archive them to Amazon if needed, index the fields of your events, and search them using highly capable interfaces like Kibana.

MozDef differs from other log management solutions that use Elastic Search in that it does not allow your log shippers direct contact with Elastic Search itself. In order to provide advanced functionality like event correlation, aggregation and machine learning, MozDef inserts itself as a shim between your log shippers (rsyslog, syslog-ng, beaver, nxlog, heka, logstash) and Elastic Search. This means your log shippers interact with MozDef directly and MozDef handles translating their events as they make they’re way to Elastic Search.

Event Pipeline

The logical flow of events is:

                   +–––––––––––+              +––––––––––––––+
                   |    MozDef +––––––––––––––+              |
+––––––––––+       |    FrontEnd              | Elastic      |
| shipper  +–––––––+–––––––––––+              | Search       |
++++++++++++                                  | cluster      |
++++++++++++                                  |              |
| shipper  +–––––––+–––––––––––+              |              |
+––––––––––+       |    MozDef +-–––––––––––––+              |
                   |    FrontEnd              |              |
                   +–––––––––––+              |              |

Choose a shipper (logstash, nxlog, beaver, heka, rsyslog, etc) that can send JSON over http(s). MozDef uses nginx to provide http(s) endpoints that accept JSON posted over http. Each front end contains a Rabbit-MQ message queue server that accepts the event and sends it for further processing.

You can have as many front ends, shippers and cluster members as you with in any geographic organization that makes sense for your topology. Each front end runs a series of python workers hosted by uwsgi that perform:

  • event normalization (i.e. translating between shippers to a common taxonomy of event data types and fields)
  • event enrichment
  • simple regex-based alerting
  • machine learning on the real-time event stream

Event Enrichment

To facilitate event correlation, MozDef allows you to write plugins to populate your event data with consistent meta-data customized for your environment. Through simple python plug-ins this allows you to accomplish a variety of event-related tasks like:

  • further parse your events into more details
  • geoIP tag your events
  • correct fields not properly handled by log shippers
  • tag all events involving key staff
  • tag all events involving previous attackers or hits on a watchlist
  • tap into your event stream for ancilary systems
  • maintain ‘last-seen’ lists for assets, employees, attackers

Event Correlation/Alerting

Correlation/Alerting is currently handled as a series of queries run periodically against the Elastic Search engine. This allows MozDef to make full use of the lucene query engine to group events together into summary alerts and to correlate across any data source accessible to python.

Incident Handling

From an incident handling point of view MozDef offers the realtime responsiveness of Meteor in a web interface. This allows teams of incident responders the ability to see each others actions in realtime, no matter their physical location.