New endpoint for Mitre

[druizz90]

Hi team,

This PR adds a new endpoint for getting data from Mitre database. It should be merged after 4064.

Below there are examples of requests:

# curl -u foo:bar -k -X GET "https://127.0.0.1:55000/mitre?pretty&platform=macos&limit=1"
{
   "error": 0,
   "data": {
      "items": [
         {
            "id": "T1001",
            "json": {
               "external_references": [
                  {
                     "url": "https://attack.mitre.org/techniques/T1001",
                     "source_name": "mitre-attack",
                     "external_id": "T1001"
                  },
                  {
                     "url": "https://arxiv.org/ftp/arxiv/papers/1408/1408.1136.pdf",
                     "source_name": "University of Birmingham C2",
                     "description": "Gardiner, J.,  Cova, M., Nagaraja, S. (2014, February). Command & Control Understanding, Denying and Detecting. Retrieved April 20, 2016."
                  }
               ],
               "object_marking_refs": [
                  "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
               ],
               "x_mitre_detection": "Analyze network data for uncommon data flows (e.g., a client sending significantly more data than it receives from a server). Processes utilizing the network that do not normally have network communication or have never been seen before are suspicious. Analyze packet contents to detect communications that do not follow the expected protocol behavior for the port that is being used. (Citation: University of Birmingham C2)",
               "x_mitre_data_sources": [
                  "Packet capture",
                  "Process use of network",
                  "Process monitoring",
                  "Network protocol analysis"
               ],
               "modified": "2018-10-17T00:14:20.652Z",
               "x_mitre_network_requirements": true,
               "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
               "x_mitre_platforms": [
                  "Linux",
                  "macOS",
                  "Windows"
               ],
               "kill_chain_phases": [
                  {
                     "phase_name": "Command and Control",
                     "kill_chain_name": "mitre-attack"
                  }
               ],
               "id": "attack-pattern--ad255bfe-a9e6-4b52-a258-8d3462abe842",
               "name": "Data Obfuscation",
               "created": "2017-05-31T21:30:18.931Z",
               "x_mitre_version": "1.0",
               "type": "attack-pattern",
               "description": "Command and control (C2) communications are hidden (but not necessarily encrypted) in an attempt to make the content more difficult to discover or decipher and to make the communication less conspicuous and hide commands from being seen. This encompasses many methods, such as adding junk data to protocol traffic, using steganography, commingling legitimate traffic with C2 communications traffic, or using a non-standard data encoding system, such as a modified Base64 encoding for the message body of an HTTP request."
            },
            "platforms": [
               "Linux",
               "macOS",
               "Windows"
            ],
            "phases": [
               "Command and Control"
            ]
         }
      ],
      "totalItems": 148
   }
}

# curl -u foo:bar -k -X GET "https://127.0.0.1:55000/mitre?pretty&attack=T1005"
{
   "error": 0,
   "data": {
      "items": [
         {
            "id": "T1005",
            "json": {
               "external_references": [
                  {
                     "url": "https://attack.mitre.org/techniques/T1005",
                     "source_name": "mitre-attack",
                     "external_id": "T1005"
                  }
               ],
               "object_marking_refs": [
                  "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
               ],
               "x_mitre_system_requirements": [
                  "Privileges to access certain files and directories"
               ],
               "x_mitre_data_sources": [
                  "File monitoring",
                  "Process monitoring",
                  "Process command-line parameters"
               ],
               "modified": "2018-10-17T00:14:20.652Z",
               "x_mitre_detection": "Monitor processes and command-line arguments for actions that could be taken to collect files from a system. Remote access tools with built-in features may interact directly with the Windows API to gather data. Data may also be acquired through Windows system management tools such as [Windows Management Instrumentation](https://attack.mitre.org/techniques/T1047) and [PowerShell](https://attack.mitre.org/techniques/T1086).",
               "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
               "x_mitre_platforms": [
                  "Linux",
                  "macOS",
                  "Windows"
               ],
               "kill_chain_phases": [
                  {
                     "phase_name": "Collection",
                     "kill_chain_name": "mitre-attack"
                  }
               ],
               "id": "attack-pattern--3c4a2599-71ee-4405-ba1e-0e28414b4bc5",
               "name": "Data from Local System",
               "created": "2017-05-31T21:30:20.537Z",
               "x_mitre_version": "1.0",
               "type": "attack-pattern",
               "description": "Sensitive data can be collected from local system sources, such as the file system or databases of information residing on the system prior to Exfiltration.\n\nAdversaries will often search the file system on computers they have compromised to find files of interest. They may do this using a [Command-Line Interface](https://attack.mitre.org/techniques/T1059), such as [cmd](https://attack.mitre.org/software/S0106), which has functionality to interact with the file system to gather information. Some adversaries may also use [Automated Collection](https://attack.mitre.org/techniques/T1119) on the local system."
            },
            "platforms": [
               "Linux",
               "macOS",
               "Windows"
            ],
            "phases": [
               "Collection"
            ]
         }
      ],
      "totalItems": 1
   }
}

# curl -u foo:bar -k -X GET "https://127.0.0.1:55000/mitre?pretty&q=platform=linux;phase=persistence&limit=1"
{
   "error": 0,
   "data": {
      "items": [
         {
            "id": "T1067",
            "json": {
               "external_references": [
                  {
                     "url": "https://attack.mitre.org/techniques/T1067",
                     "source_name": "mitre-attack",
                     "external_id": "T1067"
                  },
                  {
                     "url": "http://www.symantec.com/connect/blogs/are-mbr-infections-back-fashion",
                     "source_name": "Lau 2011",
                     "description": "Lau, H. (2011, August 8). Are MBR Infections Back in Fashion? (Infographic). Retrieved November 13, 2014."
                  },
                  {
                     "url": "https://www.fireeye.com/content/dam/fireeye-www/regional/fr_FR/offers/pdfs/ig-mtrends-2016.pdf",
                     "source_name": "MTrends 2016",
                     "description": "Mandiant. (2016, February). M-Trends 2016. Retrieved January 4, 2017."
                  }
               ],
               "object_marking_refs": [
                  "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
               ],
               "x_mitre_data_sources": [
                  "API monitoring",
                  "MBR",
                  "VBR"
               ],
               "modified": "2018-10-17T00:14:20.652Z",
               "x_mitre_detection": "Perform integrity checking on MBR and VBR. Take snapshots of MBR and VBR and compare against known good samples. Report changes to MBR and VBR as they occur for indicators of suspicious activity and further analysis.",
               "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
               "x_mitre_platforms": [
                  "Linux",
                  "Windows"
               ],
               "kill_chain_phases": [
                  {
                     "phase_name": "Persistence",
                     "kill_chain_name": "mitre-attack"
                  }
               ],
               "id": "attack-pattern--02fefddc-fb1b-423f-a76b-7552dd211d4d",
               "name": "Bootkit",
               "created": "2017-05-31T21:30:54.661Z",
               "x_mitre_version": "1.0",
               "type": "attack-pattern",
               "description": "A bootkit is a malware variant that modifies the boot sectors of a hard drive, including the Master Boot Record (MBR) and Volume Boot Record (VBR). (Citation: MTrends 2016)\n\nAdversaries may use bootkits to persist on systems at a layer below the operating system, which may make it difficult to perform full remediation unless an organization suspects one was used and can act accordingly.\n\n### Master Boot Record\nThe MBR is the section of disk that is first loaded after completing hardware initialization by the BIOS. It is the location of the boot loader. An adversary who has raw access to the boot drive may overwrite this area, diverting execution during startup from the normal boot loader to adversary code. (Citation: Lau 2011)\n\n### Volume Boot Record\nThe MBR passes control of the boot process to the VBR. Similar to the case of MBR, an adversary who has raw access to the boot drive may overwrite the VBR to divert execution during startup to adversary code.",
               "x_mitre_permissions_required": [
                  "Administrator",
                  "SYSTEM"
               ]
            },
            "platforms": [
               "Linux",
               "Windows"
            ],
            "phases": [
               "Persistence"
            ]
         }
      ],
      "totalItems": 14
   }
}

Mocha tests

New Mocha tests were added for this endpoint:

# mocha test/test_mitre.js 


  Mitre
    GET/sca/:agent_id
      ✓ Request (423ms)
      ✓ Pagination: limit = 1 (395ms)
      ✓ Pagination: limit = 5 (395ms)
      ✓ Pagination: limit = 10 (396ms)
      ✓ Pagination: limit > 10 (392ms)
      ✓ Retrieve all elements with limit=0 (382ms)
      ✓ Filters: attack (394ms)
      ✓ Filters: attack (request returns 0 items) (385ms)
      ✓ Filters: phase=initial access (393ms)
      ✓ Filters: phase=persistence (403ms)
      ✓ Filters: phase (request returns 0 items) (390ms)
      ✓ Filters: platform=linux (398ms)
      ✓ Filters: platform=macos (390ms)
      ✓ Filters: platform=windows (389ms)
      ✓ Filters: platform=windows,phase=persistence (392ms)
      ✓ Filters: platform=linux,phase=execution (388ms)
      ✓ Filters: platform=macos,phase=impact (389ms)
      ✓ Filters: platform (request returns 0 items) (388ms)
      ✓ Filters: q=attack=T1015 (419ms)
      ✓ Filters: q=platform=linux (391ms)
      ✓ Filters: q=phase=execution (395ms)
      ✓ Filters: q (request returns 0 items) (396ms)
      ✓ Filters: q (wrong query 1) (385ms)
      ✓ Filters: q (wrong query 2) (398ms)
      ✓ Filters: q (wrong query 3) (389ms)


  25 passing (10s)

Best regards,

Demetrio.