HARM: Apply classification name changes...

... in code, documentation, tools and tests

based on the classification changes in #1476 c2c0987
see also #1409

CHANGELOG.md

@@ -28,6 +28,20 @@ CHANGELOG
 ### Development
 
 ### Harmonization
+Update allowed classification fields to 2020-01-28 version (#1409, #1476). Old namings are still supported until at least version 3.0.
+- The taxonomy `abusive content` has been renamed to `abusive-content`.
+- The taxonomy `information content security` has been renamed to `information-content-security`.
+  - The validation of type `unauthorised-information-access` has been fixed, a bug prevented the use of it.
+  - The validation of type `unauthorised-information-modification` has been fixed, a bug prevented the use of it.
+- The taxonomy `intrusion attempts` has been renamed to `intrusion-attempts`.
+- The taxonomy `information gathering` has been rename to `information-gathering`.
+- The taxonomy `malicious code` has been renamed to `malicious-code`.
+  - The type `c2server` has been renamed to `c2-server`.
+- For the taxonomy 'availability', the type `misconfiguration` is new.
+- For the taxonomy 'other', the type `undetermined` is new.
+- For the taxonomy 'vulnerable':
+  - The type `vulnerable client` has been renamed to `vulnerable-client`.
+  - The type `vulnerable service` has been renamed to `vulnerable-service`.
 
 ### Bots
 - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`.

NEWS.md

@@ -13,12 +13,56 @@ See the changelog for a full list of changes.
 
 ### Harmonization
 
+A few classification scheme has been updated to better match the [Reference Security Incident Taxonomy](https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/). The following labels were renamed:
+
+| old taxonomy name | old type name | new taxonomy name | new type name |
+|-|-|-|-|-|-|-|-|
+| abusive content              |                                       | abusive-content               |                                       |
+| information content security |                                       | information-content-security  |                                       |
+| intrusion attempts           |                                       | intrusion-attempts            |                                       |
+| information gathering        |                                       | information-gathering         |                                       |
+| malicious code               |                                       | malicious-code                |                                       |
+| malicious code               | c2server                              | malicious-code                | c2-server                             |
+| vulnerable                   | vulnerable client                     | vulnerable                    | vulnerable-client                     |
+| vulnerable                   | vulnerable service                    | vulnerable                    | vulnerable-service                    |
+
+- For the taxonomy 'availability', the type `misconfiguration` is new.
+- For the taxonomy 'other', the type `undetermined` is new.
+
+The old names can still be used in code, and they are automatically converted to the new names.
+
 ### Configuration
 
 ### Libraries
 
 ### Postgres databases
-
+The following statements optionally update existing data.
+```SQL
+UPDATE events
+   SET "classification.taxonomy" = 'abusive-content'
+   WHERE "classification.taxonomy" = 'abusive content';
+UPDATE events
+   SET "classification.taxonomy" = 'information content security'
+   WHERE "classification.taxonomy" = 'information-content-security';
+UPDATE events
+   SET "classification.taxonomy" = 'intrusion attempts'
+   WHERE "classification.taxonomy" = 'intrusion-attempts';
+UPDATE events
+   SET "classification.taxonomy" = 'information gathering'
+   WHERE "classification.taxonomy" = 'information-gathering';
+UPDATE events
+   SET "classification.taxonomy" = 'malicious code'
+   WHERE "classification.taxonomy" = 'malicious-code';
+UPDATE events
+   SET "classification.type" = 'c2-server'
+   WHERE "classification.taxonomy" = 'malicious-code' AND "classification.type" = 'c2server';
+UPDATE events
+   SET "classification.type" = 'vulnerable-client'
+   WHERE "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'vulnerable client';
+UPDATE events
+   SET "classification.type" = 'vulnerable-service'
+   WHERE "classification.taxonomy" = 'vulnerable' AND "classification.type" = 'vulnerable service';
+```
 
 2.1.3 Bugfix release (unreleased)
 ---------------------------------

contrib/eventdb/README.md

@@ -2,7 +2,7 @@ EventDB Utilities
 =================
 
 - Apply Malware Name Mapping: Applies the malware name mapping to the eventdb. Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand.
-  It queries the database for all distinct malware names with the taxonomy "malicious code" and sets another column to the malware family name.
+  It queries the database for all distinct malware names with the taxonomy "malicious-code" and sets another column to the malware family name.
 - Apply Domain Suffix: Writes the public domain suffix to the `source.domain_suffix` / `destination.domain_suffix` columns, extracted from `source.fqdn` / `destination.fqdn`.
 
 Usage

contrib/eventdb/apply_mapping_eventdb.py

@@ -54,7 +54,7 @@ def eventdb_apply(malware_name_column, malware_family_column, host, port,
     cur = db.cursor(cursor_factory=DictCursor)
 
     cur.execute('SELECT DISTINCT "classification.identifier", "malware.name" FROM {table} '
-                'WHERE "classification.taxonomy" = \'malicious code\' {where}'
+                'WHERE "classification.taxonomy" = \'malicious-code\' {where}'
             ''.format(table=table, where=where))
     if dry_run:
         execute = lambda x, y: print(cur.mogrify(x, y).decode())
@@ -74,7 +74,7 @@ def eventdb_apply(malware_name_column, malware_family_column, host, port,
                 execute('UPDATE {table} SET "classification.identifier" = %s '
                         'WHERE "malware.name" = %s '
                         'AND "classification.identifier" IS DISTINCT FROM %s AND '
-                        '"classification.taxonomy" = \'malicious code\' {where}'
+                        '"classification.taxonomy" = \'malicious-code\' {where}'
                         ''.format(table=table, where=where),
                         (rule[1],  malware_name, rule[1]))
                 break

contrib/malware_name_mapping/download_mapping.py

@@ -35,7 +35,7 @@ def none_len(arg: Optional[list]):
 def generate_rule(expression: str, identifier: str, name=None):
     return {"rulename": name if name else "%s-%s" % (identifier,
                                                      hashlib.sha1(expression.encode()).hexdigest()[:10]),
-            "if": {"classification.taxonomy": "malicious code",
+            "if": {"classification.taxonomy": "malicious-code",
                    "malware.name": expression
                    },
             "then": {"classification.identifier": identifier, }}

docs/Data-Harmonization.md

@@ -89,28 +89,29 @@ The taxonomy can be automatically added by the taxonomy expert bot based on the
 
 |Taxonomy|Type|Description|
 |--------|----|-----------|
-|abusive content|spam|Or 'Unsolicited Bulk Email', this means that the recipient has not granted verifiable permission for the message to be sent and that the message is sent as part of a larger collection of messages, all having a functionally comparable content.|
+|abusive-content|spam|Or 'Unsolicited Bulk Email', this means that the recipient has not granted verifiable permission for the message to be sent and that the message is sent as part of a larger collection of messages, all having a functionally comparable content.|
 |abusive-content|harmful-speech|Discreditation or discrimination of somebody, e.g. cyber stalking, racism or threats against one or more individuals.|
 |abusive-content|violence|Child pornography, glorification of violence, etc.|
 |availability|ddos|Distributed Denial of Service attack, e.g. SYN-Flood or UDP-based reflection/amplification attacks.|
 |availability|dos|Denial of Service attack, e.g. sending specially crafted requests to a web application which causes the application to crash or slow down.|
+|availability|misconfiguration|Software misconfiguration resulting in service availability issues, e.g. DNS server with outdated DNSSEC Root Zone KSK.|
 |availability|outage|Outage caused e.g. by air condition failure or natural disaster.|
 |availability|sabotage|Physical sabotage, e.g cutting wires or malicious arson.|
 |fraud|copyright|Offering or Installing copies of unlicensed commercial software or other copyright protected materials (Warez).|
 |fraud|masquerade|Type of attack in which one entity illegitimately impersonates the identity of another in order to benefit from it.|
 |fraud|phishing|Masquerading as another entity in order to persuade the user to reveal private credentials.|
 |fraud|unauthorized-use-of-resources|Using resources for unauthorized purposes including profit-making ventures, e.g. the use of e-mail to participate in illegal profit chain letters or pyramid schemes.|
-|information content security|Unauthorised-information-access|Unauthorized access to information, e.g. by abusing stolen login credentials for a system or application, intercepting traffic or gaining access to physical documents.|
-|information content security|Unauthorised-information-modification|Unauthorised modification of information, e.g. by an attacker abusing stolen login credentials for a system or application or a ransomware encrypting data.|
-|information content security|data-loss|Loss of data, e.g. caused by harddisk failure or physical theft.|
-|information content security|dropzone|This IOC refers to place where the compromised machines store the stolen user data. Not in ENISA eCSIRT-II taxonomy.|
-|information content security|leak|IOCs relating to leaked credentials or personal data. Not in ENISA eCSIRT-II taxonomy.|
+|information-content-security|unauthorised-information-access|Unauthorized access to information, e.g. by abusing stolen login credentials for a system or application, intercepting traffic or gaining access to physical documents.|
+|information-content-security|unauthorised-information-modification|Unauthorised modification of information, e.g. by an attacker abusing stolen login credentials for a system or application or a ransomware encrypting data.|
+|information-content-security|data-loss|Loss of data, e.g. caused by harddisk failure or physical theft.|
+|information-content-security|dropzone|This IOC refers to place where the compromised machines store the stolen user data. Not in ENISA eCSIRT-II taxonomy.|
+|information-content-security|leak|IOCs relating to leaked credentials or personal data. Not in ENISA eCSIRT-II taxonomy.|
 |information gathering|scanner|Attacks that send requests to a system to discover weaknesses. This also includes testing processes to gather information on hosts, services and accounts. Examples: fingerd, DNS querying, ICMP, SMTP (EXPN, RCPT, ...), port scanning.|
 |information-gathering|sniffing|Observing and recording of network traffic (wiretapping).|
 |information-gathering|social-engineering|Gathering information from a human being in a non-technical way (e.g. lies, tricks, bribes, or threats). This IOC refers to a resource, which has been observed to perform brute-force attacks over a given application protocol.|
-|intrusion attempts|brute-force|Multiple login attempts (Guessing / cracking of passwords, brute force).|
-|intrusion attempts|exploit|An attack using an unknown exploit.|
-|intrusion attempts|ids-alert|IOCs based on a sensor network. This is a generic IOC denomination, should it be difficult to reliably denote the exact type of activity involved for example due to an anecdotal nature of the rule that triggered the alert.|
+|intrusion-attempts|brute-force|Multiple login attempts (Guessing / cracking of passwords, brute force).|
+|intrusion-attempts|exploit|An attack using an unknown exploit.|
+|intrusion-attempts|ids-alert|IOCs based on a sensor network. This is a generic IOC denomination, should it be difficult to reliably denote the exact type of activity involved for example due to an anecdotal nature of the rule that triggered the alert.|
 |intrusions|application-compromise|Compromise of an application by exploiting (un)known software vulnerabilities, e.g. SQL injection.|
 |intrusions|backdoor|This refers to hosts, which have been compromised and backdoored with a remote administration software or Trojan in the traditional sense. Not in ENISA eCSIRT-II taxonomy.|
 |intrusions|burglary|Physical intrusion, e.g. into corporate building or data center.|
@@ -120,24 +121,25 @@ The taxonomy can be automatically added by the taxonomy expert bot based on the
 |intrusions|unauthorized-command|The possibly infected device sent unauthorized commands to a remote device with malicious intent. Not in ENISA eCSIRT-II taxonomy.|
 |intrusions|unauthorized-login|A possibly infected device logged in to a remote device without authorization. Not in ENISA eCSIRT-II taxonomy.|
 |intrusions|unprivileged-account-compromise|Compromise of a system using an unprivileged (user/service) account.|
-|malicious code|c2server|This is a command and control server in charge of a given number of botnet drones.|
-|malicious code|dga domain|DGA Domains are seen various families of malware that are used to periodically generate a large number of domain names that can be used as rendezvous points with their command and control servers. Not in ENISA eCSIRT-II taxonomy.|
-|malicious code|infected-system|This is a compromised machine, which has been observed to make a connection to a command and control server.|
-|malicious code|malware|A URL is the most common resource with reference to malware binary distribution. Not in ENISA eCSIRT-II taxonomy.|
-|malicious code|malware-configuration|This is a resource which updates botnet drones with a new configuration.|
-|malicious code|malware-distribution|URI used for malware distribution, e.g. a download URL included in fake invoice malware spam.|
-|malicious code|ransomware|This IOC refers to a specific type of compromised machine, where the computer has been hijacked for ransom by the criminals. Not in ENISA eCSIRT-II taxonomy and deprecated, use 'infected system instead'.|
+|malicious-code|c2-server|This is a command and control server in charge of a given number of botnet drones.|
+|malicious-code|dga domain|DGA Domains are seen various families of malware that are used to periodically generate a large number of domain names that can be used as rendezvous points with their command and control servers. Not in ENISA eCSIRT-II taxonomy.|
+|malicious-code|infected-system|This is a compromised machine, which has been observed to make a connection to a command and control server.|
+|malicious-code|malware|A URL is the most common resource with reference to malware binary distribution. Not in ENISA eCSIRT-II taxonomy.|
+|malicious-code|malware-configuration|This is a resource which updates botnet drones with a new configuration.|
+|malicious-code|malware-distribution|URI used for malware distribution, e.g. a download URL included in fake invoice malware spam.|
+|malicious-code|ransomware|This IOC refers to a specific type of compromised machine, where the computer has been hijacked for ransom by the criminals. Not in ENISA eCSIRT-II taxonomy and deprecated, use 'infected system instead'.|
 |other|blacklist|Some sources provide blacklists, which clearly refer to abusive behavior, such as spamming, but fail to denote the exact reason why a given identity has been blacklisted. The reason may be that the justification is anecdotal or missing entirely. This type should only be used if the typing fits the definition of a blacklist, but an event specific denomination is not possible for one reason or another.|
 |other|other|All incidents which don't fit in one of the given categories should be put into this class.|
 |other|proxy|This refers to the use of proxies from inside your network. Not in ENISA eCSIRT-II taxonomy.|
 |other|tor|This IOC refers to incidents related to TOR network infrastructure. Not in ENISA eCSIRT-II taxonomy.|
 |other|unknown|Unknown classification. Not in ENISA eCSIRT-II taxonomy.|
+|other|undetermined|The categorisation of the incident is unknown/undetermined.|
 |test|test|Meant for testing.|
 |vulnerable|ddos-amplifier|Publicly accessible services that can be abused for conducting DDoS reflection/amplification attacks, e.g. DNS open-resolvers or NTP servers with monlist enabled.|
 |vulnerable|information-disclosure|Publicly accessible services potentially disclosing sensitive information, e.g. SNMP or Redis.|
 |vulnerable|potentially-unwanted-accessible|Potentially unwanted publicly accessible services, e.g. Telnet, RDP or VNC.|
-|vulnerable|vulnerable client|This attribute refers to a badly configured or vulnerable clients, which may be vulnerable and can be compromised by a third party. For example, not-up-to-date clients or client which are misconfigured, such as clients querying public domains for WPAD configurations. In addition, to specify the vulnerability and its potential abuse, one should use the classification.identifier, description and other attributes for that purpose respectively. Not in ENISA eCSIRT-II taxonomy.|
-|vulnerable|vulnerable service|This attribute refers to a badly configured or vulnerable network service, which may be abused by a third party. For example, these services relate to open proxies, open dns resolvers, network time servers (NTP) or character generation services (chargen), simple network management services (SNMP). In addition, to specify the network service and its potential abuse, one should use the protocol, destination port and description attributes for that purpose respectively. Not in ENISA eCSIRT-II taxonomy.|
+|vulnerable|vulnerable-client|This attribute refers to a badly configured or vulnerable clients, which may be vulnerable and can be compromised by a third party. For example, not-up-to-date clients or client which are misconfigured, such as clients querying public domains for WPAD configurations. In addition, to specify the vulnerability and its potential abuse, one should use the classification.identifier, description and other attributes for that purpose respectively. Not in ENISA eCSIRT-II taxonomy.|
+|vulnerable|vulnerable-service|This attribute refers to a badly configured or vulnerable network service, which may be abused by a third party. For example, these services relate to open proxies, open dns resolvers, network time servers (NTP) or character generation services (chargen), simple network management services (SNMP). In addition, to specify the network service and its potential abuse, one should use the protocol, destination port and description attributes for that purpose respectively. Not in ENISA eCSIRT-II taxonomy.|
 |vulnerable|vulnerable-system|A system which is vulnerable to certain attacks. Example: misconfigured client proxy settings (example: WPAD), outdated operating system version, etc.|
 |vulnerable|weak-crypto|Publicly accessible services offering weak crypto, e.g. web servers susceptible to POODLE/FREAK attacks.|
 
@@ -167,8 +169,8 @@ Meaning of source, destination and local values for each classification type and
 |spam|*infected device*|targeted server|internal at source||
 |test||||||
 |unknown||||||
-|vulnerable service|*vulnerable device*||| heartbleed, openresolver, snmp |
-|vulnerable client|*vulnerable device*||| wpad |
+|vulnerable-service|*vulnerable device*||| heartbleed, openresolver, snmp |
+|vulnerable-client|*vulnerable device*||| wpad |
 
 Field in italics is the interesting one for CERTs.
 

docs/Feeds.md

@@ -83,7 +83,7 @@ To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then run
 *  * `columns`: `time.source,source.ip,malware.name,status,extra.SBL,source.as_name,source.geolocation.cc`
 *  * `ignore_values`: `,,,,Not listed,,`
 *  * `skip_table_head`: `True`
-*  * `type`: `c2server`
+*  * `type`: `c2-server`
 
 
 ## Feodo Tracker IPs
@@ -753,7 +753,7 @@ To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then run
 *  * `columns`: `['time.source', 'source.url', 'source.ip', 'malware.name', '__IGNORE__']`
 *  * `default_url_protocol`: `http://`
 *  * `skip_table_head`: `True`
-*  * `type`: `c2server`
+*  * `type`: `c2-server`
 
 
 # DShield