Log Streaming record format

Updated 10 months ago by admin

Web Security - Web Hits

{
	"utc_timestamp": "2023-02-15 12:20:24.010832",
	"tz_offset": 0,
	"netbios_domain": "HQ",
	"samaccountname": "r.rossiter",
	"tag": "HQ-GATEWAY",
	"ip_address": "10.0.0.195",
	"mac_address": "00:0C:29:D2:43:DD",
	"device_name": "DESKTOP-I0743DB",
	"dst_ip_address": "72.246.148.115",
	"country_code": "gb",
	"country": "united kingdom",
	"city": "slough",
	"latitude": 51.528,
	"longitude": -0.642,
	"http_method": "GET",
	"url_scheme": "https",
	"url_domain": "polling.bbc.co.uk",
	"url_path": "/news/breaking-news/audience/domestic",
	"url_query": "",
	"browser": "Firefox 109.0",
	"operating_system": "Windows 10",
	"device_type": "Desktop",
	"xhr_flag": 0,
	"icap_agent": "ussgateway",
	"feature_control": "",
	"filter_rule": "Authorised Web Sites",
	"final_action": "Allow",
	"log_level": "Normal",
	"web_categories": ["International News"],
	"url_categories": ["bbc"],
	"matched_web_categories": ["International News"],
	"matched_url_categories": [],
	"keyword_categories": ["Keyword Template - URL & CASB", "Keyword Template - URL"],
	"matched_keyword_categories": [],
	"keywords": ["pattern"],
	"matched_keywords": [],	
	"icap_tx_uuid": "\\xb5620c23472c442db1cd13b264bcd8bd"
}

Schema Description

Field

Type

Description

utc_timestamp

timestamp

Date and time the request was received, in UTC.

tz_offset

int

The offset in minutes from UTC when the request was received (can be positive or negative). Used to generate a local timestamp for the request.

netbios_domain

string

The domain part of the captured username string. Depending on the authentication mode, this will either be the NetBIOS name or the domain part of the UPN. If no domain is found, this will default to the device hostname.

samaccountname

string

The captured username that originated the request. Depending on the authentication mode, this will either be the SAM account name or the first part of the UPN.

tag

string

An arbitrary text string applied to the request through the use of Tags.

ip_address

string

The local IPv4 address that generated the request.

mac_address

string

The local MAC address that generated the request.

dst_ip_address

string

The remote IPv4 address of the url_domain.

device_name

string

The hostname of the device, if registered.

country_code

string

The ISO country code of the destination IP address

country

string

A string representing the country e.g. united states

city

string

A string representing the city e.g. london

longitude

float

The longitude of the destination IP address

latitude

float

The latitude of the destination IP address

http_method

string

The HTTP method of the request. Possible values: GET, PUT, POST, DELETE, OPTIONS, PATCH

url_scheme

string

The HTTP protocol scheme. Possible options: http, https

url_domain

string

The bare domain from the request, without any protocol, path or query string.

url_path

string

The path part of the requested URL.

url_query

string

The query string part of the requested URL.

browser

string

The browser name and version that generated the request e.g. Chrome 91.0.4472.101 (determined from User Agent)

operating_system

string

The operating system name that generated the request e.g. Windows 10 (determined from User Agent)

device_type

string

The type of device that generated the request e.g. Windows 10 (determined from User Agent)

xhr_flag

int

Determines if the request contains an XHR (AJAX) header. Possible values: 0, 1

icap_agent

string

The version string of the installed agent or gateway, e.g. WindowsPC 4.3.20.5596, Gateway 1.2.45

feature_control

string

The name of the Feature Control rule that matched the request.

filter_rule

string

The name of the Filter Rule that matched the request.

final_action

string

The final action that was applied to the request. Possible values: Allowed, Blocked, Warn, Redirect, Quota

See Rules Engine Concepts.

log_level

int

The Log Level that was applied to the request. Possible values: Low, Normal, High, Do Not Log

matched_web_categories

array

A list of Web Category names that the filter rule matched for the requested URL.

web_categories

array

A list of Web Category names for the requested URL.

matched_url_categories

array

A list of URL Category names that the filter_rule matched for the requested URL.

url_category_names

array

A list of URL Category names for the requested URL.

matched_keyword_categories

array

A list of Keyword Category names that the filter_rule matched for the requested URL

keyword_category_names

array

A list of Keyword Category names for the requested URL.

icap_tx_uuid

string

A unique ID that can be used to match related data with this web hit. Only necessary for response scan analysis.

CASB - Inline Event

This schema is included in the "web" stream if the Cloud Application Security product is licensed.
{
    ...as per Web Hits,
    app_class: "Cloud Storage",
    app_name: "Dropbox",
    app_action: "Deleted a file/folder",
    app_data: "User deleted a file/folder.",
    baseline_risk: 75,
    custom_risk: 100
}
Schema Description

The schema is the same as Web Security with the addition of:

Field

Type

Description

app_class

string

The category of Cloud Application that the requested matched, e.g. Cloud Storage

app_name

string

The name of the Cloud Application that the request matched, e.g. Dropbox

app_action

string

The name of the action performed in the Cloud Application e.g. Upload File

app_data

string

The captured meta data from the Cloud Application action e.g. document1.docx. The type of meta data is dependent on the action. This may include HTML or URL encoded data.

baseline_risk

int

The baseline risk of the Cloud Application action

custom_risk

int

The overridden, custom risk, of the Cloud Application action

Email Security - Email Message

{
	"email_guid": "faa6690c-a7a1-49cb-b02f-2e299d563cd1",
	"originating_ip": "104.40.205.111",
	"email_size": 11040,
	"direction": "incoming",
	"delivery_state": "delivered",
	"from_address": "joey.d@acme.com",
	"to_addresses": [{
		"to_address": "frank@locateaware.com",
		"delivery_state": "delivered",
		"rcptTo": true,
		"is_internal": true
	}],
	"cc_addresses": [],
	"bcc_addresses": [],
        "dlp": [],
	"received_date": "Wed, 15 Feb 2023 14:08:46 +0000",
	"subject": "Log Streaming test",
	"attachments": [],
	"verdict": "Clean",
	"verdict_details": "No information",
	"country_code": "nl",
	"country": "netherlands",
	"city": "amsterdam",
	"latitude": 52.35,
	"longitude": 4.917,
	"rules": [{
		"rule_action": "DMARC Verifying Pass",
		"rule_description": "(Default) Signature Verification",
		"rule_data": "DKIM Pass or SPF Pass",
		"rule_data_name": "DKIM Pass or SPF Pass",
		"final_action": 0,
		"recipients": ["ALL"]
	}, {
		"rule_action": "Add Message Header",
		"rule_description": "(Default) Signature Verification",
		"rule_data": "...",
		"rule_data_name": "DKIMVereficationHeader",
		"final_action": 0,
		"recipients": ["ALL"]
	}, {
		"rule_action": "Re-write URL",
		"rule_description": "Linkscan",
		"rule_data": "...",
		"rule_data_name": "Click to Continue, Block on threat, Show target URL and Doc Scan",
		"final_action": 0,
		"recipients": ["ALL"]
	}, {
		"rule_action": "Deliver",
		"rule_description": "Deliver Inbound",
		"rule_data": "...",
		"rule_data_name": "route:[DomainRoute] ndr:[false] expire:[144]",
		"final_action": 1,
		"recipients": ["ALL"]
	}]
}
Schema Description

Field

Type

Description

direction

string

The direction of the email message. Possible values: incoming, outgoing

delivery_state

string

The delivery status for the message. Possible values: delivered, spam_quarantine, virus_quarantine, reject, delivery_error, dropped, rejected, delivery_delay

email_guid

string

A unique identifier for the email message e.g. 1cd4f6fc-2fce-4a36-a1c7-a8fb3037e95e

email_size

float

The size of the email message in bytes.

originating_ip

string

The IPv4 address of the sender.

from_address

text

The email address of the sender.

to_address

text

A JSON array of objects in the format {"to_address": "email@domain.com"}. Also includes meta data about delivery state.

cc_address

text

A JSON array of objects in the format {"cc_address": "email@domain.com"}. Also includes meta data about delivery state.

bcc_addresses

text

A JSON array of objects in the format {"bcc_address": "email@domain.com"}. Also includes meta data about delivery state.

received_date

timestamp

The timestamp with timezone that the server received the email message for processing, e.g. 2021-06-17 18:58:45+01

subject

text

The subject of the email message.

attachments

text

See Attachments Array below.

verdict

string

Deprecated. Possible values: clean, spam, virus, reject, enduser

verdict_details

text

Deprecated.

country_code

string

The ISO country code of the sender IP address

country

string

A string representing the country e.g. united states

city

string

A string representing the city e.g. london

latitude

float

The latitude of the sender IP address

longitude

float

The longitude of the sender IP address

rules

text

See Rules Array below.

Attachments Array

Field

Type

Length

Description

attachment_name

string

The filename of the attachment e.g. document1.docx

attachment_digest

string

The SHA256 hash of the file attachment.

attachment_mimetype

string

The MIME type for the attachment e.g. image/png

DLP Array (only populated if the Advanced DLP license is active)

Field

Type

Description

dlp_attachment

string

The filename of the attachment that triggerd a DLP rule

dlp_attachment_hash

string

The SHA256 of the dlp_mattchment

dlp_attachment_type

string

The MIME Type of the dlp_attachment

dlp_match

array

An array of matches containing

  • dlp_match_count - the number of matched items in the document
  • dlp_match_grammar - the grammar name that found a match
  • dlp_match_pattern - the pattern matched

dlp_match_score

int

The score calculated based on the number of matches in the document. The score will determine the severity level

dlp_match_severity

string

The severity level determined from the score - Low, Medium, High, Critical

dlp_policy_id

int

Internal ID

dlp_policy_name

string

The name of the DLP Policy

rule_id

int

Internal ID

rule_name

string

The Message Rule name containing the DLP Condition

rule_severity

string

The minimum severity level configured in the rule

Rules Array

Field

Type

Description

rule_data

string

The meta data captured by the rule, e.g. 5.0.1 user unknown

recipients

array

An array of email address strings.

rule_action

string

The rule action that was triggered e.g. Permanent Reject Error

final_action

int

Indicates whether the rule was a final action. Possible values: 0, 1

rule_data_name

string

Internal name for the rule_data.

rule_description

string

The name of the rule. See Connection Rules and Message Rules

Admin Audit

{
	"model": "RunReport",
	"payload": {
		"original": {},
		"changed": {
			"type": "web",
			"error": null,
			"count": 0,
			"filters": {
				"limit": 150,
				"interval": "hour",
				"page": 1,
				"start": 0
			}
		}
	},
	"ip_address": "46.8.172.157",
	"username": "admin@domain.com",
	"@timestamp": "2023-06-20T15:17:30.311682530Z",
	"url": "http://apiv2.clouduss.com/web/v2/hits?_dc=1687274241645",
	"event": {
		...can be ignored...
	},
	"country_code": "gb",
	"country": "united kingdom",
	"city": "london",
	"utc_timestamp": "2023-06-20 15:17:22.000000",
	"@version": "1",
	"method": "POST"
}
Schema Description

Field

Type

Description

model

string

A string representing the action performed in the Admin UI that was audited

payload

object

This has two child properties original and changed. Both contain unstructured JSON representing the before and after value of the action. This can include metadata such as filter parameters or additional context. If the action is to modify or update then the original will be included for comparison.

ip_address

string

The IPv4 address of the user that performed the action

username

string

The authenticated username of the user that performed the action

@timestamp

string

The timestamp that this log stream event was processed by logstash. Can be ignored.

url

string

The API URL or request URL where the action was performed.

event

object

A logstash wrapped version of the event. Can be ignored.

country_code

string

The ISO country code of the destination IP address.

country

string

A string representing the country e.g. united states.

city

string

A string representing the city e.g. london.

utc_timestamp

timestamp

Date and time the action was performed, in UTC.

@version

string

The logstash version. Can be ignored.

method

string

The HTTP method that performed the action e.g. POST = create, PUT = update, DELETE = delete.

How did we do?