Log Streaming to Amazon S3

Updated 1 year ago by admin

This guide is still being updated.

The Log Streaming service allows enriched logs from the USS platform to be streamed to external services such as SIEM solutions, analytics platforms and SoC services. To request access to the Log Streaming service, please contact your service provider.

This article requires an Amazon Web Services account with permission to create S3 buckets and policies
Please note that an Amazon S3 bucket will be needed for each product you want to stream data from
  1. Log in to AWS console and navigate to S3
  2. Click Create Bucket
  3. Enter a name for your bucket, e.g. clouduss-web-logs
  4. Select an AWS region appropriate for your requirements
  5. Ensure that Block all public access is selected
  6. Click Create Bucket
  7. Open the newly created bucket and click Properties and copy the AWS Region and Amazon Resource Name (ARN) somewhere safe. These need to be sent to your service provider along with an IAM Access Key and Secret, created in the next step.
  8. Optionally, it is best practice to attach a Life Cycle rule to the bucket to delete objects after 1 day. This will reduce the cost of storing the streamed files
  9. Create an IAM credential that has write access to the bucket with this policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<YOUR ARN HERE>/*"
        }
    ]
}
  1. Copy the key and secret and send these to your service provider, along with the bucket ARN and region.
  2. Wait for confirmation from your service provider that the log stream has been configured
  3. View the contents of your bucket in the Objects tab and you should see objects being created by the Log Streaming service within the clouduss-logstream prefix/folder:
Note that it could take up to an hour for objects to appear in the bucket for the first time. In most cases it is within 10-15 minutes.
  1. The log stream is now set up and you can consume the data in any service that supports Amazon S3 integration.

How did we do?