Trusted Security Circles messages

The members of Trusted Security Circles Client share threat intelligence information with other Trusted Security Circles Client member profiles by means of anonymous, encrypted messages.

Trusted Security Circles profiles share threat intelligence information with other profiles in the circle using anonymous, encrypted messages. The messages are stored in a shared intelligence table with the content of the messages shared by other profiles. The central instance contains a remote message broker that is responsible for receiving threat intelligence sharing requests and sending the encrypted messages to participants in the Trusted Security Circle. It also brokers response messages to all participants using various REST web services. Supporting tables store messages pulled to customer instances via a REST endpoint. Similarly, customer instances can post information/messages. All communication is performed asynchronously. When messages are sent to a trusted circle, they are exploded into multiple messages, each targeted at the individual members of the circle. Messages that are not picked up within the specified time limit (48 hours) are removed from the message table. Messages use a compressed data type to reduce storage size. Each customer instance with Trusted Security Circles installed sends a stateless RESTful GET request to the central instance for messages every 30 seconds for each profile created. The central instance returns an empty response unless a message has been sent to the user. When messages are sent, the message payloads are relatively small. The message data is stored based on sightings for a specific observable. As these messages are processed, they reside in local memory. After the messages have been split into specific records, however, the message data contained in the client tables is cached to the central instance to keep the data up to date. A scheduled job called Refresh Central Data runs once per day to refresh all table data from the central instance to the local profiles. Users can manually update the table data by clicking a Refresh button on list views in the client instance

Remote message broker

Trusted Circle messages are encrypted. In ServiceNow this can be accomplished with various REST web services. Supporting tables can store messages for customer instances which those instances can pull via a REST endpoint. Similarly customer instances can post information/messages. The Trusted Security Circles plugins are responsible for initiating messages to the central instance and for receiving messages from this instance. All communication is performed asynchronously.

Messages sent to Trusted Circles are exploded into multiple messages, targeted at the circle members existing at the time of message sending. Messages not picked up within the specified time limit (currently 48 hours) are removed from the message table. Additionally, messages use a compressed data type to reduce storage size.

Each customer instance with a Trusted Security Circle plugin installed sends a request for messages every 30 seconds for each profile created. This is a stateless RESTful GET request to the central instance. This request returns an empty response unless a message has been sent to the user.

On the central instance, getting any messages for a profile is a single query in addition to the REST authentication. Since this table is cleaned regularly, this should not be an expensive query. Additionally, these messages are stored in compressed data fields.

When a message is sent, the message payloads are relatively small. In order to load test we are using a maximum message size of about 1.4MB. This data is stored based on sightings for a specific observable. This leads to a sightings record with a maximum size of about 4,500 bytes.

During message processing they reside in memory but once they have been split into specific records the message are not be persisted in the customer’s instance. The downloaded payload is not encrypted (except for SSL encryption in transport)

Data caching

Data caching is a term we are using to describe the logic of keeping the client tables (sometimes referred to as "local tables") up to date with the data stored on central. This is accomplished via REST APIs to central. There are different ways to trigger updating the local "cache". They are:

  • Scheduled job - "Refresh Central Data" which runs once a day and refreshes all table data.
  • Navigating to a List View - navigating to a list view such as viewing a list of circles. If the table supports data caching, there is a "Refresh" button at the top of the list. Pressing this button refreshes all data for this table.
  • Navigating to a single record - navigating to a form refreshes that record and all its visible related lists. This is accomplished via client script onload of the record form.
  • Programmatically - any arbitrary script can call the logic to get the latest data from central. Any "get" operation also updates the local cache.
  • Attachments - Central must accept and host files shared between components

Each message is a new item in the table, keyed by recipient. The Trusted Security Circles should also make sure we don't explode table if customers stop to check for messages, since this table is the one that has potential to grow rapidly. When messages are checked, they are removed from database. We should be able to send message to a trusted circle or individual profile, so we might have 2 different fields in message json, that is sent to API, but in messages table we only store id of recipient profile. This table includes these fields:

  • id – sys_id provided by platform.
  • profile_id - id of recipient, this should be indexed, since we're mostly going to perform selects based on it.
  • message – message itself, JSON string.
  • encrypted – boolean, to specify if message would have been encrypted if possible.