Introduction to Streaming Data Policy

• Is a work in progress.
• Currently only available via API, but UI coming soon.
• Day 1
  • https://drive.google.com/file/d/1QNU4eOYDvl5WL-EI0hEDD1q_zBK2M-jp/view
• Day 2
  • https://drive.google.com/file/d/1QNtKAeYaoXDoxp6hye_0HzVbAlwS2IqN/view
• Resources
  • https://s3.us-east-2.amazonaws.com/s3.thomasandsofia.com/zenoss/training/streaming-data/zenoss-training-sources.tgz

What is Streaming Data Policy used for?

• The purpose of streaming data policy is to configure the way incoming metric and model data is processed.
• It allows for the ingest of data that does not conform to the structure Zenoss Cloud expects
  • Example: Metrics created by OpenCensus do not include dimentions or metadataFields fields, but do include a tags field.
  • OpenCensus also does not provide a way to publish models.
  • Policy to the rescue!
• The default policy for OpenCensus extracts dimensions and metadata values from metric tags, and creates an entity from them.

OpenCensus Example

Zenoss Cloud Policy Types

Data Source

• Acts as an organizer for one or more Ingest policies with a common source-type.
  • If you want to apply a policy, you must include a source-type that matches a Data Source policy.

{
  "sourceType": "${USER}/opencensus-python-exporter",
  policyReferences": [
    {
      "dataType": "METRIC",
      "policyType": "INGEST",
      "policyName": "${USER}_metricIngest_opencensus"
    },
    {
      "dataType": "DATAMAP",
      "policyType": "INGEST",
      "policyName": "${USER}_modelIngest_opencensus"
    }
  ]
}

Ingest

• An Ingest policy describes modifications to be made to matching incoming metrics, models and events.
• Currently, events are only ingested from Collection Zones.  The Zenoss Cloud event API is under development.

Data Model Policy Example

{
 "ingest"{
  "name": "train_modelIngest_opencensus",
  "dimensionKeys": [
   "source",
   "app"
  ],
  "useAllKeysAsDims": false,
  "requiredKeys": [
   "source",
   "source-type",
   "app"
  ],
  "metadataKeys": [
   "source-type",
   "name"
  ],
  "fieldTransforms": [
   {
    "sourceKey": "source",
    "targetKey": "name",
    "operation": "COPY"
   }
  ],
  "generatedEntityId": true,
  "entityDimensionKeys": [
   "source",
   "app"
  ],
  "excludeInternalFromAllKeysAsDims": true
 }
}

High-Level Policy Logic

• If:
  • A data source policy existys with a `source-type` that matches the incoming object’s source-type`
• Then:
  • Apply the inject policy specified by the data source policy
• Else:
  • Apply the default ingest policy
    • This really just checks to see if certain keys exist.

The Policy API

• Default Policy
  • Predefined policies provided by Zenoss Cloud and are read only.
• Custom Policy
  • Policies defined by your organization
  • Custom policies are always specific to your Zenoss Cloud instance (tenant)
  • Custom policies override default policies with the same name.
• Effective Policy
  • Policies that are currently in force.
    • In the case where a custom policy has the same name as a default policy, the custom policy is always the effective policy
    • Example: Creating a custom policy with the name “application.k8s”
      • Overrides the default policy with that name
      • Becomes the effective policy with that name.

API Policy Methods

• verb + category + policy-type
  • Verb: create, delete, get, list, rename, or update
  • Category specifies the category of the targeted policy: default, effective or blank for custom policies
  • Policy-Type specifies the type of the targeted policy: datasource(s) (depending on the verb) or ingest.

OpenCensus Demonstration

https://help.zenoss.com/docs/streaming-data-to-zenoss-cloud/exporting-opencensus-data-to-zenoss-cloud

• opencensus-ext-zenoss 0.0.4
• pip install opencensus-ext-zenoss