{"id":3411,"date":"2021-03-08T00:57:06","date_gmt":"2021-03-08T00:57:06","guid":{"rendered":"https:\/\/wiki.thomasandsofia.com\/?p=3411"},"modified":"2021-07-21T20:45:06","modified_gmt":"2021-07-21T20:45:06","slug":"streaming-data-policy","status":"publish","type":"post","link":"https:\/\/wiki.thomasandsofia.com\/?p=3411","title":{"rendered":"Streaming Data Policy"},"content":{"rendered":"<h1>Introduction to Streaming Data Policy<\/h1>\n<ul>\n<li>Is a work in progress.<\/li>\n<li>Currently only available via API, but UI coming soon.<\/li>\n<li>Day 1\n<ul>\n<li>https:\/\/drive.google.com\/file\/d\/1QNU4eOYDvl5WL-EI0hEDD1q_zBK2M-jp\/view<\/li>\n<\/ul>\n<\/li>\n<li>Day 2\n<ul>\n<li>https:\/\/drive.google.com\/file\/d\/1QNtKAeYaoXDoxp6hye_0HzVbAlwS2IqN\/view<\/li>\n<\/ul>\n<\/li>\n<li>Resources\n<ul>\n<li>https:\/\/s3.us-east-2.amazonaws.com\/s3.thomasandsofia.com\/zenoss\/training\/streaming-data\/zenoss-training-sources.tgz<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h1>What is Streaming Data Policy used for?<\/h1>\n<ul>\n<li>The purpose of streaming data policy is to configure the way incoming metric and model data is processed.<\/li>\n<li>It allows for the ingest of data that does not conform to the structure Zenoss Cloud expects\n<ul>\n<li>Example: Metrics created by OpenCensus do not include <code>dimentions<\/code> or <code>metadataFields<\/code> fields, but do include a <code>tags<\/code> field.<\/li>\n<li>OpenCensus also does not provide a way to publish models.<\/li>\n<li>Policy to the rescue!<\/li>\n<\/ul>\n<\/li>\n<li>The default policy for OpenCensus extracts dimensions and metadata values from metric tags, and creates an entity from them.<\/li>\n<\/ul>\n<p><strong>OpenCensus Example<\/strong><\/p>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3415\" src=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1.png\" alt=\"\" width=\"1242\" height=\"947\" srcset=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1.png 1242w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1-300x229.png 300w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1-1024x781.png 1024w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1-768x586.png 768w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/opencensus1-150x114.png 150w\" sizes=\"auto, (max-width: 1242px) 100vw, 1242px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<h1>Zenoss Cloud Policy Types<\/h1>\n<h2>Data Source<\/h2>\n<ul>\n<li>Acts as an organizer for one or more Ingest policies with a common source-type.\n<ul>\n<li>If you want to apply a policy, you must include a source-type that matches a Data Source policy.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre>{\r\n  \"sourceType\": \"${USER}\/opencensus-python-exporter\",\r\n  policyReferences\": [\r\n    {\r\n      \"dataType\": \"METRIC\",\r\n      \"policyType\": \"INGEST\",\r\n      \"policyName\": \"${USER}_metricIngest_opencensus\"\r\n    },\r\n    {\r\n      \"dataType\": \"DATAMAP\",\r\n      \"policyType\": \"INGEST\",\r\n      \"policyName\": \"${USER}_modelIngest_opencensus\"\r\n    }\r\n  ]\r\n}<\/pre>\n<h2>Ingest<\/h2>\n<ul>\n<li>An Ingest policy describes modifications to be made to matching incoming metrics, models and events.<\/li>\n<li>Currently, events are only ingested from Collection Zones.\u00a0 The Zenoss Cloud event API is under development.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>Data Model Policy Example<\/p>\n<pre>{\r\n \"ingest\"{\r\n  \"name\": \"train_modelIngest_opencensus\",\r\n  \"dimensionKeys\": [\r\n   \"source\",\r\n   \"app\"\r\n  ],\r\n  \"useAllKeysAsDims\": false,\r\n  \"requiredKeys\": [\r\n   \"source\",\r\n   \"source-type\",\r\n   \"app\"\r\n  ],\r\n  \"metadataKeys\": [\r\n   \"source-type\",\r\n   \"name\"\r\n  ],\r\n  \"fieldTransforms\": [\r\n   {\r\n    \"sourceKey\": \"source\",\r\n    \"targetKey\": \"name\",\r\n    \"operation\": \"COPY\"\r\n   }\r\n  ],\r\n  \"generatedEntityId\": true,\r\n  \"entityDimensionKeys\": [\r\n   \"source\",\r\n   \"app\"\r\n  ],\r\n  \"excludeInternalFromAllKeysAsDims\": true\r\n }\r\n}<\/pre>\n<p>&nbsp;<\/p>\n<h2>High-Level Policy Logic<\/h2>\n<ul>\n<li>If:\n<ul>\n<li>A <code>data source<\/code> policy existys with a `source-type` that matches the incoming object&#8217;s <code>source-<\/code>type`<\/li>\n<\/ul>\n<\/li>\n<li>Then:\n<ul>\n<li>Apply the inject policy specified by the data source policy<\/li>\n<\/ul>\n<\/li>\n<li>Else:\n<ul>\n<li>Apply the default ingest policy\n<ul>\n<li>This really just checks to see if certain keys exist.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h1>The Policy API<\/h1>\n<ul>\n<li>Default Policy\n<ul>\n<li>Predefined policies provided by Zenoss Cloud and are read only.<\/li>\n<\/ul>\n<\/li>\n<li>Custom Policy\n<ul>\n<li>Policies defined by your organization<\/li>\n<li>Custom policies are always specific to your Zenoss Cloud instance (tenant)<\/li>\n<li>Custom policies override default policies with the same name.<\/li>\n<\/ul>\n<\/li>\n<li>Effective Policy\n<ul>\n<li>Policies that are currently in force.\n<ul>\n<li>In the case where a custom policy has the same name as a default policy, the custom policy is always the effective policy<\/li>\n<li>Example: Creating a custom policy with the name &#8220;application.k8s&#8221;\n<ul>\n<li>Overrides the default policy with that name<\/li>\n<li>Becomes the effective policy with that name.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>API Policy Methods<\/h2>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3418\" src=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods.png\" alt=\"\" width=\"1777\" height=\"252\" srcset=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods.png 1777w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods-300x43.png 300w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods-1024x145.png 1024w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods-768x109.png 768w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods-1536x218.png 1536w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2021\/03\/policy-methods-150x21.png 150w\" sizes=\"auto, (max-width: 1777px) 100vw, 1777px\" \/><\/a><\/p>\n<ul>\n<li>verb + category + policy-type\n<ul>\n<li>Verb: create, delete, get, list, rename, or update<\/li>\n<li>Category specifies the category of the targeted policy: default, effective or blank for custom policies<\/li>\n<li>Policy-Type specifies the type of the targeted policy: datasource(s) (depending on the verb) or ingest.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h1>OpenCensus Demonstration<\/h1>\n<p><a href=\"https:\/\/help.zenoss.com\/docs\/streaming-data-to-zenoss-cloud\/exporting-opencensus-data-to-zenoss-cloud\" target=\"_blank\" rel=\"noopener\">https:\/\/help.zenoss.com\/docs\/streaming-data-to-zenoss-cloud\/exporting-opencensus-data-to-zenoss-cloud<\/a><\/p>\n<ul>\n<li>opencensus-ext-zenoss 0.0.4<\/li>\n<li>pip install opencensus-ext-zenoss<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction to Streaming Data Policy Is a work in progress. Currently only available via API, but UI coming soon. Day 1 https:\/\/drive.google.com\/file\/d\/1QNU4eOYDvl5WL-EI0hEDD1q_zBK2M-jp\/view Day 2 https:\/\/drive.google.com\/file\/d\/1QNtKAeYaoXDoxp6hye_0HzVbAlwS2IqN\/view Resources https:\/\/s3.us-east-2.amazonaws.com\/s3.thomasandsofia.com\/zenoss\/training\/streaming-data\/zenoss-training-sources.tgz What is Streaming Data Policy used for? The purpose of streaming data policy is to configure the way incoming metric and model data is processed. It allows for ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/?p=3411\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[64,59],"tags":[],"class_list":["post-3411","post","type-post","status-publish","format-standard","hentry","category-streaming-data","category-zenoss"],"_links":{"self":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3411","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3411"}],"version-history":[{"count":11,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3411\/revisions"}],"predecessor-version":[{"id":3554,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/3411\/revisions\/3554"}],"wp:attachment":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3411"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3411"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3411"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}