Product Documentation

Netscope

Overview

Netscope enables service providers and enterprises (public and private infrastructures), ISPs, and third party service providers to access detailed Radar measurement logs, and standard reports in the form of summarized actionable data.Netscope offers several standard logs and reports that customers can use to measure the quality of their services.

Log Sharing and Delivery

  • Radar logs can be delivered real-time and daily.
  • Reports run daily.
  • Results are saved to Amazon Web Services S3 (S3) or Google Cloud Storage (GCS).
  • Logs and reports both have a 7-day retention period. They will be automatically deleted one week after creation.
  • Reports are usually in TSV (tab separated value) or JSON format depending on the type of report.

Customers are given login information to access the S3 and GCS buckets.They can use a command line tool like s3cmd or the AWS CLI for S3; or gsutil for GCS.The S3cmd config file recognizes the access keys (received via the Portal UI) and helps the user connect to the S3 bucket.The AWS CLI needs to be installed in the customer’s computer to connect to S3 and access the logs.For GCS, the customer receives the access key file as a download (via the Portal UI) which can be used with the gsutil tool.

For more on how to use the s3cmd config file to download files from S3 or how to use the Key File with gsutil or gcloud command line tools refer to the FAQ.

Customers also have an option to receive email notifications as and when reports become available.

Platform Settings

You must configure your platform to support and produce the data required for Netscope. Before you start, ensure that the following settings are enabled for your platform.

  • For Anonymous Best reports, enable Radar Probe Settings.
    • For Anonymous Best RTT, enable Response Time, and Availability.
    • For Anonymous Best Throughput, enable Throughput, and Availability.
  • For Cache Node ID reports, enable Radar Probe Settings; and in Advanced Radar Settings, enable Node ID.
  • For Resource Timing Details enable ‘Include Timestamps’ in Advanced Radar Settings.

From the main menu select Netscope.The Netscope Configuration page opens.

Navigation

Platforms and Networks

Select required Platforms or Networks (or both) to start the configuration process.

Note: Logs and Reports can be configured and generated only if at least one Platform or Network is selected.

The summarized data that the customer receives will include Radar measurements of selected Platforms (for all associated networks), and/or selected Networks (for all associated platform measurements).

Selecting Platforms

For content service providers or enterprises, select Platforms that represent the CDNs, clouds, data centers or other end-points for which measurements are required.

Platforms

Selecting Networks

For Internet Service Providers (ISP), select the Networks from the drop-down list associated with different platforms or endpoints for which measurements are required.

Note: If you don’t find the required platform on the list, you can configure it in the Platfo of the portal. For unavailable Networks, contact the support team.

Networks

Platform Reports

There are four types of Platform Reports:

  1. Anonymous Best for Round Trip Time (RTT)
  2. Anonymous Best for Throughput
  3. Cache Node ID
  4. Hourly by Country/ASN

For log descriptions go to Radar Log Descriptions and Reports for Service Providers and Enterprises.

Enable Platform Reports

Click the toggle button to enable or disable reports you want (or do not want) to receive.If an existing report is disabled, new logs will stop generating, but old reports will remain in the current location.

Platform Reports

Anonymous Best Report for Platforms

  • These reports help providers compare their performance to that of other platforms within their peer group i.e.within the same country, region, or ASN.
  • The performance data of the top 15 providers in the peer group is aggregated based on the same categories, and the best performing is listed next to the specific provider’s best value.
  • Anonymous Best Report for SSL Platforms are available so that their performance can be compared with other SSL platforms.
  • The client IPs are truncated to /28.
  • The results of the “best” provider helps clouds/CDNs focus performance efforts on high volume or business-critical ASNs that are competitively weak relative to their peers.
  • The report provides details on performance broken down by DNS resolver IP, Client IP /28, and the caching Node that served the objects, and compares it with the “best” platform for the same criteria.

Available for RTT and Throughput.

Cache Node ID Report for Platforms

  • This report is used to identify the specific server or data center that responded to a request and help diagnose server issues.
  • It provides the ID of the data center or machine that responded to a specific request.
  • It helps to understand why the performance through a specific node (POP or machine, or node ID), was good or bad.
  • Performance is broken-down by probe type (e.g., response time, throughput, and availability), the DNS resolver IP, Client IP /28, and the caching Node that served the objects.
  • For log descriptions refer to [Radar Log Descriptions and Reports for Service Providers and Enterprises] (#radar-log-descriptions-and-reports-for-service-providers-and-enterprises)

Hourly by Country/ASN

Network Reports

There are three types of Network Reports:

  1. Anonymous Best for Round Trip Time (RTT)
  2. Anonymous Best for Throughput
  3. Subnet

For log descriptions refer to Radar Log Descriptions and Reports for ISPs.

Enable Network Reports

Click the toggle button to enable/disable the reports you want to (or don’t want to) receive.If disabled, new logs will stop generating but old reports will remain in place. To generate a subnet report, enter the specific subnets of your networks. If there aren’t any subnets entered (but the report is enabled), it will be generated using the ASN CIDR block as the default subnet.

Network Reports

Anonymous Best Report for ISPs

  • In the Anonymous Best report for ISPs, a peer group is used for the “best” comparison.The peer group is based on the location of the ISP. It is usually the 10 most measured ISPs in a specified country, with a minimum of over 1000 sessions.
  • The results of the “best” ISP helps ISPs focus performance efforts on high volume or business-critical Platforms and areas that are competitively weak relative to their peers.
  • The report provides details on performance broken down by geography and Platform, and compares it with the “best” ISP for the same criteria.
  • Available for RTT and Throughput.
  • For log descriptions refer to Radar Log Descriptions and Reports for ISPs.

Subnet Report for ISPs

  • This report provides ISPs with information on how the specific subnets of their networks are performing for users through platforms that we measure.
  • It provides information about the service provider that responded to a specific request.
  • It helps to understand the performance by network subnet.
  • Performance is broken-down by probe type (e.g., response time, throughput, and availability), the DNS resolver IP, Client IP /28, and the subnet of the user that was served the objects.
  • For log descriptions refer to Radar Log Descriptions and Reports for ISPs.

Radar Logs

  • Radar logs are available for Platforms and Networks.
  • They include a subset of the fields available in the raw logs, with some data anonymized: client IP /28, Referer MD5 hashed.
  • Every measurement taken for public platforms is provided, regardless of the page that generated the measurement.

Note: Netscope never exposes full client IPs. Instead, it exposes the /28. For e.g., an IP of 255.255.255.255 will show up in a report as 255.255.255.240/28.

Log Frequency

Radar Logs can be generated daily (every 24 hours) i.e.end of day, UTC time.Logs can also be generated in real-time (minute by minute).

File Format

Choose TSV or JSON to receive logs and reports in either of these formats.

Measurement Type

You can configure logs for the following measurement types: Availability, Response Time, and Throughput. In the report, 1: Availability, 0: HTTP Response Time, and 14: HTTP Throughput.

Resource Timing Details

You can choose to also include Resource Timing details by clicking the Yes or No buttons.Resource timing details include,

  • DNS Lookup Time
  • TCP Connection Time
  • Secure Connection Time
  • Download Time

For log descriptions refer to Radar Log Descriptions and Reports for Service Providers and Enterprises.

Log Configuration

Log Frequency

Navigation Timing Logs can be generated daily (every 24 hours) i.e. end of day, UTC time. Logs can also be generated in real-time (minute by minute).

File Format

Choose TSV or JSON to receive Navigation Timing logs in either of these formats.

For log descriptions refer to Navigation Timing Log Descriptions.

Navigation Timing Logs

Openmix Logs

Log Frequency

Openmix logs are generated in real-time (i.e. minute by minute). These logs provide real-time measurements taken for Openmix customers.

File Format

Choose TSV or JSON to receive Openmix and HTTP Openmix logs in either of these formats. JSON is however the recommended format.

For log descriptions refer to Openmix Log Descriptions.

Openmix Logs

Cloud Service Delivery

This option allows you to select the mode of delivery.You can choose to receive logs and reports in either the AWS S3 bucket or in the Google Cloud Storage (GCS) bucket. You will be given login information to access the S3 and GCS buckets. You can use a command line tool like s3cmd or the AWS CLI for S3. And gsutil for GCS.

AWS S3

For logs and reports to be delivered to the AWS S3 bucket, select AWS S3.

Location

The Location represents the bucket in AWS S3 where the logs and reports will be saved.

IAM Keys

If you click the Generate Keys button under AWS S3, the AWS IAM Keys (Access and Secret keys) will be generated and displayed under IAM Keys. Be sure to record the keys because they are not stored anywhere for viewing later.

Note: This pair of access and secret keys are the only copy of the private keys. The customer is responsible for storing them securely. Regenerating new keys will invalidate the existing ones. The S3cmd config file recognizes the access keys (received via the Portal UI) and helps the customer connect to the S3 bucket. To connect to S3, the AWS CLI needs to be installed in the customer’s machine.

For info on how to use the access and secret keys with s3cmd to download reports from the S3 bucket, refer to the FAQ.

AWS S3

Google Cloud Storage

For logs and reports to be delivered to GCS, click Google Cloud Storage.

Location

The Location represents the bucket in Google Cloud Storage where logs and reports will be saved.

IAM Keys

When you click the Generate Key File button, the Google Service Account Key File will be downloaded to your machine.

Note: This key file serves as the only copy of the private key.Take note of your service account’s email address and securely store the service account’s private key file.Regenerating a new key file will invalidate the existing file.

This Key File can be used with the gsutil tool to download logs and reports from the GCS bucket.For details on how to use the Key File to download log files, refer to the FAQ.

GCS

Email Notification

Enter an email address here to receive notifications on daily reports.This is an optional field.

Navigation

Click Publish Configuration to save your settings.

Radar Log Descriptions and Reports for Service Providers and Enterprises

Partner Logs

  • These logs provide Radar measurements for benchmark partners.
  • They provide every measurement taken for public platforms, regardless of the page that generated the measurement.
  • Partner logs include a subset of the fields available in the raw logs, with some data anonymized: client IP /28, Referer MD5 hashed.

Note: Netscope never exposes full client IPs. Instead, it exposes the /28. For example, an IP of 255.255.255.255 will show up in a report as 255.255.255.240/28.

Log Descriptions

These are the columns headers and descriptions for the Partner Logs. The fields appear in this order in output files:

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
Unique Node ID Also known as cache node ID, an arbitrary value (typically an IP) that may be returned by CDN edge servers to help CDNs internally identify which server handled a particular request.’’ (empty string): Comes from Radar clients that do not support UNI detection.0: The user agent does not support the features needed for UNI detection.1: The client encountered an error during UNI detection, such an HTTP 404 or other unsuccessful response.2: UNI detection was attempted but resulted in an error.
Provider ID Internal ID of the platform that is being measured.
Probe Type The probe type being measured (e.g.1: HTTP Connect Time, 0: HTTP Response Time, 14: HTTP Throughput, etc.).We only use the information that it returned successfully (or not) within the allowed time to indicate that the service is available.
Response Code Result of the measurement.e.g.0: success, 1: timeout, > 1: error.For availability calculations the percent of measurements is taken with a 0 (success) response vs.the overall number of measurements (total, regardless of response).This, strictly is the availability calculation.For other probe types (RTT and throughput), this should be a filter (i.e.only consider RTT data points with a 0 success code when calculating statistics on the RTT).Same for throughput.
Measurement Value The recorded measurement value, the meaning of which varies by probe type. It represents availability (1)/ Response Time (0) measurements in milliseconds, and Throughput (14) in kbps.
Resolver Market The market of the DNS resolver that handled the request.Generally the continent where the DNS resolver is located, where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Resolver Country The country of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Resolver Region The region of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Resolver State The state of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Resolver City The city of the DNS resolver that handled the request.Resolver city is added by looking up resolver IP address.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/cities.json.gz
Resolver ASN The Autonomous System Number (ASN) of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Resolver IP The IP address of the DNS resolver from which our infrastructure received the DNS request.
Client Market The market of the end user that generated this measurement.Generally the continent where the client IP is located; where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Client Country The country of the end user that generated this measurement.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Client Region The region of the end user that generated this measurement.Generally the geographic region where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Client State The state of the end user that generated this measurement.Generally the state where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Client City The city of the end user that generated this measurement.Generally the city where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/cities.json.gz
Client ASN The Autonomous System Number (ASN) of the end user that generated this measurement.Generally the ASN that contains the client IP.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Referer Host MD5 The Referer information (Protocol, Host, and Path) comes from the Referer Header of the HTTP request to Radar.The Referer Host is MD5 hashed.
User Agent User agent string from the browser running the page that is hosting the tag.For example, if you’re running Chrome and browsing a page with the Radar tag, the radar measurements taken in the background will record the user agent from your Chrome browser.This includes the fact that it was Chrome, which version of Chrome it was, information about the OS that you’re running on, etc.
DNS Lookup Time (Optional) Using the Resource Timing API, the difference between the Domain Lookup End and the Domain Lookup Start when both values are not null and the end time is greater than the start time. It is calculated as domainLookupEnd - domainLookupStart.
TCP Connect Time (Optional) Using the Resource Timing API, the difference between the Connect End and Connect Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - connectStart.
Secure Connect Time (Optional) Using the Resource Timing API, the difference between the Connect End and Secure Connection Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - secureConnectionStart.
Latency (Optional) Using the Resource Timing API, the difference between the Response Start and Request Start when both values are not null and the response start time is greater than the request start time. It is calculated as responseStart - requestStart
Download Time (Optional) Using the Resource Timing API, the difference between the Response End and Response Start when both values are not null and the end time is greater than the start time. It is calculated as responseEnd - responseStart.
Client Profile This helps to identify if the data is coming from mobile apps or browsers. It also allows us to differentiate between iOS, Android apps, and browsers. Each client profile is identified by a number. The values for this field are: null, 0, 1, 2, 3, 4. Where, null: Generally implies an older Radar client that does not support sending the client_profile value. 0: Browser; 1: iOS - Radar runner for iOS app written in Swift; 2: Android; 3: Browser on mobile version of website; 4: iOS - Radar Runner for iOS app written in Objective-C.
Client Profile Version The client profile version tells us what version of the Radar Runner code (for iOS) or AndroidRadar SDK (for Android) was used in the mobile app. This field is intended for internal use only.
Device Category All devices are categorized into one of the following: Smartphone, Tablet, Personal computer, Smart TV, and Other. ‘Other’ is used as the default value if the parser is unable to determine the value for any of the fields.
Device The type of device the user is on, for e.g. Apple iPhone. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
Browser The type of browser that the user is using, for e.g. Mobile Safari UI/WKWebView 0.0.0. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
OS The operating system that is being used, for e.g. iOS 11.0.3. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.

Anonymous Best Report

  • Anonymous best reports help providers compare their performance to that of other platforms within their peer group i.e.within the same country, region, or ASN.
  • The performance data of the top 15 providers in the peer group is aggregated based on the same categories, and the best performing is listed next to the specific provider’s best value.
  • Anonymous Best Report for SSL platforms are available so that their performance can be compared with other SSL platforms.
  • The client IPs are truncated to /28.
  • The results of the “best” provider helps clouds/CDNs focus performance efforts on high volume or business-critical ASNs that are competitively weak relative to their peers.
  • The report provides details on performance broken down by DNS resolver IP, Client IP /28, and the caching Node that served the objects, and compares it with the “best” platform for the same criteria.
  • Available for RTT or Throughput.

Log Descriptions

These are the columns headers and descriptions for the Anonymous Best Report. The fields appear in this order in output files.

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
Resolver Country The country of the DNS resolver that handled the request.
Resolver Region The region of the DNS resolver that handled the request.
Resolver State The state of the DNS resolver that handled the request.
Resolver ASN ID The Autonomous System Number of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.
Resolver ASN Name The name of the ASN.
Resolver IP The IP of the DNS resolver that handled the request.
Client Country The country of the end user that generated this measurement.
Client Region The region of the end user that generated this measurement.
Client State The state of the end user that generated this measurement.
Client ASN The Autonomous System Number (ASN) number of the end user that generated this measurement.Generally the ASN that contains the client IP.
Client ASN Name The name of the ASN of the end user that generated the measurement.
Client IP The IP of the end user that generated the measurement.
Success Total number of measurements that were successful.Tip: Success / Total == Availability.
Timeout The number of measurements that timed out.
Error The number of measurements that were errors.
Total The total number of measurements.
Mean The average of all the measurements values for that row.
Best Mean The best mean out of the top 15 providers in the peer group.
Best Mean Count Total number of measurements that produced the best mean count.
Median The 50th percentile value, also known as the middle value of the measurements for a particular provider, if the measurements are listed in order.
Best Median The best 50th percentile value (below which 50 percent of the measurements may be found) of the top 15 providers in the peer group.
Best Median Count Total number of measurements that produced the best_median
5th The 5th percentile value for the provider.
Best 5th The best 5th percentile value out of the top 15 providers in the peer group.
Best 5th Count Total number of measurements that produced the best_5th
10th The 10th percentile value for the provider.
Best 10th The best 10th percentile value out of the top 15 providers in the peer group.
Best 10th Count Total number of measurements that produced the best_10th
90th The 90th percentile value for the provider.
Best 90th The best 90th percentile value out of the top 15 providers in the peer group.
Best 90th Count Total number of measurements that produced the best_90th
95th The 95th percentile value for the provider.
Best 95th The best 95th percentile value out of the top 15 providers in the peer group.
Best 95th Count Total number of measurements that produced the best_95th
Stdev The standard deviation for the provider
Best Stdev The best standard deviation out of the top 15 providers in the peer group.
Best Stdev Count Total number of measurements that produced the best std.dev.
Availability The availability in percentage for the provider.Availability is the probe Success rate i.e.Successes / (Successes + Fails + Timeouts)
Best Avail The best availability value out of the top 15 providers in the peer group.
Best Avail Count The number of measurements that produced the best availability
Importance Synthetic value generated to help find actionable data.
Node IDs These are a comma-separated list of the Unique Node IDs for the measurements represented by that row.

Cache Node ID Report (previously Multi-Service Provider Report)

This report is used to identify the specific server or datacenter that responded to a request and help diagnose server issues.

  • It provides the ID of the datacenter or machine that responded to a specific request.
  • It helps to understand why the performance through a specific node (POP or machine, or node ID), was good or bad.
  • Performance is broken-down by probe type (e.g., response time, throughput, and availability), the DNS resolver IP, Client IP /28, and the caching Node that served the objects.

Log Descriptions

These are the columns headers and descriptions for the Cache Node ID Report. The fields appear in this order in output files:

Log Description
Provider Name This is the name of the provider that is being measured.
Measurement Value The recorded measurement value, the meaning of which varies by probe type. It represents connect (1)/RTT (0) measurements in milliseconds, and throughput (14) measurements in kbps.
Unique Node ID Also known as cache node ID, an arbitrary value (typically an IP) that may be returned by CDN edge servers to help CDNs internally identify which server handled a particular request.’’ (empty string): Comes from Radar clients that do not support UNI detection.0: The user agent does not support the features needed for UNI detection.1: The client encountered an error during UNI detection, such an HTTP 404 or other unsuccessful response.2: UNI detection was attempted but resulted in an error.
Resolver Country The country of the DNS resolver that handled the request.
Resolver Region The region of the DNS resolver that handled the request.
Resolver State The state of the DNS resolver that handled the request.
Resolver ASN The Autonomous System Number of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.
Resolver ASN Name The name of the ASN.
Resolver IP The IP of the DNS resolver that handled the request.
Client Country The country of the end user that generated this measurement.
Client Region The region of the end user that generated this measurement.
Client State The state of the end user that generated this measurement.
Client ASN The Autonomous System Number (ASN) number of the end user that generated this measurement.Generally the ASN that contains the client IP.
Client ASN Name The name of the ASN of the end user that generated the measurement.
Client IP The IP of the end user that generated the measurement.
Success Total number of measurements that were successful.Tip: Success / Total == Availability.
Timeout The number of measurements that timed out.
Error The number of measurements that were errors.
Total The total number of measurements.
Mean The average of the measurement values for each row.
Median The 50th percentile value, also known as the middle value of the measurements for a particular provider, if the measurements are listed in order.
5th The 5th percentile value for the provider.
10th The 10th percentile value for the provider.
90th The 90th percentile value for the provider.
95th The 95th percentile value for the provider.
Stdev The standard deviation for the provider.
Availability The availability in percentage for the provider.
Importance Synthetic value generated to help find actionable data.

Hourly by Country/ASN

  • This report helps to verify if your providers’ performance varies significantly during the course of a day.
  • It shows the time when the measurements were taken truncated down to the hour, e.g. 2018-03-11T23:00:00.

Log Descriptions

These are the columns headers and descriptions for the Hourly by Country/ASN Report. The fields appear in this order in output files:

Log Description
Timestamp 60 minutes The UTC time when the measurements were taken truncated down to the hour, e.g.2018-03-11T23:00:00.
Provider Name This is the name of the provider that is being measured.
Measurement Type The recorded measurement value, the meaning of which varies by probe type. It represents HTTP_COLD (Availability), HTTP_RTT (Round Trip Time), or HTTP_KBPS (Throughput).
Client Country The country of the end user that generated this measurement.
Client ASN The Autonomous System Number (ASN) number of the end user that generated this measurement.Generally the ASN that contains the client IP.
Client ASN Name The name of the ASN of the end user that generated the measurement.
Success Total number of measurements that were successful.Tip: Success / Total == Availability.
Timeout The number of measurements that timed out.
Error The number of measurements that were errors.
Total The total number of measurements.
Mean The average of the measurement values for each row.
Median The 50th percentile value, also known as the middle value of the measurements for a particular provider, if the measurements are listed in order.
5th The 5th percentile value for the provider.
10th The 10th percentile value for the provider.
90th The 90th percentile value for the provider.
95th The 95th percentile value for the provider.
Stdev The standard deviation for the provider.
Availability The availability in percentage for the provider.
Importance Synthetic value generated to help find actionable data.
Provider ID The internal Citrix ID number for that provider.

Radar Log Descriptions and Reports for ISPs

Radar Logs

Radar logs enable ISPs to measure their performance against global platforms in detail.ISPs can use this data to find areas where improvements should be made or to verify the expected performance.

  • Provides access to Radar measurements.
  • Provides measurements taken from ISPs on public platforms, regardless of the page that generated the measurement.
  • Radar logs include a subset of the fields available in the raw logs, with some data anonymized: client IP /28, referer MD5 hashed.
  • The log files are in TSV format.

Log Descriptions

These are the columns headers and descriptions for the Radar logs for ISPs. The fields appear in this order in output files.

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
Provider ID Internal ID of the platform that is being measured.
Probe Type The probe type being measured (e.g.1: HTTP Connect Time, 0: HTTP Response Time, 14: HTTP Throughput, etc.).We only use the information that it returned successfully (or not) within the allowed time to indicate that the service is available.
Response Code Result of the measurement.e.g.0: success, 1: timeout, > 1: error.For availability calculations the percent of measurements is taken with a 0 (success) response vs.the overall number of measurements (total, regardless of response). This, strictly is the availability calculation.For other probe types (RTT and throughput), this should be a filter (i.e.only consider RTT data points with a 0 success code when calculating statistics on the RTT). Same for throughput.
Measurement Value The recorded measurement value, the meaning of which varies by probe type. It represents availability (1)/ Response Time (0) measurements in milliseconds, and Throughput (14) in kbps.
Resolver Market The market of the DNS resolver that handled the request.Generally the continent where the DNS resolver is located, where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Resolver Country The country of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Resolver Region The region of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Resolver State The state of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Resolver ASN The Autonomous System Number (ASN) of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Resolver IP The IP address of the DNS resolver from which our infrastructure received the DNS request.
Client Market The market of the end user that generated this measurement.Generally the continent where the client IP is located; where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Client Country The country of the end user that generated this measurement.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Client Region The region of the end user that generated this measurement.Generally the geographic region where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Client State The state of the end user that generated this measurement.Generally the state where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Client ASN The Autonomous System Number (ASN) of the end user that generated this measurement.Generally the ASN that contains the client IP.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Client IP The IP of the end user that generated this measurement.
Referer Host MD5 The Referer information (Protocol, Host, and Path) comes from the Referer Header of the HTTP request to Radar.The Referer Host is MD5 hashed.
User Agent This is the user agent string from the browser running the page that is hosting the tag. For example, if you’re running Chrome and browsing a page with the Radar tag, the radar measurements taken in the background will record the user agent from your Chrome browser.This includes the fact that it was Chrome, which version of Chrome it was, information about the OS that you’re running on, etc.
DNS Lookup Time (Optional) Using the Resource Timing API, the difference between the Domain Lookup End and the Domain Lookup Start when both values are not null and the end time is greater than the start time. It is calculated as domainLookupEnd - domainLookupStart.
TCP Connect Time (Optional) Using the Resource Timing API, the difference between the Connect End and Connect Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - connectStart.
Secure Connect Time (Optional) Using the Resource Timing API, the difference between the Connect End and Secure Connection Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - secureConnectionStart.
Latency (Optional) Using the Resource Timing API, the difference between the Response Start and Request Start when both values are not null and the response start time is greater than the request start time. It is calculated as responseStart - requestStart
Download Time (Optional) Using the Resource Timing API, the difference between the Response End and Response Start when both values are not null and the end time is greater than the start time. It is calculated as responseEnd - responseStart.
Client Profile This helps to identify if the data is coming from mobile apps or browsers. It also allows us to differentiate between iOS, Android apps, and browsers. Each client profile is identified by a number. The values for this field are: null, 0, 1, 2, 3, 4. Where, null: Generally implies an older Radar client that does not support sending the client_profile value. 0: Browser; 1: iOS - Radar runner for iOS app written in Swift; 2: Android; 3: Browser on mobile version of website; 4: iOS - Radar Runner for iOS app written in Objective-C.
Client Profile Version The client profile version tells us what version of the Radar Runner code (for iOS) or AndroidRadar SDK (for Android) was used in the mobile app. This field is intended for internal use only.
Device Category All devices are categorized into one of the following: Smartphone, Tablet, Personal computer, Smart TV, and Other. ‘Other’ is used as the default value if the parser is unable to determine the value for any of the fields.
Device The type of device the user is on, for e.g. Apple iPhone. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
Browser The type of browser that the user is using, for e.g. Mobile Safari UI/WKWebView 0.0.0. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
OS The operating system that is being used, for e.g. iOS 11.0.3. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.

Subnet Report for ISPs

  • This report provides ISPs with information on how the specific subnets of their networks are performing for their users through the platforms that we measure.
  • It provides information about the service provider that responded to a specific request.
  • It helps to understand the performance by network subnet.
  • Performance is broken-down by probe type (e.g., response time, throughput, and availability), the DNS resolver IP, Client IP /28, and the subnet of the user that was served the objects.

Log Descriptions

These are the columns headers and descriptions for the Subnet Report for ISPs.The fields appear in this order in output files:

Log Description
ASN Name The name of the Autonomous System from where the measurement was taken.
Measurement Value The recorded measurement value, the meaning of which varies by probe type. It represents connect (1)/RTT (0) measurements in milliseconds, and throughput (14) measurements in kbps.
Subnet The subnet of the user from where the request originated.
Resolver ASN The Autonomous System Number of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.
Resolver IP The IP of the DNS resolver that handled the request.
Client ASN The Autonomous System Number (ASN) number of the end user that generated this measurement.Generally the ASN that contains the client IP.
Client IP The IP of the end user that generated the measurement.
Platform ID The ID of the Service Provider platform to which the query was attempted.
Platform Name The name of the Service Provider platform to which the query was attempted.
Success Total number of measurements that were successful.Tip: Success / Total == Availability.
Timeout The number of measurements that timed out.
Error The number of measurements that were errors.
Total The total number of measurements.
Mean The average of the measurement values for each row.
Median The 50th percentile value, also known as the middle value of the measurements for a particular provider, if the measurements are listed in order.
5th The 5th percentile value for the provider.
10th The 10th percentile value for the provider.
90th The 90th percentile value for the provider.
95th The 95th percentile value for the provider.
Stdev The standard deviation for the provider.
Availability The availability in percentage for the provider.
Importance Synthetic value generated to help find actionable data.

Anonymous Best Report for ISPs

  • In the Anonymous Best report, a peer group is used for the “best” comparison.The peer group is based on the location of the ISP. It is usually the 10 most measured ISPs in a specified country, with a minimum of over 1000 sessions.
  • The results of the “best” ISP helps ISPs focus performance efforts on high volume or business-critical platforms and areas that are competitively weak relative to their peers.
  • The report provides details on performance broken down by geography and Platform, and compares it with the “best” ISP for the same criteria.
  • Available for RTT and Throughput.

Log Descriptions

These are the columns headers and descriptions for the Anonymous Best Report. The fields appear in this order in output files.

Log Description
Resolver Country The country of the DNS resolver that handled the request.
Resolver Region The region of the DNS resolver that handled the request.
Resolver State The state of the DNS resolver that handled the request.
Resolver ASN The Autonomous System Number of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.
Resolver IP The IP of the DNS resolver that handled the request.
Client Country The country of the end user that generated this measurement.
Client Region The region of the end user that generated this measurement.
Client State The state of the end user that generated this measurement.
Client ASN The Autonomous System Number (ASN) number of the end user that generated this measurement.Generally the ASN that contains the client IP.
Platform ID The ID of the Service Provider platform to which the query was attempted.
Platform Name The name of the Service Provider platform to which the query was attempted.
Success Total number of measurements that were successful.Tip: Success / Total == Availability.
Timeout The number of measurements that timed out.
Error The number of measurements that were errors.
Total The total number of measurements.
Mean The average of all the measurements values for that row.
Best Mean The best mean out of the top 15 providers in the peer group.
Best Mean Count Total number of measurements that produced the best mean count.
Median The 50th percentile value, also known as the middle value of the measurements for a particular provider, if the measurements are listed in order.
Best Median The best 50th percentile value (below which 50 percent of the measurements may be found) of the top 15 providers in the peer group.
Best Median Count Total number of measurements that produced the best_median
5th The 5th percentile value for the provider.
Best 5th The best 5th percentile value out of the top 15 providers in the peer group.
Best 5th Count Total number of measurements that produced the best_5th
10th The 10th percentile value for the provider.
Best 10th The best 10th percentile value out of the top 15 providers in the peer group.
Best 10th Count Total number of measurements that produced the best_10th
90th The 90th percentile value for the provider.
Best 90th The best 90th percentile value out of the top 15 providers in the peer group.
Best 90th Count Total number of measurements that produced the best_90th
95th The 95th percentile value for the provider.
Best 95th The best 95th percentile value out of the top 15 providers in the peer group.
Best 95th Count Total number of measurements that produced the best_95th
Stdev The standard deviation for the provider.
Best Stdev The best standard deviation out of the top 15 providers in the peer group.
Best Stdev Count Total number of measurements that produced the best std.dev.
Availability The availability in percentage for the provider.Availability is the probe Success rate i.e.Successes / (Successes + Fails + Timeouts)
Best Avail The best availability value out of the top 15 providers in the peer group.
Best Avail Count The number of measurements that produced the best availability.

Navigation Timing data provides insights into the various parts of the page load process for a webpage.

This data may vary due to the end user’s location, network issues, changes made by the provider, etc. Customers can use Navigation Timing data to optimize the end user’s experience in loading the monitored web page.

Measurements can be taken for every Radar session (if enabled). Each session is attached to an ID number that helps track all measurements from a session. These measurements are shared with customers as Navigation Timing Logs via Netscope.

These are the columns headers and descriptions for Navigation Timing logs. The fields appear in this order in output files:

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
Response Code Result of the measurement.e.g.0: success, 1: timeout, > 1: error.For availability calculations the percent of measurements is taken with a 0 (success) response vs.the overall number of measurements (total, regardless of response).This, strictly is the availability calculation.For other probe types (RTT and throughput), this should be a filter (i.e.only consider RTT data points with a 0 success code when calculating statistics on the RTT).Same for throughput.
Resolver Market The market of the DNS resolver that handled the request.Generally the continent where the DNS resolver is located, where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Resolver Country The country of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Resolver Region The region of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Resolver State The state of the DNS resolver that handled the request.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Resolver ASN The Autonomous System Number (ASN) of the DNS resolver that handled the request.Generally the ASN that contains the DNS resolver.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Resolver IP The IP address of the DNS resolver from which our infrastructure received the DNS request.
Client Market The market of the end user that generated this measurement.Generally the continent where the client IP is located; where, 0: Unknown (XX), 1:North America (NA) 5: Africa (AF), 3: Europe (EU), 4: Asia (AS), 2: Oceania (OC), 6: South America (SA).
Client Country The country of the end user that generated this measurement.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/countries.json.gz
Client Region The region of the end user that generated this measurement.Generally the geographic region where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/regions.json.gz Note: Not all countries of the world have defined regions.
Client State The state of the end user that generated this measurement.Generally the state where the client IP is located.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/states.json.gz Note: Not all countries of the world have defined states.
Client ASN The Autonomous System Number (ASN) of the end user that generated this measurement.Generally the ASN that contains the client IP.IDs can be mapped to names at https://s3-eu-west-1.amazonaws.com/community-radar/ref/asns.json.gz
Client IP The IP of the end user that generated the measurement.
Referer Host The Referer information (Protocol and Host) comes from the Referer Header of the HTTP request to Radar.
Referer Protocol The Referer information (Protocol and Host) comes from the Referer Header of the HTTP request to Radar.
Device Category All devices are categorized into one of the following: Smartphone, Tablet, Personal computer, Smart TV, and Other. ‘Other’ is used as the default value if the parser is unable to determine the value for any of the fields.
Device The type of device the user is on, for e.g. Apple iPhone. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
Browser The type of browser that the user is using, for e.g. Mobile Safari UI/WKWebView 0.0.0. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
OS The operating system that is being used, for e.g. iOS 11.0.3. This is detected by the user agent string from the browser running on the page that is hosting the Radar tag.
DNS Lookup Time Using the Resource Timing API, the difference between the Domain Lookup End and the Domain Lookup Start when both values are not null and the end time is greater than the start time. It is calculated as domainLookupEnd - domainLookupStart.
TCP Connect Time Using the Resource Timing API, the difference between the Connect End and Connect Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - connectStart.
Secure Connect Time Using the Resource Timing API, the difference between the Connect End and Secure Connection Start when both values are not null and the end time is greater than the start time. It is calculated as connectEnd - secureConnectionStart.
Load Event This is the duration or time taken to go from the start of a load event to the end of a load event. It is calculated as LoadEventEnd - LoadEventStart when both values are not null and the end time is greater than the start time.
Redirect This is the duration or time taken to go from Navigation Start to Fetch Start. It is calculated as FetchStart - NavigationStart when both values are not null and the end time is greater than the start time.
Total Page Load This is the duration or time taken to go from the start of navigation to the end of the page load event. It is calculated as - Load Event End - Navigation Start when both values are not null and the end time is greater than the start time.
DOM The duration or time taken to go from DOM loading to DOM complete. It is calculated as DomComplete - DomLoading when both values are not null and the end time is greater than the start time.
Latency Using the Resource Timing API, the difference between the Response Start and Request Start when both values are not null and the response start time is greater than the request start time. It is calculated as responseStart - requestStart
Download Time Using the Resource Timing API, the difference between the Response End and Response Start when both values are not null and the end time is greater than the start time. It is calculated as responseEnd - responseStart.
DOM interactive The duration or time taken to go from Navigation Start to DOM Interactive. It is calculated as DomInteractive - NavigationStart when both values are not null and the end time is greater than the start time.
Start Render The duration or time taken to go from Navigation Start to Start Render. It is calculated as startRender - NavigationStart when both values are not null and the end time is greater than the start time.

Openmix and HTTP Openmix Logs

Openmix and HTTP Openmix logs allow customers to make use of real-time measurements to monitor the behavior of their Openmix apps. They can use this data to find areas where improvements can be made or to verify the expected performance of their apps.

  • These logs provide real-time measurements taken for Openmix customers.
  • The recommended file format for these logs are JSON, but they are available in TSV format as well.

Openmix Log Descriptions

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
App Owner Zone ID The zone ID for the application owner servicing the request.This value is always equal to 1.
App Owner Customer ID The customer ID for the application owner servicing the request.For HTTP requests this is generally coded in the request path and is used to look up which application to execute.
App ID The application ID within the customer’s account servicing the request.This is also generally coded in the HTTP request path.Application IDs start at 1 and are only unique to the customer, therefore you should fully qualify queries for a specific app ID by also querying on the appOwnerCustomerId.
App Version The version of the application that serviced the account.Each time an application is updated via the portal or the API, the version is incremented.The version that was running at the time of the request is recorded.This information can be specifically used to separate versioned logic over time as applications are updated.Note that hosts throughout the network will generally receive updates in a similar timeframe, but almost never at precisely the same moment.Therefore, it is likely that overlapping decisions in time may be using different versions of an app during the process of update.
App Name The name of the application that serviced the account.
Market The market of the end user that generated this measurement.
Country The country of the end user that generated this measurement.
Region The region of the end user that generated this measurement.
State The state of the end user that generated this measurement.
ASN ID The Autonomous System Number (ASN) of the end user that generated this measurement. Generally the Autonomous System Number that contains the client IP.
ASN Name The name of the ASN of the end user that generated the measurement.
Effective IP The effective IP is the IP used to process the request.This is the query string-specified IP overriding the requesting IP (vs.the resolver/ECS/EDNS ID for the DNS flow). It is the address that the system considers the target when processing the information.This is either the IP of the requesting resolver, or the ECS IP address of the client if EDNS ECS is supported. So all probe performance data, geographic information, etc.passed to the application logic is based on this IP.
Resolver Market The market of the DNS resolver that handled the request.
Resolver Country The country of the DNS resolver that handled the request.
Resolver Region The region of the DNS resolver that handled the request.
Resolver State The state of the DNS resolver that handled the request.
Resolver ASN ID The Autonomous System Number (ASN) of the DNS resolver that handled the request. Generally the Autonomous System Number that contains the DNS resolver.
Resolver ASN Name The name of the ASN of the resolver that handled the request.
Resolver IP The IP of the DNS resolver that handled the request.
Decision Provider Name Alias of the platform that was selected by the application.
Reason Code Reason Code set within the application describing the reason behind the decision.
Reason Log This is a customer defined output from the Openmix app. It is an optional string field that enables customers to log information about their Openmix app decisions.
Fallback Mode This indicates whether the app was in fallback mode when it handled the request.Fallback happens when something failed during the preparation of the request for execution.
Used EDNS True if the application uses EDNS Client Subnet extension.
TTL The TTL (Time To Live) that was handed back.
Response The CNAME returned from the request.
Result The value in this field is always 1.
Context This is the summary of the Radar data that was available to Openmix when the request was handled.Openmix resolves Radar data relative to the effective values for every request, so two clients making requests at the same time may have different context strings.

Openmix HTTP API Log Descriptions

Log Description
Timestamp It is the UTC time of the request in YYYY-MM-DDTHH:MI:SSZ format.This is the actual value (down to the second) in the log tables, and is rounded to the nearest hour (2018-03-30T23:00:00Z) or day (2018-03-30T00:00:00Z) in the hour/day tables, respectively.This is always in UTC in all datasets.
App Owner Zone ID The zone ID for the application owner servicing the request.This value is always equal to 1.
App Owner Customer ID The customer ID for the application owner servicing the request.For HTTP requests this is generally coded in the request path and is used to look up which application to execute.
App ID The application ID within the customer’s account servicing the request.This is also generally coded in the HTTP request path.Application IDs start at 1 and are only unique to the customer, therefore you should fully qualify queries for a specific app ID by also querying on the appOwnerCustomerId.
App Version The version of the application that serviced the account.Each time an application is updated via the portal or the API, the version is incremented.The version that was running at the time of the request is recorded.This information can be specifically used to separate versioned logic over time as applications are updated.Note that hosts throughout the network will generally receive updates in a similar timeframe, but almost never at precisely the same moment.Therefore, it is likely that overlapping decisions in time may be using different versions of an app during the process of update.
App Name The name of the application that serviced the account.
Market The market of the end user that generated this measurement.
Country The country of the end user that generated this measurement.
Region The region of the end user that generated this measurement.
State The state of the end user that generated this measurement.
ASN ID The ID of the Autonomous System Number (ASN) of the end user that generated this measurement i.e. the network ID number associated with ASN Name
ASN Name The name of the ASN of the end user that generated the measurement.
Effective IP The effective IP is the IP used to process the request.This is the query string-specified IP overriding the requesting IP (vs.the resolver/ECS/EDNS ID for the DNS flow).It is the address that the system considers the target when processing the information.This is either the IP of the requesting resolver, or the ECS IP address of the client if EDNS ECS is supported. So all probe performance data, geographic information, etc., passed to the application logic is based on this IP.
Decision Provider Name Alias of the platform that was selected by the application.
Reason Code Reason Code set within the application describing the reason behind the decision.
Reason Log This is a customer defined output from the Openmix app. It is an optional string field that enables customers to log information about their Openmix app decisions.
Fallback Mode This indicates whether the app was in fallback mode when it handled the request.Fallback happens when something failed during the preparation of the request for execution.
Response Code Result of the measurement.e.g.0: success, 1: timeout, > 1: error.For availability calculations the percent of measurements is taken with a 0 (success) response vs.the overall number of measurements (total, regardless of response).This, strictly is the availability calculation.For other probe types (RTT and throughput), this should be a filter (i.e.only consider RTT data points with a 0 success code when calculating statistics on the RTT).Same for throughput.
HTTP Method The HTTP method (GET/POST/OPTIONS/etc) relates to the request that was made to the HTTP Openmix server from customer service.Together these constitute portions of the URL inbound and the HTTP responses outbound.
URI This is the request path.If customers aren’t getting the behavior they want, it could be due to an improperly structured request.Looking here in the logs may be a clue as it will show you what our servers are receiving.
Referer (Protocol, Host & Path) The Referer information (Protocol, Host, and Path) comes from the Referer Header of the HTTP request to Radar.For HTTP OPX the whole Referer (protocol, host, and path) is included in one string labeled Referer.
User Agent User agent string from the browser running the page that is hosting the tag.For example, if you’re running Chrome and browsing a page with the Radar tag, the radar measurements taken in the background will record the user agent from your Chrome browser.This includes the fact that it was Chrome, which version of Chrome it was, information about the OS that you’re running on, etc.
Context This is the summary of the Radar data that was available to Openmix when the request was handled.Openmix resolves Radar data relative to the effective values for every request, so two clients making requests at the same time may have different context strings.

Custom Reports for Third Party Organizations

Customers can work with Citrix to get custom reports based on Radar data that Citrix collects. Citrix can generate reports to run on a schedule.These reports are available as data files, usually in TSV format.

FAQ

Radar

How frequently are files pushed to S3 and GCS?

The frequency of file deposits is once a minute for Radar and daily for reports.

Where are the reports stored?

S3 Legacy (Location 1):

s3://public-radar/[customer name]/

S3 (Location 2):

s3://cedexis-netscope/[customer id]/

GCS (Location 3):

gs://cedexis-netscope-[customer id]/

How to obtain S3 access credentials if you don’t have them already?

The Portal UI provides an ‘Access’ and ‘Secret’ key that customers can use with ‘s3cmd’ or ‘awscli’ (the AWS command line client) or other tools to access S3 and get the files.For Google Storage, the Portal downloads a file with access credentials to use with the ‘gsutil’ tool.

How to use the access and secret keys with s3cmd to download logs and reports from the S3 bucket?

First you would need to download and install the s3cmd from https://s3tools.org/download, and refer to https://s3tools.org/usage for usage, options, and commands. Then run the following command:

s3cmd --access_key=[access key] --secret_key=[secret key] ls s3://cedexis-netscope/<customer id>/radar/

To download the files, run the following command:

s3cmd --access_key=[access_key] --secret_key=[secret_key] get s3://cedexis-netscope/<customer id>/radar/[the_filename_to_download] [the_name_of_the_local_file]

How to use the s3cmd config to list files in the S3 bucket

The first step is to Install s3cmd. You can install it from http://s3tools.org/download

To configure s3cmd, run the following command

s3cmd ls s3://cedexis-netscope/[customer id]/

If you’re already using s3cmd with another set of access and secret keys, then follow these steps:

If you already use s3cmd, then make a copy of the default config, at ~/.s3cfg.For example, you can make a copy called ~/.s3cfg_netscope.Replace the access and secret key entries in ~/.s3cfg_netscope with the ones we provide. Use the new configuration instead of the default one (your company’s) to access the our S3 bucket with the following command:

s3cmd -c ~/.s3cfg_netscope ls s3://cedexis-netscope/[customer id]/

The main difference is you have to put in a -c and where the config file is with the Citrix-provided access and secret keys.

If you need to be able to switch between sets of keys, your best bet is to embed them in a file and refer to the file with the -c option to specify which key pair you are using.

Note: The -c parameter just says where the config file is containing the access and secret keys.

How to use the Key File with gsutil or gcloud to download log files

Once the google service account JSON Key File is downloaded, it can be used to authenticate your google account credentials so that you can view or download your log files. For example, here’s one way to do that using the google gcloud and gsutil command line utilities:

Step 1: Activate the Key File

The authenticating commands gcloud auth activate- or gsutil config -e are required to authenticate the key file for running gcloud or gsutil commands.

For gcloud:

Run the following command using the downloaded Key File:

gcloud auth activate-service-account --key-file [downloaded config file]

Or

gcloud auth activate-service-account --key-file=[path and file name of key file]

For gsutil:

Run the following command using the downloaded config file:

gsutil config -e

Step 2: List the files in the GCS (Google Cloud Storage) Bucket

Once you’ve activated the service account key file (as described in the previous step), use the following command to list the files in the GCS bucket:

gsutil ls gs://cedexis-netscope-<customer id>

Step 3 (if required): Restore original credentials (or switch back and forth between accounts)

You can switch between the Citrix account and other Google Cloud credentials you’ve authenticated by doing the following.

First, run the following command to list all your accounts:

gcloud auth list

Then use the following command to switch to another account:

gcloud config set account [email of the account to switch to as shown in gcloud auth list]

You can switch back and forth between accounts using the same command, by replacing the email with the account email that you want to switch to.

What does the filename look like?

Legacy Daily:

The Radar daily log share file names have this structure:

<prefix><date: YYYY-MM-DD>.<customer_id>.part<uniq_id>.kr.txt.gz

E.g. Cedexis_Daily-2017-11-07.21222.part-cc901e1dd55eal4e.kr.txt.gz (non-standard example)

Legacy Real-time:

The Radar real-time log share file names have this structure:

<prefix><customer_id>-YYYY-MM-DDTHH:MM<uniq_id>.txt.gz

E.g. Cedexis_3-32291-2017-11-08T20:56-cc907e8fd71eaf4e.txt.gz

Netscope Format:

The Netscope format for daily and real-time log share files have this structure:

<freq><log_type><prefix><id_type><id><iso_dt><uniq_id>.<line_format>.gz

Where,

  • freq: “daily” “rt” “hr”
  • log_type: “radar” “opx” “hopx”
  • prefix: log_share.prefix
  • id_type: “customer” “provider” “asn”
  • id: log_share.match_id
  • iso_dt: iso 8601 Date_time “YYYYMMDDTHHMMSSZ”
  • uniq_id: hash(UUID)
  • line_format: “tsv” “json”

E.g. rt-radar-TestRadar1-provider-20363-20171209183034Z-cc907e8fd71eaf4e.tsv.gz

What is the format of the output file?

For Radar, the output file format is TSV (tab-separated value), gzipped.

Openmix and Openmix HTTP API

How frequently are files pushed to S3?

The frequency of file deposits is once a minute for Openmix and HTTP Openmix.

What if you’re unable to see the option to configure Openmix & Openmix HTTP API real time log sharing?

Your Account Manager can enable the required role for you to configure and enable Openmix & Openmix HTTP API real time log sharing.

How do you turn on Openmix & Openmix HTTP API real time log sharing and access files?

Once the role is enabled on your account, you will see the Manage Logs icon.Clicking it opens the Logs dialog where you can access Openmix Log Configuration settings.These settings are basically all you need to turn on Openmix & HTTP Openmix real time log sharing and access files.

Openmix log configuration

What is the back-end process?

Turning on Openmix log sharing enables Openmix HTTP API log sharing as well.The Openmix and Openmix HTTP API log sharing services should start outputting logs for the customer within 10 minutes.

Where are the Openmix and HTTP Openmix reports stored?

S3 Legacy (Location 1):

s3://logshare/[zone ID]/[customer ID]/logs/openmix/json/[YYYY]/[MM]/[DD]/[HH]/.

S3 (Location 2):

s3://cedexis-netscope/[customer id]/

GCS (Location 3):

gs://cedexis-netscope-[customer id]/

What does the filename look like?

The filename structure for Openmix and HTTP Openmix typically looks like this:

Legacy Real-time:

[zone ID, 1][customerID]-openmix-json[YYYY][MM][DD][HH][mm][ss]Z-m1-w9-c0.gz

Netscope Format:

The Netscope format for daily and real-time log share files have this structure:

<freq><log_type><prefix><id_type><id><iso_dt><uniq_id>.<line_format>.gz

Where,

  • freq: “daily” “rt” “hr”
  • log_type: “radar” “opx” “hopx”
  • prefix: log_share.prefix
  • id_type: “customer” “provider” “asn”
  • id: log_share.match_id
  • iso_dt: iso 8601 Date_time “YYYYMMDDTHHMMSSZ”
  • uniq_id: hash(UUID)
  • line_format: “tsv” “json”

E.g. hr-opx-TestOpenmix1-provider-20363-20171209183034Z-cc907e8fd71eaf4e.tsv.gz

What is the output file format?

The file format for Openmix and Openmix HTTP API is JSON (gzipped)