Service Graph for cloud native (Kubernetes) apps

Using the Service Graph feature in Citrix ADM, you can:

  • Ensure end-to-end observability of your application overall performance

  • Identify bottlenecks created by inter-dependency of different components of your applications

  • Gather insights into the dependencies of different components of your applications

  • Monitor services within the Kubernetes cluster

  • Monitor which service has issues

  • Check the factors contributing to performance issues

  • View detailed visibility of service HTTP transactions

  • Analyze the following metrics:

    • Total number of hits

    • Service Response time

    • Data volume

    • Errors

By visualizing these metrics in Citrix ADM, you can analyze the root cause of issues and take necessary troubleshooting actions faster. Service Graph displays your applications into various component services. These services running inside the Kubernetes cluster can communicate with various components within and outside the application.

Before you begin

To use service graph in Citrix ADM, ensure you have:

  • Kubernetes cluster with Citrix ADC CPX as a proxy.

  • Citrix ADM agent installed and configured to enable communication between Citrix ADM and Kubernetes cluster or managed instances in your data center or cloud.

  • Added required parameters in CPX yaml file to ensure successful CPX registration with Citrix ADM.

  • Static routes configured on Citrix ADM agent to enable communication between Citrix ADM and Citrix ADC CPX.

  • Kubernetes cluster added on Citrix ADM.

  • Auto-select Virtual Servers enabled to license the virtual servers.

  • Changed the Web Transaction Settings to All for Citrix ADM agent to get HTTP transactions.

  • Updated the Istream.conf file and change the TCP setting to All for Citrix ADM agent to get TCP metrics.

Configure Citrix ADM agent to register with Kubernetes cluster

To enable communication between Kubernetes cluster and Citrix ADM, you must install and configure a Citrix ADM agent. You can configure an agent using a hypervisor, public cloud services (such as Microsoft Azure, AWS), or built-in agent available on Citrix ADC instances (ideal for HA deployments).

Follow the procedure to configure an agent.

Note

You can also use an existing agent.

Add parameters in CPX yaml file

You must include the following parameters in the CPX yaml file to ensure CPX registration with Citrix ADM:

- name: "NS_MGMT_SERVER"
  value: "10.106.150.72"
- name: "NS_MGMT_FINGERPRINT"
  value: "E3:3A:2B:F7:CC:A6:3D:72:8F:3E:3E:4F:0D:C1"
- name: "NS_HTTP_PORT"
  value: "9080"
- name: "NS_HTTPS_PORT"
  value: 9443"
- name: "LOGSTREAM_COLLECTOR_IP"
  value: "10.106.150.72"

  • NS_MGMT_SERVER – Indicates the Citrix ADM agent IP address

  • NS_MGMT_FINGERPRINT – Indicates the authentication for CPX in Citrix ADM agent. To get the fingerprint:

    1. In Citrix ADM, navigate to Networks > Agents

    2. Select the agent and then click View Fingerprint

      Fingerprint

  • NS_HTTP_PORT – Indicates the HTTP port for communication

  • NS_HTTPS_PORT – Indicates the HTTPS port for communication

  • LOGSTREAM_COLLECTOR_IP – Indicates the Citrix ADM agent IP address, where Logstream protocol must be enabled to transfer log data from CPX to ADM

Configure static routes in Citrix ADM agent

Inside the Kubernetes cluster, all containerized pods use an overlay network. Establishing the communication using those private IP addresses directly is not possible. To enable communication from Citrix ADM to Kubernetes cluster, you need to configure static routing in Citrix ADM agent.

Consider that you have the following IP addresses for your Kubernetes cluster:

  • Kubernetes master – 10.106.157.112

  • Kubernetes worker 1 – 10.106.157.110

  • Kubernetes worker 2 – 10.106.157.111

After successfully configuring a Citrix ADM agent:

  1. Using an SSH client, log on to Citrix ADM agent

  2. Configure the static routing using the command route add -net <public IP address range> <Kubernetes IP address>

    For example:

    route add -net 192.168.0.0/24 10.106.157.112

    route add -net 192.168.1.0/24 10.106.157.111

    route add -net 192.168.2.0/24 10.106.157.110

  3. Verify the configuration by using netstat -rn

    static routing

  4. Append these route commands in /mpsconfig/svm.conf file (in Citrix ADM Agent).

Add Kubernetes cluster in Citrix ADM

After you configure a Citrix ADM agent and configure static routes, you must add the Kubernetes cluster in Citrix ADM.

To add the Kubernetes cluster:

  1. Log on to Citrix ADM with administrator credentials.

  2. Navigate to Orchestration > Kubernetes > Cluster. The Clusters page is displayed.

  3. Click Add.

  4. In the Add Cluster page, specify the following parameters:

    1. Name - Specify a name of your choice.

    2. API Server URL - You can get the API Server URL details from the Kubernetes Master node.

      1. On the Kubernetes master node, run the command kubectl cluster-info.

        API Server URL

      2. Enter the URL that displays for “Kubernetes master is running at.”

    3. Authentication Token - Specify the authentication token. The authentication token is required to validate access for communication between Kubernetes cluster and Citrix ADM. To generate an authentication token:

      1. On the Kubernetes master node, run the following commands:

        kubectl get secrets | grep ^default

        kubectl describe secret <SECRET_NAME>

        Note

        You can also create RBAC role and service account yamls for your Kubernetes cluster, and create an authentication token for the admin user.

      2. Copy the token that is generated.

        For more information, see Kubernetes documentation.

    4. Select the agent from the list.

    5. Click Create.

      add cluster

      You can view data in Service Graph, after enabling the auto-select virtual servers for licensing.

Enable Auto-select virtual servers for licensing

After you add Kubernetes cluster in Citrix ADM, you must ensure to auto-select virtual servers for licensing. Virtual servers need to be licensed to display data in Service Graph. To auto-select virtual servers:

  1. Navigate to Accounts > Subscriptions.

  2. Under Virtual Server License Summary, enable Auto-select Virtual Servers and Auto-select non addressable Virtual Servers.

    Auto-select virtual server

Enable Web Transaction setting

After you add the Kubernetes cluster and enable the auto-select virtual servers, change the Web Transaction Settings to All. To enable this setting:

  1. Navigate to Analytics > Settings.

    The Settings page is displayed.

  2. Click Enable Features for Analytics.

  3. Under Web Transaction Settings, select All and click OK.

    web-transaction-settings

View details in Service Graph

Navigate to Application > Service Graph and select the time duration from the list to view the service graph details.

Details-service-graph

1 - End-to-end network map of your application that shows how your component services are communicating

2 – Graph that indicates hits and errors for a specific time duration

3 – Search bar to search for services

4 – Time list to select the time duration

5 - Apply filters to display services

6 – Setting icon

7 – Zoom in and zoom out view

Based on the selected time duration, you can view the service graph. Select the time period from the graph that indicates hits to drill-down further for additional information.

Details-service-graph1

The service graph is now displayed with the protocol used by the services. Consider that you have the following services running in your Kubernetes cluster as shown in the image:

Services-kubernetes

You can view the following status for your services:

  • Critical (red) - Indicates when average service response time > 200 ms AND error count > 0

  • Review (orange) - Indicates when average service response time > 200 ms OR error count > 0

  • Good (green) - Indicates no error and average service response time < 200 ms

The following are protocols that enable you to identify the protocol used by a service:

  • TCP – Indicates the service is using the TCP protocol.

  • SSL, HTTP – Indicates the service is using the SSL over HTTP protocol.

  • SSL, TCP – Indicates the service is using the SSL over TCP protocol.

    Note

    The service without a protocol name indicates the service is using the HTTP protocol.

View client metrics

Hover the mouse pointer to view client metrics details for the communication between client and Ingress.

Client-rtt

  • Hits – Indicates the total number of requests from client to ingress.

  • Client RTT – Indicates the average client RTT from client to ingress.

  • Data Volume – Indicates the total data volume processed by client.

View ingress metrics

Hover the mouse pointer on Ingress to view the metrics details for Ingress.

Ingress-rtt

  • Hits - Indicates the total number of requests received by the ingress

  • ADC processing time – Indicates the average time taken by ADC instance to process the requests

  • Data volume – Indicates the total volume of data processed by the ingress

Using the TCP and SSL metrics, you can:

  • View TCP connection details between services

  • Determine if TCP-related issues are from the source or destination service

  • View if the SSL error is from the source or destination service

  • View the SSL protocol version used by SSL services

TCP metrics

Hover the mouse pointer over a TCP service or its associated incoming service to view the TCP metrics.
TCP

  • TCP connections – Total connections established between the services

  • Data Volume – Total data processed by the service

  • TCP Server Reset – Total TCP resets initiated from the server

SSL metrics

Hover the mouse pointer on a service that uses SSL protocol to view the SSL metrics.

SSL

  • SSL Server Errors – Indicates the total SSL errors from the server. (For example, SSL certificate unknown)

  • SSL Protocol – Indicates the SSL protocol version used by the service

  • SSL Client Errors - Indicate the total SSL errors from the client. (For example, SSL client authentication error)

Apply filters

You can apply filters to view specific service information. Click No Filters list to get the filter options.

Filter options

For example, if you want to view services that have latency less than 150 ms, then click the bar graph under Service Response Time to display the results.

Filter options

Click Service Labels to view services based on the labels provided to services.

Service-labels

Click Clear All to clear all filters.

Clear-all

Alternatively, you can also use the search text box and type a service name to display the results on the service graph.

Search-bar

Using the settings option

Settings-icon

1 – Settings icon

2 – Options to display the service graph as Default, Layer-Based, or Force-Directed views

3 – Select the options from the list to view the services based on categories. After you select a category from the list, click + on the graph to view all services

4 – Enables you select the option on how you want to display the services.

5 - Options to either save the settings or to reset to default.

Services-views

Analyze the errors

Hover the mouse pointer on a service that indicates errors.

Error Description
TCP error The TCP Server Reset indicates the total TCP resets initiated from the server.
TCP client error The TCP Client Reset indicates the total TCP resets initiated by the client.
SSL error The SSL Client Errors indicate the total SSL errors from the client. (For example, SSL client authentication error).
  The SSL Server Errors Indicate the total SSL errors from the server. (For example, SSL certificate unknown)

Note

  • Client error count (irrespective of the protocol type) is displayed in any service if the client error count is 1 or higher.

  • Clients error count displayed for any service indicates that the errors are from the client end.

View HTTP transaction details

According to the example shown in the image, you can view an end-to-end network map of your application that shows how your component services are communicating.

When you hover the mouse pointer on the Ecommerce-Service, you can view metrics details for Ecommerce-Service.

Error-details

Citrix ADM also enables you to view transaction details between Ingress and services. Hover the mouse pointer to view details such as total errors, average service response time, and so on between the Ingress and service.

Ingress-service-details

Hits – Indicates the total number of hits received by the service.

Service Response Time – Indicates the average response time taken from the service to respond for Time To First Byte (TTFB).

Errors – Indicates the total errors such as 4xx, 5xx, and so on.

Data volume – Indicates the total volume of data processed by the service.

Click the arrow between Ingress and service to view the detailed transactions.

View Web transaction logs

The transaction details for the selected service are displayed.

Transaction-summary

You can select the options available under Transaction Summary.

Transaction summary option

  • Browser - Search transactions based on the browsers used by the users.

  • Client OS - Search transactions based on the operating systems installed by the users.

  • Request Type – Search transactions based on the request from the service.

  • Response Code – Search transactions based on the response from the service. For example: 501, 404, 200.

  • Response content type – Search transactions based on the content type. If the client request is for text/html, then the response from the service must be text/html.

  • SSL protocol – Search transactions based on the protocols used by the users.

  • SSL Cipher Strength – Search transactions based on the status such as high, medium, and low.

  • SSL Key Strength – Search transactions based on the length of the key used for security. For example: 2048.

  • SSL Frontend Failure - Search transactions based on the reason for handshake failure.

The Transaction Summary also has a search text box and time duration list, where you can view the transactions as per your requirement. When you click the search box, the search box gives you a list of search suggestions. You can also use operators in your search queries to narrow the focus of your search.

The following are the operators you can use for your search queries:

Operators Description Example Output
= Equals to some value App-Response Time = 500 Displays all transactions with 500 ms response time
> Greater than some value App-Response-Time > 500 Displays all transactions with more than 500 ms response time
< Lesser than some value App-Response-Time < 300 Displays all transactions with less than 300 ms response time
>= Greater or equal to some value Client-RTT >= 1024 Displays all transactions with client RTT greater or equal to 1024 kb
<= Less or equal to some value Client-RTT <= 1024 Displays all transactions with lesser or equal to 1024 kb
!= Not equal to some value Total-Bytes != 0 Displays all transactions with total bytes, except 0 bytes
~ Contains some value Virtual-Server ~ mas Displays all transactions that are processed with virtual server containing mas as name

View transaction details

You can view detailed information about a particular transaction. Consider that you want to see details for 500 error transactions. Click Response Code from Transaction Summary and select 500 to display the 500 error transactions.

5xx-error-transaction

Click to view details that display the information from Ecommerce-Service to Inventory-Service.

From the details, you can analyze the factors that have caused 500 error and take necessary actions to fix the issue faster.