Product Documentation
ADC
14.1 - Current Release 13.1 13.0 12.1
View PDF

Export transaction logs directly from NetScaler to Elasticsearch

April 8, 2024
Contributed by: 
C

You can now export transaction logs from NetScaler to industry-standard log aggregator platforms such as Elasticsearch. The transaction log is the record of application traffic flow events on the NetScaler such as HTTP requests and responses, connection start and end. For more information on transaction logs, see AppFlow.

You can export transaction logs in JSON format for various insights such as Web Insight, security, gateway, HDX insights. To export transaction logs to Elasticsearch, you must configure Elasticsearch as an HTTP server and use the HTTP event collector to send transaction logs over HTTP (or HTTPS) directly to the Elasticsearch platform from your NetScaler. Using the visualization tools at Kibana, you can get meaningful insights about the exported data.

Note:

The IP addresses that are exported as part of the transaction logs appear in the decimal format instead of the standard format. For example, if your NetScaler IP address is 10.102.154.153, the same in the transaction logs on Elasticsearch is displayed as 174496409. You can use the inbuilt expressions available on Elasticsearch to convert the IP address from decimal format to standard format.

Export transaction logs from NetScaler to Elasticsearch configured as an HTTP server

To configure the export of transaction logs you must perform the following steps:

  1. Configure Elasticsearch to receive transaction logs.
  2. Create a collector service and an analytics profile on NetScaler.

Configure Elasticsearch to receive transaction logs

You can configure Elasticsearch to receive the transaction logs forwarded by NetScaler by following the configuration steps avaialble in Elasticsearch documentation.

Once you have configured, copy the authentication token and save it for reference. You need to specify this token while configuring the analytics profile on NetScaler.

Configure analytics profile on NetScaler

Do the following to export NetScaler transaction logs to Elasticsearch.

  1. Create a collector service for Elasticsearch.

    add service <collector> <elasticsearch-server-ip-address> <protocol> <port>

    Example:

    add service elasticsearch_service 10.102.34.155 HTTP 8088

    In this configuration:

    • ip-address: Elasticsearch server IP address.
    • collector-name: Name of the collector.
    • protocol: Specify the protocol as HTTP or SSL.
    • port: Port number.

  2. Create an analytics profile.

    add analytics profile `profile-name` -type <insight> -collectors `collector-name` -analyticsAuthToken `<auth-scheme> <authorization-parameters>` `-analyticsEndpointContentType` `application/json` -analyticsEndpointUrl `endpoint-url`

    Example:

    add analytics profile transaction-log-profile -type webinsight -collectors elasticsearch_collector -analyticsAuthToken "ApiKey 1234-5678-12345" -analyticsEndpointContentType "application/json" -analyticsEndpointUrl "/_bulk"

    Note:

    The -allHttpHeaders option is not supported for Elasticsearch transaction log export.

    add analytics profile <profile-name> -type webinsight -allHttpHeaders

    set analytics profile <profile-name> -type webinsight -allHttpHeaders

    In this configuration:

    • insight: Types of insights that you can export The following options are available:
      • botinsight
      • CIinsight
      • Gatewayinsight
      • hdxinsight
      • lsninsight
      • securityinsight
      • tcpinsight
      • udpinsight
      • videoinsight
      • webinsight

    • -analyticsAuthToke auth-scheme authorization-parameters: Specify the authentication token to be included in the authorization header with the auth-scheme “ApiKey” while sending logs to Elasticsearch. This token is the authentication token created on the Elasticsearch server while configuring the HTTP event collector.

    • analyticsEndpointContentType: The Content-Type header. The hardcoded value is application/json.

    • analyticsEndpointUrl: The path on Elasticsearch to which transactions need to be posted. For example, /_bulk.

      NOTE:

      You can modify the analytics profile parameters using the set analytics profile command.

  3. Verify the analytics profile configuration using the show analytics profile command.

    > sh analytics profile

  4. Bind the analytics profile to the vserver.

    bind lb vserver <vserver-name> -analyticsProfile transaction-log-profile

    Example

    bind lb vserver sample-log -analyticsProfile transaction-log-profile

After the configuration is completed, based on traffic, transactions will be logged and exported to Elasticsearch.

Field-based filtering of data records

By default, NetScaler exports hundreds of fields in transaction log even when the endpoints do not require all of the exported data. Also, each endpoint expects the JSON payload to be encoded in a specific format such as the start and end of a data record, the delimiter between data records, and buffer start and end.

Elasticsearch expects the JSON payload from coming from NetScaler to be encoded in the following format:

  • Buffer start and end: No value required for BUFFER-START and BUFFER-END. At the end of the buffer, you must add a blank line.

  • Data record start and end: The data record must start with {"index": {"_index": "transactions"}} { and end with }. All the fields that get exported must be captured between DATA-START and DATA-END.

    The data records must start with the following:

     RECORD-START
 {"index": {"_index": "transactions"}}
 {
 DATA-START

    The data records must end with the following:

     DATA-START
 }
 RECORD-END
  • Delimiter between data records: One blank line between the data records.

By default, elastic_format.txt is available at var/analytics_conf folder that contains the the JSON payload format that Elasticsearch expects and also contains a few default fields for which the data gets exported. You can configure elastic_format.txt as the value for -dataFormatFile parameter in the add analytics profile command.

The following is a sample data format file for Elasticsearch:

    BUFFER-START
    RECORD-START
    {"index": {"_index": "transactions"}}
    {
    DATA-START
    153 observationPointId
    547 nsPartitionId
    154 exportingProcessId
    159 transactionId
    801 httpReqUrl
    685 httpReqMethod
    683 httpReqHost
    689 httpReqUserAgent
    680 httpContentType
    691 httpReqXForwardedFor
    682 httpDomainName
    803 appName
    851 appNameVserverLs
    484 httpRspStatus
    53 httpRspLen
    684 httpResLocation
    687 httpResSetCookie
    DATA-END
    }
    RECORD-END
    RECORD-DELIMITER


    RECORD-DELIMITER-END


    BUFFER-END

The JSON_fields.txt file under var/analytics_conf is a reference master file that contains the complete list of fields along with their identification numbers. The fields in the master file are categorized based on the insights. For example, if you want to know the fields associated with HDX insight, you can look at the HDX insights category of JSON_fields.txt file to know the fields associated with HDX insights category.

You can customize the elastic_format.txt file based on your requirement. For example, if you want to export HDX insights, you can look at the HDX insights category in JSON_fields.txt file and add the required fields in the the elastic_format.txt file. Similarly, you can delete fields that you do not wish to export.

After customizing the elastic_format.txt file, execute the following command to update the analytics profile:

update analytics profile <profile-name> -dataFormatFile <data-format-file-name>

Example:

update analytics profile ns _analytics_ default_http_profile -dataFormatFile elastic_format.txt

You can also specifiy the value of data format file using the GUI. Navigate to System > AppFlow > Analytics Profiles and click Add. On the Create Analytics Profile page, if you select one of the following options for Type, then the Data Format File field appears where you can specificy the file name:

  • GLOBAL
  • WEB INSIGHT
  • TCP INSIGHT
  • SECURITY INSIGHT
  • VIDEO INSIGHT
  • HDX INSIGHT
  • GATEWAY INSIGHT
  • LSN INSIGHT
  • BOT INSIGHT
  • TIME SERIES

Sample outputs

This section contains sample outputs for different transaction logs.

HTTP transaction log sample output

The following is a sample output for the HTTP transaction log.

{
    appName: VS1
    clientMss: 1460
    clntFastRetxCount: 0
    clntTcpJitter: 0
    cintTcpPacketsRetransmited: 0
    clntTcpRtoCount: 0
    clntTcpZeroWindowCount: 0
    cltDstIpv4Address: 174496411
    cltIpv4Address: 174496407
    connEndTimestamp: 0
    connStartTimestamp: 7329468222993076980
    exportingProcessId: 0
    httpRegHost: 10.102.154.155
    httpReqMethod: GET
    httpReqUrl: /big.html
    httpRspLen: 114380
    httpRspStatus: 200
    mainPageCoreId: 0
    mainPageId: 0
    nsPartitionId: 0
    observationPointId: 174496409
    originRspLen: 0
    srvrIcpPacketsRetransmited: 0
    srvrTcpZeroWindowCount: 0
    svrDstIpv4Address: 174496415
    svrIpv4Address: 174496408
    tepSrvrConnRstCode: 0
    transClntRTT: 0
    transCltDstPort: 20480
    transCltFlowEndUsecRx: 7329468222993084980
    transCltFlowEndUsecTx: 7329468222993084980
    transCltFlowStartUsecRx: 7329468222993076980
    transCltFlowStartUsecTx: 7329468222993077984
    transCltSrcPort: 60315
    transCltTotRx0ctCnt: 1766
    transCltTotTx0ctCnt: 117580
    transSrvDstPort: 36895
    transSrvSrcPort: 15213
    transSrvrRTT: 0
    transSvrFlowEndUsecRx: 7329468222993084980
    transSvrFlowEndUsecTx: 7329468222993084980
    transSvrFlowStartUsecRx: 7329468222993077984
    transSvrFlowStartUsecTx: 0
    transSvrTotRx0ctCnt: 117580
    transSvrTotTx0ctCnt: 1766
    transactionId: 4890
}

TCP transaction log sample output

The following is a sample output for TCP transaction log.

{
    appName: vs1
    clientConnEndTimestamp: 7333165210582386064
    clientConnStartTimestamp: 7333165210582386054
    clientMss: 1460
    clntFastRetxCount: 0
    clntTcpJitter: 0
    clntTcpPacketsRetransmited: 0
    clntTcpRtoCount: 0
    clntTcpZeroWindowCount: 0
    cltDstIpv4Address: 174496411
    cltDstPort: 20480
    cltIpv4Address: 174496407
    cltSrcPort: 42939
    connectionChainHopCount: 0
    exportingProcessId: 0
    nsPartitionId: 0
    observationPointId: 174496409
    serverConnEndTimestamp: 7333165201992708470
    serverConnStartTimestamp: 7333165201992708459
    srvDstPort: 36895
    srvSrcPort: 51973
    srvrTcpPacketsRetransmited: 0
    srvrTcpZeroWindowCount: 0
    svrDstIpv4Address: 174496415
    svrIpv4Address: 174496408
    tcpClntConnRstCode: 0
    tcpSrvrConnRstCode: 0
    transClntRTT: 0
    transCltTotRxOctCnt: 208
    transCltTotTxOctCnt: 331
    transSrvrRTT: 0
    transSvrTotRxOctCnt: 331
    transSvrTotTxOctCnt: 208
    transactionId: 330
    vlanNumber: 1
}

SSL transaction log sample output

The following is a sample output for SSL transaction log.

{
    appName: sslvs
    clientConnEndTimestamp: 0
    clientConnStartTimestamp: 7333182669624439854
    clientMss: 1460
    clntFastRetxCount: 0
    clntTcpJitter: 0
    clntTcpPacketsRetransmited: 0
    clntTcpRtoCount: 0
    clntTcpZeroWindowCount: 0
    cltDstIpv4Address: 174496411
    cltDstPort: 47873
    cltIpv4Address: 174496407
    cltSrcPort: 17499
    connectionChainHopCount: 0
    exportingProcessId: 0
    httpContentType: text/html
    httpReqHost: 10.102.154.155
    httpReqMethod: GET
    httpReqUrl: /index.html
    httpReqUserAgent: curl/7.69.1
    httpRspLen: 291
    httpRspStatus: 200
    nsPartitionId: 0
    observationPointId: 174496409
    originRspLen: 0
    serverConnEndTimestamp: 0
    serverConnStartTimestamp: 7333182665330184556
    srvDstPort: 36895
    srvSrcPort: 34802
    srvrTcpPacketsRetransmited: 0
    srvrTcpZeroWindowCount: 0
    sslCipherValueBE: 0
    sslCipherValueFE: 50331701
    sslClientCertSizeBE: 0
    sslClientCertSizeFE: 0
    sslClntCertSigHashBE: 0
    sslClntCertSigHashFE: 0
    sslFLagsBE: 0
    sslFLagsFE: 1096
    sslServerCertSizeBE: 0
    sslServerCertSizeFE: 4096
    sslSessionIDBE: 0
    sslSessionIDFE: 2433458443
    sslSigHashAlgBE: 0
    sslSigHashAlgFE: 0
    sslSrvrCertSigHashBE: 0
    sslSrvrCertSigHashFE: 668
    svrDstIpv4Address: 174496415
    svrIpv4Address: 174496408
    tcpClntConnRstCode: 0
    tcpSrvrConnRstCode: 0
    transClntRTT: 0
    transCltFlowEndUsecRx: 7333182669624447854
    transCltFlowEndUsecTx: 7333182669624446854
    transCltFlowStartUsecRx: 7333182669624439854
    transCltFlowStartUsecTx: 7333182669624439854
    transCltTotRxOctCnt: 1501
    transCltTotTxOctCnt: 2223
    transSrvrRTT: 0
    transSvrFlowEndUsecRx: 7333182669624446854
    transSvrFlowEndUsecTx: 7333182669624446854
    transSvrFlowStartUsecRx: 7333182669624446854
    transSvrFlowStartUsecTx: 0
    transSvrTotRxOctCnt: 331
    transSvrTotTxOctCnt: 168
    transactionId: 2640
    vlanNumber: 1
}

Web Insight transaction log sample output

The following is a sample output for Web Insight transaction log.

{
    appName: vs1
    clientConnEndTimestamp: 0
    clientConnStartTimestamp: 7333336201820249485
    clientMss: 1460
    clntFastRetxCount: 0
    clntTcpJitter: 0
    clntTcpPacketsRetransmited: 0
    clntTcpRtoCount: 0
    clntTcpZeroWindowCount: 0
    cltDstIpv4Address: 174496411
    cltDstPort: 20480
    cltIpv4Address: 174758625
    cltSrcPort: 46824
    connectionChainHopCount: 0
    exportingProcessId: 0
    httpContentType: text/html
    httpReqHost: 10.102.154.155
    httpReqMethod: GET
    httpReqUrl: /
    httpRspLen: 291
    httpRspStatus: 200
    nsPartitionId: 0
    observationPointId: 174496409
    originRspLen: 0
    serverConnEndTimestamp: 0
    serverConnStartTimestamp: 7333336201820250487
    srvDstPort: 36895
    srvSrcPort: 6465
    srvrTcpPacketsRetransmited: 0
    srvrTcpZeroWindowCount: 0
    svrDstIpv4Address: 174496415
    svrIpv4Address: 174496408
    tcpClntConnRstCode: 0
    tcpSrvrConnRstCode: 0
    transClntRTT: 0
    transCltFlowEndUsecRx: 7333336201820251488
    transCltFlowEndUsecTx: 7333336201820251488
    transCltFlowStartUsecRx: 7333336201820249485
    transCltFlowStartUsecTx: 7333336201820250487
    transCltTotRxOctCnt: 190
    transCltTotTxOctCnt: 371
    transSrvrRTT: 0
    transSvrFlowEndUsecRx: 7333336201820251488
    transSvrFlowEndUsecTx: 7333336201820250487
    transSvrFlowStartUsecRx: 7333336201820250487
    transSvrFlowStartUsecTx: 7333336201820250487
    transSvrTotRxOctCnt: 371
    transSvrTotTxOctCnt: 202
    transactionId: 11218
    vlanNumber: 1
}
Export transaction logs directly from NetScaler to Elasticsearch
April 8, 2024
Contributed by: 
C
April 8, 2024
Contributed by: 
C

In this article

Site feedback | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.