uberAgent

No Data in Splunk Even Though uberAgent Sends Successfully

Scenario 1: uberAgent is sending data directly to Splunk

Symptoms

According to uberAgent’s log file, there are no events in its in-memory send queue, indicating a successful data transfer to the Splunk server(s): ReceiverStatistics,Splunk; SPLUNKSERVER:19500 - Events in queue: 0, queue size: 0.0 KB, sent: 1597, added to queue: 1597, rejected from queue: 0 When searching in Splunk’s internal logs for errors using the following search:index=_internal error you see messages like the following: ERROR TcpInputProc - Message rejected. Received unexpected 707406419 byte message! from src=IPADDRESS:PORT. Maximum message allowed: 67108864

Cause

The port uberAgent sends data to (default: 19500) is configured to receive data from Universal Forwarders.

Splunk Universal Forwarders do not send raw event data, they use a specific protocol. uberAgent, on the other hand, sends raw event data. If the formats sent by the source and expected on the target do not match the above error message may be logged and the incoming data is ignored on the Splunk server.

Resolution

If uberAgent is sending data directly to Splunk do not open port 19500 via Forwarding and receiving on the Splunk server. Install the uberAgent_indexer app instead which opens port 19500 as a raw TCP port.

Scenario 2: uberAgent is sending data to a locally installed Splunk Universal Forwarder

Symptoms

According to uberAgent’s log file, there are no events in its in-memory send queue, indicating a successful data transfer to the locally installed Universal Forwarder: ReceiverStatistics,Splunk; localhost:19500 - Events in queue: 0, queue size: 0.0 KB, sent: 1597, added to queue: 1597, rejected from queue: 0 When searching for incoming data on port 19500 on the Splunk server using the following search: index=* source="tcp:19500" you see messages like the following (shortened):

--splunk-cooked-mode-v3--\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...
<!--NeedCopy-->

Cause

The port Universal Forwarder sends data to (default: 9997) is configured to receive data from uberAgent.

Splunk Universal Forwarders do not send raw event data, they use a specific protocol. uberAgent, on the other hand, sends raw event data. If the formats sent by the source and expected on the target do not match above messages may be seen and the dashboards will be empty.

Resolution

If uberAgent is sending data to Splunk via a locally installed Universal Forwarder open TCP port 19500 on the Universal Forwarder (the same machine uberAgent runs on) and configure uberAgent to send to localhost:19500 as described here.

To enable Splunk to receive the Universal Forwarder’s data open a receiving Port (default: 9997) on Splunk via Forwarding and receiving as described here. Do not open port 9997 as a raw TCP port.

No Data in Splunk Even Though uberAgent Sends Successfully