Product documentation
uberAgent
Current Release (7.3.0) 7.2.1 7.2.0 7.1.2 7.1.1 7.1.0 7.0.2 7.0.1 7.0.0
View PDF

Persistent Output Queue With Intelligent Disk Buffering

July 19, 2024
Contributed by: 
C

Persistent Output Queue with Intelligent Disk Buffering is uberAgent’s sophisticated technology that ensures no outgoing events are lost while maintaining a minimal footprint and generating no disk IO at all during normal (online) operations.

How Does the Persistent Output Queue Work

Case A: Backend is Reachable

uberAgent constantly monitors network connectivity to its backend. As long as the backend is reachable and processes new events correctly, the endpoint agent does not store events on disk but sends them to the backend directly, thus avoiding any disk IO on the endpoint.

Case B: Backend is Unreachable

If the backend becomes unreachable, either because network connectivity is lost or because the backend has issues accepting data, the agent writes new events to the Persistent Output Queue’s local database. uberAgent periodically checks if connectivity is back up. Once that is the case, it sends all events that were buffered to disk and reverts to the efficient direct sending (see case A).

Persistent Output Queue and Splunk Universal Forwarder

On mobile devices, uberAgent was traditionally coupled with Splunk’s Universal Forwarder (UF) due to the UF’s persistent queue functionality. With uberAgent’s built-in POQ, it’s not necessary anymore to deploy the UF just for its disk buffering feature.

Disk Space Requirements

The maximum disk space to be used by the POQ can be limited in uberAgent’s configuration (see below). The following table is intended as a rough indicator of disk space requirements (PC with average usage, 10 hours uptime, default uberAgent configuration):

Backend Disk space per day
Elastic 200 MB
Kafka 410 MB
Microsoft OMS 220 MB
Splunk via HTTP 120 MB
Splunk via TCP 70 MB

Configuration Settings

Persistent Output Queue is configured per receiver. Please note that receivers with a POQ must have a unique name.

Enabling or Disabling the POQ

PersistentOutputQueue

Enables or disables the Persistent Output Queue for the current receiver.

If the Persistent Output Queue is disabled, uberAgent reverts to simple memory buffering (see the configuration setting MaxQueueSizeRamMb).

  • Setting name: PersistentOutputQueue
  • Valid values: Off, SQLite
  • Default: SQLite
  • Required: no

Limiting POQ Disk Space Usage

PersistentOutputQueueMaxAgeDays

Configures the maximum retention time for POQ entries. Entries older than the specified number of days are removed.

  • Setting name: PersistentOutputQueueMaxAgeDays
  • Valid values: positive numbers (0 = unlimited)
  • Default: 120
  • Required: no

PersistentOutputQueueMaxSizeMb

Configures the maximum disk space in MB for the Persistent Output Queue. The file size is checked after events are stored. It is therefore possible that the configured limit is slightly exceeded.

  • Setting name: PersistentOutputQueueMaxSizeMb
  • Valid values: positive numbers (0 = unlimited)
  • Default: 500
  • Required: no

POQ events are stored in the format expected by the receiver’s backend. Some backends expect the data in a JSON format, while others accept raw data. Especially the JSON format bloats events considerably, which means that the required disk space can be much larger than the raw event data as indexed by Splunk.

POQ Location on Disk

PersistentOutputQueuePathWindows

Specifies the POQ storage location on Windows machines. Environment variables are supported.

  • Setting name: PersistentOutputQueuePathWindows
  • Valid values: any valid path
  • Default: %PROGRAMDATA%\vast limits\uberAgent\Output Queue
  • Required: no

When the default path is used, uberAgent sets the permissions on the directory so that only members of the local group Administrators and the local system have read/write access. When a custom path is specified, uberAgent does not set permissions on the POQ directory. We highly recommend securing the directory.

PersistentOutputQueuePathMacOS

The setting PersistentOutputQueuePathMacOS specifies the POQ storage location on Windows machines. Environment variables are not supported.

  • Setting name: PersistentOutputQueuePathMacOS
  • Valid values: any valid path
  • Default: /Library/Application Support/uberAgent/Output Queue
  • Required: no

When the default path is used, uberAgent sets the permissions on the directory so that only the root user has read/write access. When a custom path is specified, uberAgent does not set permissions on the POQ directory. We highly recommend securing the directory.

Miscellaneous Settings

PersistentOutputQueueErrorHandling

Configures how errors during POQ event processing are handled.

  • Setting name: PersistentOutputQueueErrorHandling
  • Valid values: IgnoreFile, StopReceiver, Shutdown
  • Default: IgnoreFile
  • Required: no

In case of an error (e.g., out of disk space), uberAgent may either stop the current receiver (StopReceiver), shut down the agent entirely (Shutdown), or ignore the current POQ file (IgnoreFile). In case all receivers have stopped, uberAgent shuts itself down.

Advanced Settings

PersistentOutputQueueNextSendAttemptSeconds

Configures when to perform a next send attempt in case the backend is not reachable.

  • Setting name: PersistentOutputQueueNextSendAttemptSeconds
  • Valid values: positive numbers (0 = no wait delay)
  • Default: 25
  • Required: no

PersistentOutputQueueMaxInMemoryBulkOperations

Configures how many bulk send operations are kept in memory before storing them in the Persistent Output Queue. A single bulk send operation consists of all events within a run that are generated for an uberAgent metric (e.g., process detail). A value of 0 (zero) disables the in-memory cache. In this case, all events are stored in the Persistent Output Queue before sending them to the backend. This provides maximum preservation of data at the expense of I/O.

  • Setting name: PersistentOutputQueueMaxInMemoryBulkOperations
  • Valid values: positive numbers (0 = disables in-memory cache)
  • Default: 10
  • Required: no

Advanced Topics

Data Collection Without Network Connectivity

Receivers with a POQ can be used without a configured backed. In other words: the Servers setting can be left empty. This is useful to collect data on a machine without network connectivity, such as in a DMZ. The collected data can then be transferred to a machine with network connectivity simply by copying the POQ files.

Recommendations

DMZ machine:

  • Configure the receiver’s settings Name, Type, and Protocol to match the settings on the machine you later plan to transfer the POQ files to.
  • Once sufficient data has been collected, stop uberAgent and copy the POQ files off the machine.

Machine with network connectivity:

  • Stop uberAgent.
  • Copy the POQ files from the DMZ machine to the local POQ path.
  • Start uberAgent. The agent sends the DMZ machine’s data off to its backend.

POQ File Data Format

The internal format of the Persistent Output Queue is an SQLite database that is not protected and can be read by SQLite browsers like DB Browser for SQLite.

Persistent Output Queue With Intelligent Disk Buffering
July 19, 2024
Contributed by: 
C
July 19, 2024
Contributed by: 
C

In this article

Site feedback | Your privacy choices footer linkYour Privacy Choices | Privacy and legal terms | Cookie preferences | Consent settings | docs.cloud.com
© 1999- Cloud Software Group, Inc. All rights reserved.