The default behavior when using data connectors is to retrieve data into memory for visual analysis to then occur, where the data is aggregated and filtered in memory. This retrieval may be the consumption of a whole dataset, or through the use of parameters, the retrieval of a dynamically selected subset of the data. This approach is however limited by the memory of the machine, and the overhead of retrieving and processing large volumes of data on the desktop.
On-demand queries provide ROLAP functionality to the Panopticon products, where the aggregation and filtering tasks are largely offloaded to the underlying data repository.
The software will dynamically generate q query for:
q Filter domains (Categorical Listing & Min/Max for Numeric Fields)
q Aggregated & Filtered Data returned in the visualizations
Each filter and visualization are driven by a separately generated q query, ensuing that each query is simplified, and returns the minimum amount of data.
This on-demand capability dramatically reduces the amount of data to be transferred across the network and onto the application and ensures that the heavy data intensive tasks occur in Kx kdb+ instances. However, when using this mode, the following functionality is disabled:
q Percentile Filtering
q Copy Raw Data
q Pivot & Unpivot Data Transforms
q Non-Additive Data support
q Shared selection across visualizations
q Numeric Bucketing
q Date/Time Part Specific Options (Decade, Quarter, Weekday, Millisecond, Nanosecond)
q R Transform
q Python Transform