Query results

This document contains details on query result - fetching, formatting and etc.

Lens query service allows users to fetch results of query by specifying the handle-id. It provides rest api to fetch result-metadata and the results.

Here are various options provided for result formatting and fetching:

Persisting results by server

Server can persists the results in configured location, also format the results of the query, with a custom format.

User can provide a custom output formatter which implements InMemoryOutputFormatter or PersistedOutputFormatter

Default implementation of formatting offers writing it into a file, but a new implementation can be provided which persist in a db or whatever.

The file based formatting provides following options:

  • Optionally write header
  • Format the rows of the results like csv, or separate them with ctrl character characters, which can be described through a Hive SerDe.
  • Optionally write the output with a charset encoding.
  • Optionally write footer
  • Provide an extension to the file name
  • Optionally compress the file
  • Skip formatting if result size exceeds a configured threshold
  • Optionally allow result to be split into multiple files, with configured number of rows in ech file.

Persisting results by driver

The driver can persist the query output in a location. When driver persists the results, the rows of the result can be formatted by the driver.

HiveDriver allows users to specify the format of the result through a configuration. It will persist the result in queryhandle-id directory inside hdfsout directory in the parent result directory.

Since persisting the results by driver only allows formatting of the rows, if user requires further formatting, the results should be persisted by server again.

Formatting summary

Below table depicts all the formatting options provided by query service

Formatting of the results Not persisted by driver Persisted by driver
Not persisted by server No formatting Row level formatting
Persisted by server All query output formatting All query output formatting on top of row level formatting done by driver

See here for all the configuration property names

Fetching the results

If the results are persisted either by server or driver, the user would get the result set as PersistentQueryResult. If results are not persisted, then user would get the result as InMemoryQueryResult.

ResultSet Not persisted by driver Persisted by driver
Not persisted by server InMemoryQueryResult PersistentQueryResult
Persisted by server PersistentQueryResult PersistentQueryResult

If results are persisted by server, user can get the result as a http end point, which he can download or see it in the browser. The http result set can be obtained by the rest endpoint /queryapi/queries/queryhandle/httpresultresult

The results can be fetched as PersistentQueryResult or InMemoryQueryResult from the end point /queryapi/queries/queryhandle/resultresult