Installing and running lens

Building Lens from source

Here are the commands to build from source.

   git clone https://git-wip-us.apache.org/repos/asf/incubator-lens.git

   cd incubator-lens

   git checkout master

   mvn clean package

Once above package command completes successfully, lens-dist/target will have lens-dist-$project.version-server and lens-client-dist/target will have lens-client-dist-$project.version-client packaged directories for client and server. The build will also produce debian for both client and server in lens-client-dist/target and lens-dist/target, respectively.

Installing and running lens-server

Users can cd into the above server and client distribution directories built from source and try it out. Or install the debs from an apt repo or the debians built from source directly. Server and client debians will be installed in /usr/local/lens and /usr/local/lens-client respectively.

Let us go over how the install looks, in the subsections below.

Users can also use the docker container to mount the src base directory into the container.

Server install and configuration

The directory structure of the server installation :

  ├── bin
  │   ├── lens-config.sh
  │   └── lens-ctl
  ├── conf
  │   ├── lens-env.sh
  │   ├── lens-site.xml
  │   ├── hive-site.xml
  │   └── log4j.properties
  ├── lib
  │   ├── ST4-4.0.4.jar
  │   ├── activation-1.1.jar
  │   ├── ant-1.9.1.jar
  │   ├── ant-launcher-1.9.1.jar
  │   ├── antlr-2.7.7.jar
  …….
  │   └── zookeeper-3.4.5.jar
  └── webapp
      └── lens-server.war

The bin folder provides lens-ctl which provides commands for starting and stopping the lens server. Other .sh files in bin are the scripts used by lens-ctl.

The server war is available in webapp/ folder, which would be expanded in the same directory, if not overridden. And the lib folder contains all the dependent libraries.

Server configuration can be overridden in lens-site.xml. See server configuration for all the configuration parameters available and their default values. You can set up the environment variables such as JAVA_HOME, java options, heap size and etc in lens-env.sh.

The server logs go into logs/ folder in the install directory, unless overridden. The proper logging configuration can be provided in log4j.properties file.

Lens server dependes on hive which is forked from Apache Hive into InMobi Hive. See here for all the latest releases.

Build it from source and set HIVE_HOME.

   git clone https://github.com/inmobi/hive.git

   cd hive

   git checkout <release-tag>

   mvn clean package -DskipTests -Phadoop-2,dist

Once above package command completes successfully, packaging/target will have apache-hive-$project.version-bin. This build also produces source, binary tar.gz files and deb package for hive.

Ex: export HIVE_HOME=${hive-source-location}/packaging/target/apache-hive-$project.version-bin/apache-hive-$project.version-bin/

Driver configuration

If server has HiveDriver as one of the drivers, the configuration for hive driver should be set up in hive-site.xml. All the available configuration of the HiveDriver can be found here.

Running the server

Before starting the server, you need to set HADOOP_HOME and HIVE_HOME environment variables. These should point to the Hadoop and Hive installation locations on the server.

Once env variables are set, you can use the following command to start the server:

  bin/lens-ctl start

Once the server is up, you can go to the browser and open the server baseurl. You would see a message saying ‘Lens server is up’.

To check all other services are up and running, try baseurl/session, baseurl/queryapi, baseurl/metastore and all of them will respond with respective service is up.

You can try out examples described below or use the REST api link for each service.

The command to stop the server:

  bin/lens-ctl stop

Client install and configuration

The directory structure of the client installation :

  ├── bin
  │   ├── lens-cli.sh
  │   ├── lens-config.sh
  │   └── run-examples.sh
  ├── conf
  │   ├── lens-client-site.xml
  │   ├── lens-env.sh
  │   └── log4j.properties
  ├── examples
  │   ├── data
  │   │   ├── dim1_local
  │   │   │   └── dim1-local.data
  │   │   └── dim2_local
  │   │       └── dim2-local.data
  │   ├── queries
  │   │   └── dimension-queries.txt
  │   └── resources
  │       ├── db-storage.xml
  │       ├── dim1-local-part.xml
  │       ├── dim1-storage-tables.xml
  │       ├── dim2-local-part.xml
  │       ├── dim2-storage-tables.xml
  │       ├── dim_table.xml
  │       ├── dim_table2.xml
  │       ├── fact1-storage-tables.xml
  │       ├── fact1.xml
  │       ├── fact2-storage-tables.xml
  │       ├── fact2.xml
  │       ├── local-cluster-storage.xml
  │       ├── local-storage.xml
  │       ├── rawfact-storage-tables.xml
  │       ├── rawfact.xml
  │       └── sample-cube.xml
  └── lib
      ├── ST4-4.0.4.jar
       …….
      └── xmlenc-0.52.jar

The bin folder provides various scripts for the client.

  • run-examples.sh provides commands to run examples - create a sample metastore, populate metastore with sample data and run some example queries. This would validate your server is all up and running fine.
  • lens-cli.sh provides CLI shell for running various commands. Its usage is more described in CLI user guide.
  • lens-config.sh is used internally by other scripts

Client configuration can be overridden in lens-client-site.xml. See client configuration for all the configuration parameters available and their default values. You can set up the environment variables such as JAVA_HOME, java options, heap size and etc in lens-env.sh.

The client logs go into logs/ folder in the install directory, unless overridden. The proper logging configuration can be provided in log4j.properties file.

Examples folder contains sample resource files, data files and query files which will be used by run-examples.sh script.

Running Lens in Docker

Lens docker files are located in 

  incubator-lens/lens-docker/lens-test

The above directory contains the lens docker images that can be used for development and testing.

To build the image

  cd tools/scripts
  ./build-docker.sh

Before running the build docker must be installed in the system. The above script will build the lens docker image and download the dependent images of hive and hadoop.

To run the image

  cd tools/scripts
  ./run-docker.sh

This will start the docker container and the container will have its Lens directories mounted to your actual checked out code, allowing you to modify and recompile your Lens source and have them immediately usable in the docker images (without rebuilding the image).