Source Repository

Lens uses git for its code repository. The repository is available at https://git-wip-us.apache.org/repos/asf/incubator-lens.git.

If you are comfortable working in github environment by forking a github repo, sothat you can push changes to your repository before they are accepted in apache, we have a mirror of source at https://github.com/apache/incubator-lens. Its better to add the apache repo as remote than github repo, because github repo might be delayed as it is a mirror of apache repo.

Build tools

• A Java Development Kit. You can use java6 or java7. Build does not pass with java8 yet.
• Apache maven (3.x+)

  Ensure all the tools are installed by executing mvn, git and javac respectively.

  As the Lens builds use the external Maven repository to download artifacts, Maven needs to be set up with the proxy settings needed to make external HTTP requests. The first build of every Lens project needs internet connectivity to download Maven dependencies.

• Be online for that first build, on a good network
• See Maven proxy settings

Integrated Development Environment (IDE)

You are free to use whatever IDE you prefer or your favorite text editor. Note that:

• Building and testing is often done on the command line or at least via the Maven support in the IDEs.
• Set up the IDE to follow the source layout rules of the project.

Building Lens from Source

Download Apache Lens source release from here.

   unzip apache-lens-<version>-source.release.zip
   cd apache-lens-<version>
   mvn clean package -DskipTests

OR Clone Apache Lens source code from https://git-wip-us.apache.org/repos/asf/incubator-lens.git

   git clone https://git-wip-us.apache.org/repos/asf/incubator-lens.git
   cd incubator-lens
   mvn clean package -DskipTests

Once one of the above sets of commands completes successfully, the build will produce lens-dist/target/apache-lens-<version-bin/apache-lens-version-bin/server> and lens-dist/target/apache-lens-<version-bin/apache-lens-version-bin/client>. Former can be used as the Lens server installation directory and later can be used as Lens client installation directory to run lens server and lens client.

The build will also produce debians for both client and server in lens-dist/target. Client debian uses /usr/local/lens/client as the Lens client installation directory and Server debian uses /usr/local/lens/server as the Lens server installation directory.

Apache Lens depends on Hive. Please build Hive from Source or install it using the documentation here. After installing Lens and Hive, refer here for running lens client and lens server from installation directories.

Building Hive from Source

   git clone https://github.com/InMobi/hive.git

   cd hive

   git checkout <release-tag>

   mvn clean package -DskipTests -Phadoop-2,dist

Once above package command completes successfully, packaging/target will have apache-hive-$project.version-bin. This build also produces source, binary tar.gz files and deb package for hive.

Set the environment variable HIVE_HOME to point to the Hive installation directory built from source:

  cd packaging/target/apache-hive-$project.version-bin/apache-hive-$project.version-bin
  export HIVE_HOME=`pwd`

Code compliance

All contributions should satisfy the following requirements

• All public classes and methods should have informative javadoc comments.
  • Do not use @author tags.
• All existing unit tests and integration tests should pass.
• New unit tests should be provided to demonstrate bugs and fixes. Lens uses TestNG test framework. If any code change does not include unit test, the contributor should give the reason why it is not possible to include a unit test.
• Project documentation corresponding to the change should be updated along with the code change. See Documentation section to know how Lens documentation is organized and how to update
• Code must be formatted according to java standards, with the following changes:
  • Trailing White spaces: Remove all trailing white spaces.
  • Indentation: Never use tabs! Always use 2 space indents.
  • Line wrapping: Always use a 120-column line width.
• All working files (java, xml, others) should have the ASF license header in all versioned files.
• If new features requires illustrative examples, they should be added in lens-examples.

Naming convention for configuration properties

Developers should follow the following naming convention for configuration properties

• All server configuration names start with lens.server.
• All configuration overridable for each query start with lens.query.
• All configuration of drivers start with lens.driver. For HiveDriver the names start with lens.driver.hive. and for JDBCDriver it is lens.driver.jdbc
• All configuration overridable for each session start with lens.session.

Generating patch

   git status

  git add src/.../MyNewClass.java
  git add src/.../TestMyNewClass.java

  git diff --no-prefix > LENS-1234.patch

Review request

Install reviewboard command line tool:

• Make sure you have python installed and easy_install is available.
• $ easy_install -U setuptools
• More help available at Reviewboard Documentation.

  rbt post --parent=master

  git diff master..HEAD > ~/LENS-##.patch

  rbt post -u --parent=master

  rbt close --close-type={submitted/discarded} #####(request id)

Project documentation

The design, architecture and feature documentation needs to be updated in src/site of parent module. The documentation is organized into four menus under the site.

Configuration documentation

Configuration pages should be linked from user guide and admin guide with respect to the configuration exposed to them. See Developer FAQ on how to update config docs.

REST api documentation

REST api documentation is auto generated through Enunciate. Once the javadoc for the resource api is updated correctly, the REST api doc should get updated.

If a new resource is added in lens-server module, it should be updated in lens-server/enunciate.xml sothat the REST api doc gets generated. If a new module is added with resources and the module pom entry need to updated with enunciate plugin usage and tools/scripts/generate-site-public.sh needs to be updated with site generation and publishing the docs.

Feature documentation

TODO

Design and architecture documentation

TODO

Confluence usage

Cwiki should be used for documentation that cannot go into code, which is some adhoc documentation. This falls into following main categories

• Design doc for a feature, which is just proposed or it is under implementation. Once the feature is implemented doc should be updated in project documentation
• Project roadmap
• Discussions : Placeholder for discussions that cannot be done in jira
• Presentations : Links to slideshare or google docs
• Events : Developer/User Meetup minutes

  Cwiki should not be used for documentation that is already present in code, which is the following

• Architecture documentation
• Feature documentation
• How to contribute/commit/release pages
• Generated doc for REST API/Javadoc
• Getting started pages

How to update documentation?

The new doc files can added in src/site of parent module. Or doc change can be done in existing files under src/site. Update the config files and run the TestGenerateConfigDoc test calss for updating the config docs. Once the changes are done, contributor can run mvn site:run, which will start localhost doc server on port 8080, which can opened through browser and doc can be validated.

How to update the config docs?

Add/Delete/Modify the config to the resource config files(Ex: lens-client-default.xml file for Client resource). Run the TestGenerateConfigDoc, which automatically updates the config apt files under src/site.

How to add license headers for newly added files?

Run the command mvn license:format. This is add license headers for all the files automatically. If some files need to excluded they should be put in excludes section in parent project pom's license-maven-plugin.

How to check all licenses are fine?

Run the command mvn apache-rat:check. If check needs to be excluded for any file, it should be put in excludes section in parent project pom's apache-rat-plugin.

What is versioning strategy in Lens?

Lens follows three number versioning which is major.minor.revision. If the current release is 2.0.0, the next usual development release will be 2.1.0. If there needs to be separate release on released version and not from development branch (usually critical patch release), it will be 2.0.1. If the next release is not compatible with previous release, then the major version needs to be incremented, then it would become 3.0.0. This way all 2.x.x releases will be compatible with one another. And incompatibility is clearly communicated to users through major version number change.

The jira fix version for all the issues in 2.0.x is called 2.0, and 2.1.x is called 2.1. For patch releases from release branch, the jira fix version can be exact patch release version number.

What is the branching strategy in Lens?

Lens has two main branches - master and current-release-line. All the day to day development happens on master branch. current-release-line branch is used to make releases. When master branch is ready for release (all improvements/features/bug fixes marked for release are fixed and all tests passing), master will be merged to current-release-line. The version number on master will be incremented to next development version. The only issues that can be cherry-picked into current-release-line till release is rolled are critical/blocker bugs, documentation changes and test case changes. Once all the issues are verified for the release, and release will be triggered from current-release-line.

If a critical release (not pulling code from master) needs to be made, a new branch will be created with release number, by checking out current-release-line branch. And changes will be put on the branch. Once the branch is ready they will merged to current-release-line and released. The changes should be cherry-picked back into master from current-release-line once the release is made and resolving conflicts in master if any. Having two main branches makes all release tags to be created on current-release-line branch and removes the pile up of old and stale branches, which are created by one for each release.

For major version increments, current-release-line will be branched to a a $major.x-line and current-release-line and master will be moved to next major version.

There can be feature branches created from master, if feature is not actively developed in master branch directly. For a feature branch to be created a contirbutor can start discuss thread on dev list for consensus on whether it is required.