Ref Guide Publication Process

The PDF version of the Guide is the official release, and requires a vote before release.

The HTML version is considered a "convenience" version, and does not require a vote.

It’s strongly preferred that both PDF and HTML versions are available during the vote for the PDF. However, since the HTML version is not an official release, it is more of an unwritten rule to publish the HTML at the same time as producing a release candidate for the PDF.

These instructions assume you have the proper dependencies for building HTML and are publishing both versions simultaneously. See Additional Pre-Requisites for HTML for details.

Pre-Requisites

The PDF version of the Solr Reference Guide is the official version. As such, it is voted on by the community before release, and is treated as an official artifact of the Lucene/Solr project.

Pre-Requisites for PDF

The build process downloads the required .jar to build the PDF version, so the only pre-requisites are for the publication process:

  • You have checked out the Lucene/Solr source code on the machine you will be doing the release from. You will need scripts in the dev-tools directory.
  • You have generated a GPG key. See the Apache documentation on generating a code signing key.
  • You have Python 3 installed. This is needed to poll the mirrors after release to be sure it’s propagated enough to make the announcement.
  • You have Subversion installed. This is needed for committing the PDF to dist/dev and dist/releases repos.

Additional Pre-Requisites for HTML

To build the PDF locally, you need several required dependencies, listed in solr-ref-guide/README.adoc. If you do not have these dependencies locally, you cannot build and publish the Ref Guide.

Builds are available via Jenkins for several branches. However, these HTML pages will have the DRAFT status noted in them and will not be suitable for publishing.

Generate the PDF & HTML

No local dependencies are required to build the PDF. The Ant target will download the jars and other items it requires.

The build process generates the PDF, including the page hierarchy, and then runs an optimization script on the PDF to make it smaller.

To build the PDF:

  1. Run ant clean default -Dsolr-guide-version=X.Y, where X.Y is the version you want to publish (i.e., 7.4).
  2. The resulting PDF will be in solr/build/solr-ref-guide. The HTML files will be in solr/build/solr-ref-guide/html-site.
The -Dsolr-guide-version system property is optional. By default the build system uses the version.properties of the current branch and assumes this is a DRAFT build. Including this system property ensures the DRAFT watermark and labels are removed from the HTML files.

Prepare & Upload Release Candidate

Upload PDF to dist/dev

The dist/dev Subversion repository includes a directory for the Solr Ref Guide at lucene/solr/ref-guide which can host the release candidate (RC) during the VOTE stage of the process.

These steps walk through checking out this directory and uploading the Guide to it.

  1. Checkout the directory: svn co https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide solr-ref-guide-rc
    • If you have already checked out this directory, you can simply update it: svn update solr-ref-guide-rc
  2. Change directories so solr-ref-guide-rc is your working directory (cd solr-ref-guide-rc).
The next step requires that you have already generated your GPG keys. Your GPG passphrase will be required.
  1. Run the Prep Ref Guide script to prepare the RC. This script ensures proper naming of the PDF file, generates .sha512 and .asc files and creates the proper RC sub-directories under solr-ref-guide-rc.
    1. The structure of the input is: prep-solr-ref-guide-rc.sh <path/PDFfilename> <Solrversion-RC#> GPGkey.

    2. From the solr-ref-guide-rc directory, it will look something like this:

      $ ~/lucene-source/dev-tools/scripts/prep-solr-ref-guide-rc.sh apache-solr-ref-guide-7.0.pdf 7.0-RC0

+ mkdir apache-solr-ref-guide-7.0-RC0
+ mv apache-solr-ref-guide-7.0.pdf apache-solr-ref-guide-7.0-RC0/apache-solr-ref-guide-7.0.pdf
+ cd apache-solr-ref-guide-7.0-RC0
+ shasum -a 512 apache-solr-ref-guide-7.0.pdf
+ gpg -u DEADBEEF --armor --output apache-solr-ref-guide-7.0.pdf.asc --detach-sig apache-solr-ref-guide-7.0.pdf

You need a passphrase to unlock the secret key for
user: "Your Name <you@apache.org>"
4096-bit RSA key, ID DEADBEEF, created 1969-07-04
  2. Add and commit the new release candidate to the dist/dev with these steps:
    1. svn add apache-solr-ref-guide-7.0-RC0
    2. svn commit -m "7.0 ref guide RC0"

Upload HTML Files

Step 1: Update extpaths.txt in CMS Staging

  1. Checkout CMS trunk:

    svn co --depth=immediates https://svn.apache.org/repos/asf/lucene/cms/trunk/content website-source
    • If you already have this repo checked out, you can simply svn up website-source to update to the latest revision.
  2. cd website-source
  3. Add Guide branch dir: echo solr/guide/X_Y >> extpaths.txt

  4. Commit changes:

    svn commit -m "Update CMS production sync exceptions for X_Y_Z Guide" extpaths.txt

Step 2: Push Guide to Website Production

Go to the checkout directory where you have built the Guide and push the documentation via subversion import. You must push it to the path you just added to extpaths.txt, so if the path you added was solr/guide/6_5, you’ll use the path as shown in the below example:

svn -m "Add Ref Guide for Solr 6.5" import <checkoutroot>/solr/build/solr-ref-guide/html-site https://svn.apache.org/repos/infra/websites/production/lucene/content/solr/guide/6_5

Confirm you can browse to these URLs manually, and especially that Solr javadocs link back to lucene’s correctly. Example: https://lucene.apache.org/solr/guide/6_5

Step 3: Push Staging extpaths.txt to Production

The extpaths.txt works by listing paths that should be ignored when the CMS syncs the staging and production repositories. Publishing staging to production will only succeed if the paths listed in extpaths.txt exist in production. At the same time, if a path exists in production but not in staging it will be deleted unless it is defined in extpaths.txt.

After pushing the content to production, check that the extpaths.txt in production includes the proper path to ensure that the Guide is not deleted incorrectly. If it does not exist in production, try to publish the site again to make sure it is updated.

Production URL: https://lucene.apache.org/extpaths.txt

Hold a VOTE

Votes must be sent to the lucene-dev mailing list (dev@lucene.apache.org).

  1. Send an email to dev@lucene.apache.org with subject, "VOTE: Release Apache Solr Ref Guide for Solr X.Y".
  2. The body of the email should include the full URL of the RC directory in the dist/dev repo. Such as: https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-7.0-RC0
  3. You can add your own +1 to the vote announcement email.
  4. If there are issues that need to be resolved, you can start the process over, using RC1, RC2, etc., as needed.
Ideally, the HTML version will also be available for voters to review.

Publish the Guide

Move the PDF to dist/releases

Once at least three PMC members have voted for release (see Apache Voting Process for details on the rules for votes), the release candidate can be released.

  1. Use the Publish Solr Ref Guide script (publish-solr-ref-guide.sh) to generate the proper SVN commands to be run to execute a remote move of the RC files to the final dist/releases repository.
    1. The script takes only the version and RC number that passed the vote as inputs, such as 7.0-RC0.

    2. The input and output of the script will look like this:

      $ ~/lucene-source/dev-tools/scripts/publish-solr-ref-guide-rc.sh X.Y-RCZ

## Run the following commands when ready...
svn move -m 'publishing apache-solr-ref-guide-X.Y-RCZ' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf.asc https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf.sha512 https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/

svn rm -m 'cleaning up apache-solr-ref-guide-X.Y-RCZ' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ

  2. The release should propagate to as many mirrors as possible before announcing the release, generally 24 hours is long enough. Use the Poll Mirrors script (poll-mirrors.py) to check the status:

    python3 -u ~/lucene-source/dev-tools/scripts/poll-mirrors.py -details -p lucene/solr/ref-guide/apache-solr-ref-guide-X.Y.pdf
    • This script requires Python 3 to be installed on your machine.
    • If you have over 85% of the mirrors with the file, it’s OK to go ahead with the announcement.
  3. You may get an automated email about updating the ASF release repository; you can safely ignore this email.

Archive Old PDF Versions

  1. The dist/releases repo is only meant to keep the latest releases. Shortly after new releases are mirrored, they are copied to archive.apache.org, so older releases can safely be deleted from dist/releases since they have been backed up in the archives.
    1. Run the Archive Ref Guide script (archive-solr-ref-guide.sh) using the X.Y version of the Ref Guide that has just been published. Older RCs will also be removed.

    2. Again, this script doesn’t do any direct removal of files, it only outputs SVN commands for you to copy and paste:

      $ ~/lucene-source/dev-tools/scripts/archive-solr-ref-guide.sh X.Y
## Run the following commands when ready...

# Delete old releases
svn rm -m 'removing archived ref guide files prior to X.Y' https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf.asc https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf.sha512

# Delete old RC files
svn rm -m 'cleaning up old RCs now that X.Y has been released' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RC0/ https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RC1/

Make New HTML Version the Default

When the PDF is announced as available, the HTML version should already be available on the Solr website. There are a few steps to take to make the new HTML version the default.

You can use the CMS system for these changes, or you can edit the file locally and commit it to the staging repo.
  1. Update the landing page at https://lucene.apache.org/solr/guide (the file is at content/solr/guide/index.mdtext in SVN) to link to the newest version.

  2. Update the Guide redirect rule that looks like the below in content/.htaccess so URLs without a version in the path are redirected to the latest available version.

    RedirectMatch temp /solr/guide/(?!index.html)([a-z].*) /solr/guide/7_0/$1

    In the above example, you would change the 7_0 part of the path to the right version (7_1, etc.).

Announce the Release

Announce the availability of the new Ref Guide on solr-user@lucene.apache.org and CC general@lucene.apache.org and announce@apache.org.

You must send the announcement email from your @apache.org email address or announce@apache will reject it.

Always use the link to the download redirector for the announcement, as it will automatically direct users to the closest mirror for download: https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y.pdf.

You should include a link to the HTML version in your announcement.

Sample announcement
The Lucene PMC is pleased to announce that the Solr Reference Guide
for 7.0 is now available.

This 1,035-page PDF is the definitive guide to using Apache Solr, the
search server built on Lucene.

The PDF Guide can be downloaded from:
https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-guide-7.0.pdf.

It is also available online at https://lucene.apache.org/solr/guide/7_0.

If the Guide is being published more than a day or two after the application itself, you should update the Solr website news page with the announcement (https://lucene.apache.org/solr/news.html).

Updating HTML Files after Publication

If you need to re-publish an existing online copy of the Guide, you will need to checkout the directory in production website repository and overwrite the existing files:

  1. Build the new HTML files locally (ant clean build-site), or download them from Jenkins.
  2. Checkout the directory you need to update from the production repo: svn co https://svn.apache.org/repos/infra/websites/production/lucene/content/solr/guide/<dir>;.
    • This command checks out the Guide version directory into a local subdirectory with the same name as the version (such as "6_5"). You can provide a better name locally if you prefer by adding it to the end of the command shown above.
    • Don’t shortcut this and download the whole production website. It will take an incredibly long time and that will feel like forever.
  3. Copy the files from the build location to the checked out Guide directory. For example, if we needed to replace the current Guide for Solr 6.5, we’d do cp -r <checkoutroot>/solr/build/html-site 6_5/.
  4. Use svn status to see the files modified.
  5. If there are any pages added or deleted, use svn add <file> or svn rm <file> as needed.
  6. Commit the changes: svn commit -m "Update production 6.5 Ref Guide"