Chukwa Change Log

Trunk (unrelease changes)

Release 0.1.2 - 2009-05-14

• NEW FEATURES    (13)
  1. CHUKWA-236. Added migration script for moving database schema for Chukwa 0.1.1 to Chukwa 0.1.2.
     (Eric Yang)
  2. CHUKWA-100. Added ability to run aggregation by command line utility.
     (Eric Yang)
  3. CHUKWA-78. Added down sample SQL aggregation for job data, task data and utilization data.
     (Eric Yang)
  4. CHUKWA-17. Collect PS output every 10 minutes and parse into key/value pairs sequence file.
     (Cheng Zhang via Eric Yang)
  5. CHUKWA-55. Aggregate data from HDFS bytes usage, and Mapreduce Job slot time to compute user usage of the cluster.
     (Eric Yang)
  6. CHUKWA-50. Added parser for extract Job History and Job Conf into key value pairs, and database loader dictionary file.
     (Contribute by Cheng Zhang via Eric Yang)
  7. CHUKWA-69. Calculate trapezoid area for a given series of data.
     (Contribute by Cheng Zhang via Eric Yang)
  8. CHUKWA-62. Add script to start tailing files.
     (asrabkin)
  9. CHUKWA-12. Add instrumention Api for Chukwa components.
     (Jerome Boulon via asrabkin)
  10. CHUKWA-14. Added permalink support to HICC GUI.
      (Eric Yang)
  11. CHUKWA-45. Added docs to folder structure.
      (Corinne Chandel via asrabkin)
  12. CHUKWA-13. Create a new Macro class for macro substitution. Changed Aggregator to use Macro class. Added macro substitution support to Chukwa Charting. Added Macro test case.
      (Eric Yang)
  13. HADOOP-4989. Added capability to add scatter chart.
      (Eric Yang)
• IMPROVEMENTS    (33)
  1. CHUKWA-226. Changed HDFS usage collection frequency from 10 minutes to 60 minutes.
     (Cheng Zhang via Eric Yang)
  2. CHUKWA-219. Improved usability of host selector, and updated caching for host list.
     (Eric Yang)
  3. CHUKWA-104. Remove permission setting code for Log4JMetricsContext and ChukwaDailyRollingFileAppender.
     (Jerome Boulon via Eric Yang)
  4. CHUKWA-178. Extend aggregation by 5 minutes for cluster metrics.
     (Eric Yang)
  5. CHUKWA-176. Rearrange the parameter order for aggregator.sh.
     (Eric Yang)
  6. CHUKWA-174. Added test cases to test database partitioning, database aggregation, and data loading.
     (Eric Yang)
  7. CHUKWA-173. Parameterize configuration, and enable substitution at build time.
     (Jerome Boulon via Eric Yang)
  8. CHUKWA-131. Added additional Mapred job/task metrics.
     (Eric Yang)
  9. CHUKWA-145. Tuned hadoop parameters for demux.
     (Jerome Boulon via Eric Yang)
  10. CHUKWA-163. Updated reference to mdl.xml file.
      (Terence Kwan via Eric Yang)
  11. CHUKWA-157. Added javadoc target, api-xml, api-report, and change log 2 html.
      (Eric Yang)
  12. CHUKWA-141. Updated Chukwa Docs overview page for 0.1.2.
      (Corinne Chandel via Eric Yang)
  13. CHUKWA-138. Updated Chukwa Admin Guide.
      (Corinne Chandel via Eric Yang)
  14. CHUKWA-140. Updated Chukwa Docs tab for 0.1.2.
      (Corinne Chandel via Eric Yang)
  15. CHUKWA-112. Updated README file.
      (Corinne Chandel via Eric Yang)
  16. CHUKWA-134. Add release audit target.
      (Giridharan Kesavan via Eric Yang)
  17. CHUKWA-80. Extracted rpm spec file from build.xml file. Fix start up script, and config script.
      (Eric Yang)
  18. CHUKWA-26. DemuxManager, ArchiveManager and PostProcessorManager are now a single daemon process each. Each one working independently from others, as soon as something is available. Start-data-processor is now using those new daemons instead of pocessSink.sh Daily will process a daily compaction only when all hourly would have been done. Demux is now able to send NSCA commands to Nagios.
      (Jerome Boulon via Eric Yang)
  19. CHUKWA-81. Fix file descriptor leak.
      (asrabkin)
  20. CHUKWA-29. Added TaskTracker and DataNode client trace log parser and database loader.
      (Chris Douglas via Eric Yang)
  21. CHUKWA-86. Improved JDBC compatibility layer
      (Cheng Zhang via Eric Yang)
  22. CHUKWA-64. Changed version number to 0.1.2, and release number to alpha.
      (Eric Yang)
  23. CHUKWA-59. Collect HDFS Usage information for users.
      (Contribute by Cheng Zhang via Eric Yang)
  24. CHUKWA-54. Filters web based input from HICC to prevent cross site scripting attack.
      (Eric Yang)
  25. CHUKWA-31. Added duration logging for SQL statements.
      (Eric Yang)
  26. HADOOP-5373. Collectors track lifetime-received chunks.
      (asrabkin)
  27. HADOOP-5370. Collectors don't write empty sink files
      (asrabkin)
  28. HADOOP-5228. Chukwa tests shouldn't write to /tmp.
      (asrabkin)
  29. HADOOP-5035. Improved Y axis ticker labelling. Used TreeMap to build non-time series data for charting. Improved handling of "not a number "values.
      (Eric Yang)
  30. HADOOP-5030. Changed RPM install location to the value specified by build.properties file.
      (Eric Yang)
  31. HADOOP-5029. Added mdl script to manually load chukwa sequence file to database.
      (Eric Yang)
  32. HADOOP-4843. Enable job history log file streaming into Chukwa by using JobTrackerInstrumentation API.
      (Eric Yang)
  33. HADOOP-4827. Replace Consolidator with Aggregator macros Updated database table schema Fix interval for aggregation execute Improved SQL macros support, add group_avg() and past_*_minutes key word
      (Eric Yang)
• OPTIMIZATIONS    (none)
• BUG FIXES    (86)
  1. CHUKWA-250. Updated hadoop jar reference in chukwa config.
     (Cheng Zhang via Eric Yang)
  2. CHUKWA-211. Manage symlink correctly for RPM upgrade and uninstall.
     (Eric Yang)
  3. CHUKWA-245. Includes ChukwaJobTrackerInstrumentation class as part of the input tools compilation.
     (Eric Yang)
  4. CHUKWA-232. Fixed classpath for chukwa core jar file.
     (Eric Yang)
  5. CHUKWA-215. Corrected postProcess.sh environment setup.
     (Jerome Boulon via Eric Yang)
  6. CHUKWA-223. Renamed read only session caching object to cache.*, and skip those values for caching.
     (Eric Yang)
  7. CHUKWA-239. Demux settings now work out of the box.
     (Ari Rabkin)
  8. CHUKWA-238. Resolve race condition in archiving.
     (Ari Rabkin)
  9. CHUKWA-243. Set execute permission on bin scripts.
     (Ari Rabkin)
  10. CHUKWA-228. Added rpm.hdfsusage.uid option to run HDFS usage as a separate user.
      (Cheng Zhang via Eric Yang)
  11. CHUKWA-227. Improved the user privilege for reading HDFS usage.
      (Cheng Zhang via Eric Yang)
  12. CHUKWA-229. Fix descriptor leak for ExecPlugin.
      (Ari Rabkin via Eric Yang)
  13. CHUKWA-212. Fix file descriptor leak in MDL.
      (Jerome Boulon via Eric Yang)
  14. CHUKWA-220. Correct min, max settings for yaxis charting.
      (Terence Kwan via Eric Yang)
  15. CHUKWA-103. Fix documentation broken link.
      (Eric Yang)
  16. CHUKWA-222. Reduce Sar version dependency.
      (Ari Rabkin)
  17. CHUKWA-208. Skip copying of test/sample directory if it doesn't exist.
      (Eric Yang)
  18. CHUKWA-206. Removed hard coded path from configuration files, and sample data.
      (Eric Yang)
  19. CHUKWA-201. Removed sourcing of Timeline widget from simile.mit.edu.
      (Eric Yang)
  20. CHUKWA-193. Remove unrecognized tag from Job History Log file parser.
      (Cheng Zhang via Eric Yang)
  21. CHUKWA-189. Added JAVA_LIBRARY_PATH to chukwa-env.sh for enabling compression.
      (Jerome Boulon via Eric Yang)
  22. CHUKWA-186. Remove duplicated timestamp from aggregation script.
      (Eric Yang)
  23. CHUKWA-187. Correction to status for job and task status.
      (Cheng Zhang via Eric Yang)
  24. CHUKWA-183. Added HICC startup script.
      (Eric Yang)
  25. CHUKWA-177. Added test case for verify the value between JSON values and database values.
      (Terence Kwan via Eric Yang)
  26. CHUKWA-181. Changed chukwa check point file to $CHUKWA_LOG_DIR.
      (Jerome Boulon via Eric Yang)
  27. CHUKWA-182. Added nagios appender configuration.
      (Eric Yang)
  28. CHUKWA-158. Consolidate namenode address to a single configuration object.
      (Eric Yang)
  29. CHUKWA-168. Added watchdog for database.
      (Eric Yang)
  30. CHUKWA-175. Removed error message for shutting down data processors.
      (Eric Yang)
  31. CHUKWA-155. Store final job status only
      (Cheng Zhang via Eric Yang)
  32. CHUKWA-164. Use year corresponding to the sender time stamp.
      (Cheng Zhang via Eric Yang)
  33. CHUKWA-166. Handle null parameter case for XSSFilter.
      (Terence Kwan via Eric Yang)
  34. CHUKWA-156. Test Macro testcase changed to use timestamp check for the generated macros.
      (Eric Yang)
  35. CHUKWA-154. Handle adaptor exception, close file pointers on failure condition.
      (Jerome Boulon via Eric Yang)
  36. CHUKWA-139. Rewrite collector bail out code.
      (Cheng Zhang via Eric Yang)
  37. CHUKWA-119. Removed dependency of ChukwaAgent from ChunkImpl for preventing multiple MetricsContext to be initialized in the same VM.
      (Jerome Boulon via Eric Yang)
  38. CHUKWA-121. Added logic to detect partition number less than 0.
      (Eric Yang)
  39. CHUKWA-132. Handle multi-line output in Job History file more gracefully.
      (Cheng Zhang via Eric Yang)
  40. CHUKWA-124. Fixed JDBC connection initialization and closing. Added escapeQuotes utility method.
      (Eric Yang)
  41. CHUKWA-122. Updated logic for finding time partition for Macro.
      (Eric Yang)
  42. CHUKWA-125. Updated chukwa demux template to include JobHistory parser.
      (Eric Yang)
  43. CHUKWA-98. Added Daemon watcher to capture signal for pid file management.
      (Cheng Zhang via Eric Yang)
  44. CHUKWA-120. Added the missing commons-cli library.
      (Jerome Boulon via Eric Yang)
  45. CHUKWA-61. Added report widget for accounting information.
      (Eric Yang)
  46. CHUKWA-77. Added Database Schema for Hadoop accounting information.
      (Eric Yang)
  47. CHUKWA-92. AbstractMetricsContext was using the wrong value
      (Jerome Boulon via asrabkin)
  48. CHUKWA-70. Rewrite FileAdaptor.
      (Jerome Boulon via asrabkin)
  49. CHUKWA-93. Fix NPE in SeqFileWriter.
      (Jiaqi Tan via asrabkin)
  50. CHUKWA-1. Remove lzo job configuration from Chukwa data processors.
      (Contribute by Jerome Boulon via Eric Yang)
  51. CHUKWA-40. Check for null pointer exception before unregister adaptor.
      (Contribute By Jerome Boulon via Eric Yang)
  52. CHUKWA-75. Filter out iostat values that is greater than 1+e10.
  53. CHUKWA-47. ChukwaJobTrackerInstrumentation class extends JobTrackerMetricsInst and added finalizedJob method for stop streaming job history log file.
      (Jerome Boulon via Eric Yang)
  54. CHUKWA-66. Corrected TerminatorThread logger class name.
      (Eric Yang)
  55. CHUKWA-65. Redirect metrics log file to CHUKWA_LOG_DIR, and setup read/writable permission on the metrics log files.
      (Eric Yang)
  56. CHUKWA-73. Added Socket Timeout 60 seconds.
      (Jerome Boulon via Eric Yang)
  57. CHUKWA-58. Changed watchdog to look for CHUKWA_PID_DIR.
      (Eric Yang)
  58. CHUKWA-43. ChukwaLog4jAppender should send the current file offset instead of sending 0.
      (Jerome Boulon via asrabkin)
  59. CHUKWA-41. ChukwaLog4jAppender does not escape \n for exception.
      (Jerome Boulon via asrabkin)
  60. CHUKWA-28. Late initalization of log4j adaptor.
      (Jerome Boulon via asrabkin)
  61. CHUKWA-48. Cleanup code to resolve compiler warnings.
      (asrabkin)
  62. CHUKWA-9. MetricDataLoader should close JDBC connection.
      (Jerome Boulon via asrabkin)
  63. CHUKWA-37. Remove ChuckwaArchiveBuilder.
      (asrabkin)
  64. CHUKWA-33. Reformat code to fit hadoop style.
      (asrabkin)
  65. CHUKWA-8. Remove deprecated conf files.
      (eyang via asrabkin)
  66. CHUKWA-11. Remove non .template files from conf dir.
      (asrabkin)
  67. HADOOP-5409. Changed opt directory copy to a optional operation, if opt exist.
      (Eric Yang)
  68. HADOOP-5138. Fixing failing test cases by separating TestCharFileTailing and TestFileTailing adaptors.
      (Jerome Boulon via acmurthy)
  69. HADOOP-5401. Standardize control port conf option name as chukwaAgent.control.port
      (asrabkin)
  70. HADOOP-5087. Fix regex for command parsing.
      (asrabkin)
  71. HADOOP-5057. Better test coverage for missing checkpoint cases.
      (asrabkin)
  72. HADOOP-5055. Changed alert.conf location from $CHUKWA_HOME/conf/alert.conf to $CHUKWA_CONF_DIR/alert.conf.
      (Eric Yang)
  73. HADOOP-5054. Improved database partitioning by date.
      (Eric Yang)
  74. HADOOP-5051. * Added macro token subsitution for sum(table_name) * Added correct hdfs throughput aggregation SQL macros.
      (Eric Yang)
  75. HADOOP-5032. Export CHUKWA_CONF_DIR in chukwa-config.sh.
  76. HADOOP-4960. Changed metrics time from current system time to data source time.
  77. HADOOP-4959. Support parsing of top output for Redhat EL 5.1.
      (Eric Yang)
  78. HADOOP-4916. * Added external property file to reference location and ownership of chukwa * Added ability to control the user name to run chukwa
      (Eric Yang)
  79. HADOOP-4914. Added description fields to chukwa init.d scripts.
  80. HADOOP-4889. Move chown from post install phase to build phase of the RPM file.
      (Eric Yang)
  81. HADOOP-4884. Change date format from dd/mm/yyyy to yyyy/mm/dd for display in the chart tool tips.
      (Eric Yang)
  82. HADOOP-4860. Changed test cases into 3 different test classes to prevent Agent from loading chukwa_check_point file which interfered with test cases.
      (Eric Yang)
  83. HADOOP-4825. Replaced jps with ps ax for shutdown scripts. Clean up reference of jar file from top level instead of build directory.
      (Eric Yang)
  84. HADOOP-4805. Removed black list collector feature from Chukwa Agent HTTP Sender.
      (Eric Yang)
  85. HADOOP-4796. Change component unit test targets from "ant test-agent" to "ant -Dtestcase=TestAgent test".
      (Eric Yang)
  86. HADOOP-4791. Add build configuration parameter to specify where Chukwa will be installed for RPM packaging.
      (Eric Yang)

Older Releases

Release 0.1.1 - Unreleased

• IMPROVEMENTS    (4)
  1. HADOOP-4431. Add versionning/tags to Chukwa Chunk.
     (Jerome Boulon via Johan)
  2. HADOOP-4433. Improve data loader for collecting metrics and log files.
     (Eric Yang via omalley)
  3. HADOOP-5205. Change the value of CHUKWA_IDENT_STRING from demo to TODO-AGENTS-INSTANCE-NAME in chukwa-env.sh.template
     (Jerome Boulon via asrabkin)
  4. HADOOP-5033. Simplified ChukwaWriter API.
     (asrabkin)
• NEW FEATURES    (1)
  1. HADOOP-3719. Initial checkin of Chukwa, which is a data collection and analysis framework.
     (Jerome Boulon, Andy Konwinski, Ari Rabkin, and Eric Yang)