org.apache.tez.mapreduce.output

Class MROutput

java.lang.Object

org.apache.tez.runtime.api.AbstractLogicalOutput

org.apache.tez.mapreduce.output.MROutput

All Implemented Interfaces:

org.apache.tez.runtime.api.LogicalOutput, org.apache.tez.runtime.api.LogicalOutputFrameworkInterface, org.apache.tez.runtime.api.Output, org.apache.tez.runtime.api.OutputFrameworkInterface

@InterfaceAudience.Public
public class MROutput
extends org.apache.tez.runtime.api.AbstractLogicalOutput

MROutput is an Output which allows key/values pairs to be written by a processor. It is compatible with all standard Apache Hadoop MapReduce OutputFormat implementations. This class is not meant to be extended by external projects.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapreduce.OutputCommitter	committer

Constructor Summary

Constructors

Constructor and Description
MROutput(org.apache.tez.runtime.api.OutputContext outputContext, int numPhysicalOutputs)

Method Summary

Methods

Modifier and Type	Method and Description
void	abort() MROutput expects that a Processor call abort in case of any error ( including an error during commit ) prior to the Processor's completion
List<org.apache.tez.runtime.api.Event>	close()
void	commit() MROutput expects that a Processor call commit prior to the Processor's completion
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat) Create an MROutput.MROutputConfigBuilder
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat, String outputPath) Create an MROutput.MROutputConfigBuilder for a FileOutputFormat or FileOutputFormat based OutputFormats.
void	flush() Call this in the processor before finishing to ensure outputs that outputs have been flushed.
org.apache.tez.runtime.library.api.KeyValueWriter	getWriter() Get a key value write to write Map Reduce compatible output
void	handleEvents(List<org.apache.tez.runtime.api.Event> outputEvents)
void	initCommitter(org.apache.hadoop.mapred.JobConf job, boolean useNewApi)
List<org.apache.tez.runtime.api.Event>	initialize()
boolean	isCommitRequired()
void	start()

Methods inherited from class org.apache.tez.runtime.api.AbstractLogicalOutput

getContext, getNumPhysicalOutputs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

committer

protected org.apache.hadoop.mapreduce.OutputCommitter committer

Constructor Detail

MROutput

public MROutput(org.apache.tez.runtime.api.OutputContext outputContext,
        int numPhysicalOutputs)

Method Detail

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                 @Nullable
                                                                                 Class<?> outputFormat)

Create an MROutput.MROutputConfigBuilder

The preferred usage model is to provide all of the parameters, and use methods to configure the Output.

For legacy applications, which may already have a fully configured Configuration instance, the outputFormat can be specified as null

Parameters:

conf - Configuration for the MROutput. This configuration instance will be modified in place

outputFormat - OutputFormat derived class. If the OutputFormat specified is null, the provided configuration should be complete.

Returns:

MROutput.MROutputConfigBuilder

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                 @Nullable
                                                                                 Class<?> outputFormat,
                                                                                 @Nullable
                                                                                 String outputPath)

Create an MROutput.MROutputConfigBuilder for a FileOutputFormat or FileOutputFormat based OutputFormats.

The preferred usage model is to provide all of the parameters, and use methods to configure the Output.

For legacy applications, which may already have a fully configured Configuration instance, the outputFormat and outputPath can be specified as null

Parameters:

conf - Configuration for the MROutput. This configuration instance will be modified in place

outputFormat - FileInputFormat derived class. If the InputFormat specified is null, the provided configuration should be complete.

outputPath - Output path. This can be null if already setup in the configuration

Returns:

MROutput.MROutputConfigBuilder

initialize

public List<org.apache.tez.runtime.api.Event> initialize()
                                                  throws IOException,
                                                         InterruptedException

Specified by:

initialize in interface org.apache.tez.runtime.api.OutputFrameworkInterface

Specified by:

initialize in class org.apache.tez.runtime.api.AbstractLogicalOutput

Throws:

IOException

InterruptedException

start

public void start()

initCommitter

public void initCommitter(org.apache.hadoop.mapred.JobConf job,
                 boolean useNewApi)
                   throws IOException,
                          InterruptedException

Throws:

IOException

InterruptedException

isCommitRequired

public boolean isCommitRequired()
                         throws IOException

Throws:

IOException

getWriter

public org.apache.tez.runtime.library.api.KeyValueWriter getWriter()
                                                            throws IOException

Get a key value write to write Map Reduce compatible output

Throws:

IOException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> outputEvents)

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Throws:

IOException

flush

public void flush()
           throws IOException

Call this in the processor before finishing to ensure outputs that outputs have been flushed. Must be called before commit.

Throws:

IOException

commit

public void commit()
            throws IOException

MROutput expects that a Processor call commit prior to the Processor's completion

Throws:

IOException

abort

public void abort()
           throws IOException

MROutput expects that a Processor call abort in case of any error ( including an error during commit ) prior to the Processor's completion

Throws:

IOException