org.apache.hadoop.chukwa.datacollection.sender

Class ChukwaHttpSender

java.lang.Object

org.apache.hadoop.chukwa.datacollection.sender.ChukwaHttpSender

All Implemented Interfaces:

ChukwaSender

Direct Known Subclasses:

AsyncAckSender

public class ChukwaHttpSender
extends Object
implements ChukwaSender

Encapsulates all of the http setup and connection details needed for chunks to be delivered to a collector. This class should encapsulate the details of the low level data formatting. The Connector is responsible for picking what to send and to whom; retry policy is encoded in the collectors iterator.

On error, tries the list of available collectors, pauses for a minute, and then repeats.

Will wait forever for collectors to come up.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ChukwaHttpSender.CommitListEntry

Field Summary

Fields

Modifier and Type	Field and Description
static String	COLLECTOR_TIMEOUT_OPT
protected Iterator<String>	collectors

Constructor Summary

Constructors

Constructor and Description
ChukwaHttpSender(org.apache.hadoop.conf.Configuration c)

Method Summary

Methods

Modifier and Type	Method and Description
protected List<String>	doRequest(org.apache.commons.httpclient.HttpMethodBase method, String dest) Responsible for performing a single operation to a specified collector URL.
protected boolean	failedCollector(String downCollector) A hook for taking action when a collector is declared failed.
List<ChukwaHttpSender.CommitListEntry>	postAndParseResponse(org.apache.commons.httpclient.methods.PostMethod method, List<ChukwaHttpSender.CommitListEntry> expectedCommitResults)
protected List<String>	reliablySend(org.apache.commons.httpclient.HttpMethodBase method, String pathSuffix) Responsible for executing the supplied method on at least one collector
List<ChukwaHttpSender.CommitListEntry>	send(List<Chunk> toSend) grab all of the chunks currently in the chunkQueue, stores a copy of them locally, calculates their size, sets them up
void	setCollectors(Iterator<String> collectors) Set up a list of connectors for this client to send Chunks to
void	stop()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COLLECTOR_TIMEOUT_OPT

public static final String COLLECTOR_TIMEOUT_OPT

See Also:

Constant Field Values

collectors

protected Iterator<String> collectors

Constructor Detail

ChukwaHttpSender

public ChukwaHttpSender(org.apache.hadoop.conf.Configuration c)

Method Detail

setCollectors

public void setCollectors(Iterator<String> collectors)

Set up a list of connectors for this client to send Chunks to

Specified by:

setCollectors in interface ChukwaSender

Parameters:

collectors -

send

public List<ChukwaHttpSender.CommitListEntry> send(List<Chunk> toSend)
                                            throws InterruptedException,
                                                   IOException

grab all of the chunks currently in the chunkQueue, stores a copy of them locally, calculates their size, sets them up

Specified by:

send in interface ChukwaSender

Parameters:

toSend - a list of chunks to commit

Returns:

array of chunk id's which were ACKed by collector

Throws:

InterruptedException - if interrupted while trying to send

IOException

postAndParseResponse

public List<ChukwaHttpSender.CommitListEntry> postAndParseResponse(org.apache.commons.httpclient.methods.PostMethod method,
                                                          List<ChukwaHttpSender.CommitListEntry> expectedCommitResults)
                                                            throws IOException,
                                                                   InterruptedException

Parameters:

method - the data to push

expectedCommitResults - the list

Returns:

the list of committed chunks

Throws:

IOException

InterruptedException

reliablySend

protected List<String> reliablySend(org.apache.commons.httpclient.HttpMethodBase method,
                        String pathSuffix)
                             throws InterruptedException,
                                    IOException

Responsible for executing the supplied method on at least one collector

Parameters:

method -

Returns:

Throws:

InterruptedException

IOException - if no collector responds with an OK

failedCollector

protected boolean failedCollector(String downCollector)

A hook for taking action when a collector is declared failed. Returns whether to retry current post, or junk it

Parameters:

downCollector -

doRequest

protected List<String> doRequest(org.apache.commons.httpclient.HttpMethodBase method,
                     String dest)
                          throws IOException,
                                 org.apache.commons.httpclient.HttpException

Responsible for performing a single operation to a specified collector URL.

Parameters:

dest - the URL being requested. (Including hostname)

Throws:

IOException

org.apache.commons.httpclient.HttpException

stop

public void stop()

Specified by:

stop in interface ChukwaSender