Package org.apache.vxquery.hdfs2

Class HDFSFunctions

java.lang.Object

org.apache.vxquery.hdfs2.HDFSFunctions

public class HDFSFunctions
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
protected static Logger	LOGGER

Constructor Summary

Constructors

Constructor	Description
HDFSFunctions(Map<String,org.apache.hyracks.api.client.NodeControllerInfo> nodeControllerInfos, String hdfsConf)	Create the configuration and add the paths for core-site and hdfs-site as resources.

Method Summary

Modifier and Type	Method	Description
void	addScheduleToDistributedCache()	Writes the schedule to a temporary file, then uploads the file to the HDFS.
Document	convertStringToDocument(String xmlStr)
org.apache.hadoop.fs.FileSystem	getFileSystem()	Get instance of the HDFSfile system if it is configured correctly.
org.apache.hadoop.mapreduce.InputFormat	getinputFormat()
org.apache.hadoop.mapreduce.Job	getJob()
HashMap<String,ArrayList<Integer>>	getLocationsOfSplits()	Create a HashMap that has as key the hostname and values the splits that belong to this hostname;
org.apache.hadoop.mapreduce.RecordReader	getReader()
HashMap<Integer,String>	getSchedule()
ArrayList<Integer>	getScheduleForNode(String node)	Return the splits belonging to this node for the existing schedule.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits()
boolean	isLocatedInHDFS(String filename)	Returns true if the file path exists or it is located somewhere in the home directory of the user that called the function.
boolean	put(String filepath, String dir)	Upload a file/directory to HDFS.Filepath is the path in the local file system.dir is the destination path.
void	readNodesFromXML()	Read the hostname and the ip address of every node from the xml cluster configuration file.
void	scheduleSplits()
org.apache.hadoop.fs.Path	searchInDirectory(org.apache.hadoop.fs.Path directory, String filename)	Searches the given directory for the file.
void	setJob(String filepath, String tag)	Create the needed objects for reading the splits of the filepath given as argument.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

protected static final Logger LOGGER

Constructor Detail

HDFSFunctions

public HDFSFunctions(Map<String,org.apache.hyracks.api.client.NodeControllerInfo> nodeControllerInfos,
                     String hdfsConf)

Create the configuration and add the paths for core-site and hdfs-site as resources. Initialize an instance of HDFS FileSystem for this configuration.

Parameters:

nodeControllerInfos -

hdfsConf -

Method Detail

setJob

public void setJob(String filepath,
                   String tag)

Create the needed objects for reading the splits of the filepath given as argument. This method should run before the scheduleSplits method.

Parameters:

filepath -

isLocatedInHDFS

public boolean isLocatedInHDFS(String filename)
                        throws IllegalArgumentException,
                               IOException

Returns true if the file path exists or it is located somewhere in the home directory of the user that called the function. Searches in subdirectories of the home directory too.

Parameters:

filename -

Returns:

Throws:

IOException

IllegalArgumentException

searchInDirectory

public org.apache.hadoop.fs.Path searchInDirectory(org.apache.hadoop.fs.Path directory,
                                                   String filename)

Searches the given directory for the file.

Parameters:

directory - to search

filename - of file we want

Returns:

path if file exists in this directory.else return null.

put

public boolean put(String filepath,
                   String dir)

Upload a file/directory to HDFS.Filepath is the path in the local file system.dir is the destination path.

Parameters:

filepath -

dir -

Returns:

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

Get instance of the HDFSfile system if it is configured correctly. Return null if there is no instance.

Returns:

getLocationsOfSplits

public HashMap<String,ArrayList<Integer>> getLocationsOfSplits()
                                                        throws IOException

Create a HashMap that has as key the hostname and values the splits that belong to this hostname;

Returns:

Throws:

IOException

scheduleSplits

public void scheduleSplits()
                    throws IOException,
                           ParserConfigurationException,
                           SAXException

Throws:

IOException

ParserConfigurationException

SAXException

readNodesFromXML

public void readNodesFromXML()
                      throws ParserConfigurationException,
                             SAXException,
                             IOException

Read the hostname and the ip address of every node from the xml cluster configuration file. Save the information inside nodes.

Throws:

ParserConfigurationException

IOException

SAXException

addScheduleToDistributedCache

public void addScheduleToDistributedCache()
                                   throws FileNotFoundException,
                                          UnsupportedEncodingException

Writes the schedule to a temporary file, then uploads the file to the HDFS.

Throws:

UnsupportedEncodingException

FileNotFoundException

getReader

public org.apache.hadoop.mapreduce.RecordReader getReader()

getSchedule

public HashMap<Integer,String> getSchedule()

Returns:

schedule.

getScheduleForNode

public ArrayList<Integer> getScheduleForNode(String node)

Return the splits belonging to this node for the existing schedule.

Parameters:

node -

Returns:

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits()

getJob

public org.apache.hadoop.mapreduce.Job getJob()

getinputFormat

public org.apache.hadoop.mapreduce.InputFormat getinputFormat()

convertStringToDocument

public Document convertStringToDocument(String xmlStr)