OutOfCoreEngine (Apache Giraph Parent 1.4.0-SNAPSHOT API)

java.lang.Object
- org.apache.giraph.ooc.OutOfCoreEngine

All Implemented Interfaces:

ResetSuperstepMetricsObserver
```
public class OutOfCoreEngine
extends Object
implements ResetSuperstepMetricsObserver
```
Class to represent an out-of-core engine.

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`CHECK_IN_INTERVAL` Number of 'units of processing' after which an active thread should check-in with the out-of-core engine in order to re-claim its permission to stay active.
`static String`	`GRAPH_PERCENTAGE_IN_MEMORY` Name of metric for percentage of graph on disk

Constructor Summary

Constructors
Constructor and Description
`OutOfCoreEngine(ImmutableClassesGiraphConfiguration<?,?,?> conf, CentralizedServiceWorker<?,?,?> service, NetworkMetrics networkMetrics)` Constructor

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`activeThreadCheckIn()` A processing thread would check in with out-of-core engine every once in a while to make sure that it can still remain active.
`void`	`doneProcessingPartition(int partitionId)` Notify out-of-core engine that processing of a particular partition is done
`void`	`gcCompleted(com.sun.management.GarbageCollectionNotificationInfo info)` Notify the out-of-core engine that a GC has just been completed
`OutOfCoreDataAccessor`	`getDataAccessor()`
`FlowControl`	`getFlowControl()`
`OutOfCoreIOScheduler`	`getIOScheduler()` Get a reference to IO scheduler
`OutOfCoreIOStatistics`	`getIOStatistics()` Get a reference to IO statistics collector
`MetaPartitionManager`	`getMetaPartitionManager()` Get a reference to meta partition information
`NetworkMetrics`	`getNetworkMetrics()`
`Integer`	`getNextPartition()` Get the id of the next partition to process in the current iteration over all the partitions.
`OutOfCoreOracle`	`getOracle()` Get a reference to out-of-core oracle
`ServerData`	`getServerData()` Get a reference to the server data
`CentralizedServiceWorker`	`getServiceWorker()` Get a reference to the service worker
`long`	`getSuperstep()`
`ReadWriteLock`	`getSuperstepLock()` Get a reference to superstep lock
`void`	`initialize()` Initialize/Start the out-of-core engine.
`void`	`ioCommandCompleted(IOCommand command)` Notify out-of-core engine that an IO command is completed by an IO thread
`void`	`newSuperstep(SuperstepMetricsRegistry superstepMetrics)` Starting a new superstep.
`void`	`processingThreadFinish()` Notify the out-of-core engine that a processing (input/compute) thread has finished.
`void`	`processingThreadStart()` Notify the out-of-core engine that a processing (input/compute) thread has started.
`void`	`reset()` Reset partitions and messages meta data.
`void`	`retrievePartition(int partitionId)` Retrieve a particular partition.
`void`	`setFlowControl(FlowControl flowControl)`
`void`	`shutdown()` Shutdown/Stop the out-of-core engine.
`void`	`startIteration()` Notify out-of-core engine that iteration cycle over all partitions is about to begin.
`void`	`updateActiveThreadsFraction(double fraction)` Update the fraction of processing threads that should remain active.
`void`	`updateRequestsCreditFraction(double fraction)` Update the credit announced for this worker in Netty.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - CHECK_IN_INTERVAL
```
public static final int CHECK_IN_INTERVAL
```
    Number of 'units of processing' after which an active thread should check-in with the out-of-core engine in order to re-claim its permission to stay active. For a compute thread, the 'unit of processing' is processing of one vertex, and for an input thread, the 'unit of processing' is reading a row of input data.
    
    See Also:
    
    Constant Field Values
  - GRAPH_PERCENTAGE_IN_MEMORY
```
public static final String GRAPH_PERCENTAGE_IN_MEMORY
```
    Name of metric for percentage of graph on disk
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - OutOfCoreEngine
```
public OutOfCoreEngine(ImmutableClassesGiraphConfiguration<?,?,?> conf,
                       CentralizedServiceWorker<?,?,?> service,
                       NetworkMetrics networkMetrics)
```
    Constructor
    
    Parameters:
    
    conf - Configuration
    
    service - Service worker
    
    networkMetrics - Interface for network stats
- Method Detail
  - initialize
```
public void initialize()
```
    Initialize/Start the out-of-core engine.
  - shutdown
```
public void shutdown()
```
    Shutdown/Stop the out-of-core engine.
  - getServerData
```
public ServerData getServerData()
```
    Get a reference to the server data
    
    Returns:
    
    ServerData
  - getServiceWorker
```
public CentralizedServiceWorker getServiceWorker()
```
    Get a reference to the service worker
    
    Returns:
    
    CentralizedServiceWorker
  - getIOScheduler
```
public OutOfCoreIOScheduler getIOScheduler()
```
    Get a reference to IO scheduler
    
    Returns:
    
    OutOfCoreIOScheduler
  - getMetaPartitionManager
```
public MetaPartitionManager getMetaPartitionManager()
```
    Get a reference to meta partition information
    
    Returns:
    
    MetaPartitionManager
  - getSuperstepLock
```
public ReadWriteLock getSuperstepLock()
```
    Get a reference to superstep lock
    
    Returns:
    
    read/write lock used for global superstep lock
  - getIOStatistics
```
public OutOfCoreIOStatistics getIOStatistics()
```
    Get a reference to IO statistics collector
    
    Returns:
    
    IO statistics collector
  - getOracle
```
public OutOfCoreOracle getOracle()
```
    Get a reference to out-of-core oracle
    
    Returns:
    
    out-of-core oracle
  - getNextPartition
```
public Integer getNextPartition()
```
    Get the id of the next partition to process in the current iteration over all the partitions. If all partitions are already processed, this method returns null.
    
    Returns:
    
    id of a partition to process. 'null' if all partitions are processed in current iteration over partitions.
  - doneProcessingPartition
```
public void doneProcessingPartition(int partitionId)
```
    Notify out-of-core engine that processing of a particular partition is done
    
    Parameters:
    
    partitionId - id of the partition that its processing is done
  - startIteration
```
public void startIteration()
```
    Notify out-of-core engine that iteration cycle over all partitions is about to begin.
  - retrievePartition
```
public void retrievePartition(int partitionId)
```
    Retrieve a particular partition. After this method is complete the requested partition should be in memory.
    
    Parameters:
    
    partitionId - id of the partition to retrieve
  - ioCommandCompleted
```
public void ioCommandCompleted(IOCommand command)
```
    Notify out-of-core engine that an IO command is completed by an IO thread
    
    Parameters:
    
    command - the IO command that is completed
  - updateActiveThreadsFraction
```
public void updateActiveThreadsFraction(double fraction)
```
    Update the fraction of processing threads that should remain active. It is the responsibility of out-of-core oracle to update the number of active threads.
    
    Parameters:
    
    fraction - the fraction of processing threads to remain active. This number is in range [0, 1]
  - activeThreadCheckIn
```
public void activeThreadCheckIn()
```
    A processing thread would check in with out-of-core engine every once in a while to make sure that it can still remain active. It is the responsibility of the out-of-core oracle to update the number of active threads in a way that the computation never fails, and yet achieve the optimal performance it can achieve.
  - processingThreadStart
```
public void processingThreadStart()
```
    Notify the out-of-core engine that a processing (input/compute) thread has started.
  - processingThreadFinish
```
public void processingThreadFinish()
```
    Notify the out-of-core engine that a processing (input/compute) thread has finished.
  - updateRequestsCreditFraction
```
public void updateRequestsCreditFraction(double fraction)
```
    Update the credit announced for this worker in Netty. The lower the credit is, the lower rate incoming messages arrive at this worker. Thus, credit is an indirect way of controlling amount of memory incoming messages would take.
    
    Parameters:
    
    fraction - the fraction of max credits others can use to send requests to this worker
  - reset
```
public void reset()
```
    Reset partitions and messages meta data. Also, reset the cached value of superstep counter.
  - getSuperstep
```
public long getSuperstep()
```
    Returns:
    
    cached value of the superstep counter
  - gcCompleted
```
public void gcCompleted(com.sun.management.GarbageCollectionNotificationInfo info)
```
    Notify the out-of-core engine that a GC has just been completed
    
    Parameters:
    
    info - GC information
  - newSuperstep
```
public void newSuperstep(SuperstepMetricsRegistry superstepMetrics)
```
    Description copied from interface: ResetSuperstepMetricsObserver
    
    Starting a new superstep. Re-initialize your metrics.
    
    Specified by:
    
    newSuperstep in interface ResetSuperstepMetricsObserver
    
    Parameters:
    
    superstepMetrics - SuperstepMetricsRegistry being used.
  - getFlowControl
```
public FlowControl getFlowControl()
```
  - setFlowControl
```
public void setFlowControl(FlowControl flowControl)
```
  - getDataAccessor
```
public OutOfCoreDataAccessor getDataAccessor()
```
  - getNetworkMetrics
```
public NetworkMetrics getNetworkMetrics()
```

Class OutOfCoreEngine

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

CHECK_IN_INTERVAL

GRAPH_PERCENTAGE_IN_MEMORY

Constructor Detail

OutOfCoreEngine

Method Detail

initialize

shutdown

getServerData

getServiceWorker

getIOScheduler

getMetaPartitionManager

getSuperstepLock

getIOStatistics

getOracle

getNextPartition

doneProcessingPartition

startIteration

retrievePartition

ioCommandCompleted

updateActiveThreadsFraction

activeThreadCheckIn

processingThreadStart

processingThreadFinish

updateRequestsCreditFraction

reset

getSuperstep

gcCompleted

newSuperstep

getFlowControl

setFlowControl

getDataAccessor

getNetworkMetrics