public class QueueSharder<U,T extends QueueBase<U>> extends Object implements Closeable
A utility for shard a distributed queue.
Due to limitations in ZooKeeper's transport layer, a single queue will break if it has more than 10K-ish items in it. This class provides a facade over multiple distributed queues. It monitors the queues and if any one of them goes over a threshold, a new queue is added. Puts are distributed amongst the queues.
NOTE: item ordering is maintained within each managed queue but cannot be maintained across queues. i.e. items might get consumed out of order if they are in different managed queues.
Constructor and Description |
---|
QueueSharder(CuratorFramework client,
QueueAllocator<U,T> queueAllocator,
String queuePath,
String leaderPath,
QueueSharderPolicies policies) |
Modifier and Type | Method and Description |
---|---|
void |
close() |
T |
getQueue()
Return one of the managed queues - the selection method cannot be relied on.
|
Collection<String> |
getQueuePaths()
Return the current set of shard paths
|
int |
getShardQty()
Return the current number of mananged queues
|
void |
start()
The sharder must be started
|
public QueueSharder(CuratorFramework client, QueueAllocator<U,T> queueAllocator, String queuePath, String leaderPath, QueueSharderPolicies policies)
client
- clientqueueAllocator
- allocator for new queuesqueuePath
- path for the queuesleaderPath
- path for the leader that monitors queue sizes (must be different than queuePath)policies
- sharding policiespublic void close()
close
in interface Closeable
close
in interface AutoCloseable
public T getQueue()
public int getShardQty()
public Collection<String> getQueuePaths()
Copyright © 2011–2023 The Apache Software Foundation. All rights reserved.