org.apache.hadoop.hdfs.server.blockmanagement

Class BlockManager

java.lang.Object

org.apache.hadoop.hdfs.server.blockmanagement.BlockManager

@InterfaceAudience.Private
public class BlockManager
extends Object

Keeps information related to the blocks stored in the Hadoop cluster.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	blockLog
static float	DEFAULT_MAP_LOAD_FACTOR Default load factor of map
int	defaultReplication Default number of replicas
Map<String,LightWeightLinkedSet<Block>>	excessReplicateMap Maps a StorageID to the set of blocks that are "extra" for this DataNode.
short	maxReplication The maximum number of replicas allowed for a block
short	minReplication Minimum copies needed or else write is disallowed
org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks	neededReplications Store set of Blocks that need to be replicated 1 or more times.

Constructor Summary

Constructors

Constructor and Description
BlockManager(Namesystem namesystem, FSClusterStats stats, Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	activate(Configuration conf)
BlockInfo	addBlockCollection(BlockInfo block, BlockCollection bc)
short	adjustReplication(short replication) Clamp the specified replication between the minimum and the maximum replication levels.
boolean	checkMinReplication(Block block)
void	checkReplication(BlockCollection bc) Check replication of the blocks in the collection.
DatanodeDescriptor[]	chooseTarget(String src, int numOfReplicas, DatanodeDescriptor client, HashMap<Node,Node> excludedNodes, long blocksize, List<String> favoredNodes) Choose target datanodes according to the replication policy.
void	clearQueues() Clear all queues that hold decisions previously made by this NameNode.
void	close()
boolean	commitOrCompleteLastBlock(MutableBlockCollection bc, Block commitBlock) Commit the last block of the file and mark it as complete if it has meets the minimum replication requirement
LocatedBlock	convertLastBlockToUnderConstruction(MutableBlockCollection bc) Convert the last block of the file to an under construction block.
NumberReplicas	countNodes(Block b) Return the number of nodes hosting a given block, grouped by the state of those replicas.
LocatedBlocks	createLocatedBlocks(BlockInfo[] blocks, long fileSizeExcludeBlocksUnderConstruction, boolean isFileUnderConstruction, long offset, long length, boolean needBlockToken, boolean inSnapshot) Create a LocatedBlocks.
Iterator<DatanodeDescriptor>	datanodeIterator(Block block)
void	findAndMarkBlockAsCorrupt(ExtendedBlock blk, DatanodeInfo dn, String reason) Mark the block belonging to datanode as corrupt
BlockInfo	forceCompleteBlock(MutableBlockCollection bc, BlockInfoUnderConstruction block) Force the given block in the given file to be marked as complete, regardless of whether enough replicas are present.
DataEncryptionKey	generateDataEncryptionKey()
int	getActiveBlockCount()
BlockCollection	getBlockCollection(Block b)
ExportedBlockKeys	getBlockKeys()
BlockPlacementPolicy	getBlockPlacementPolicy()
BlocksWithLocations	getBlocks(DatanodeID datanode, long size) return a list of blocks & their locations on datanode whose total size is size
BlockTokenSecretManager	getBlockTokenSecretManager() get the BlockTokenSecretManager
int	getCapacity()
long[]	getCorruptReplicaBlockIds(int numExpectedBlocks, Long startingBlockId) Return a range of corrupt replica block ids.
Iterator<Block>	getCorruptReplicaBlockIterator() Return an iterator over the set of blocks for which there are no replicas.
long	getCorruptReplicaBlocksCount() Used by metrics
DatanodeManager	getDatanodeManager()
long	getExcessBlocksCount() Used by metrics
int	getMaxReplicationStreams()
long	getMissingBlocksCount()
DatanodeDescriptor[]	getNodes(BlockInfo block)
int	getPendingDataNodeMessageCount() Used by metrics
long	getPendingDeletionBlocksCount() Used by metrics
long	getPendingReplicationBlocksCount() Used by metrics
long	getPostponedMisreplicatedBlocksCount() Used by metrics
long	getScheduledReplicationBlocksCount() Used by metrics
BlockInfo	getStoredBlock(Block block)
int	getTotalBlocks()
long	getUnderReplicatedBlocksCount() Used by metrics
int	getUnderReplicatedNotMissingBlocks() Return number of under-replicated but not missing blocks
void	metaSave(PrintWriter out) Dump meta data to out.
int	numCorruptReplicas(Block block)
int	numOfUnderReplicatedBlocks()
void	processAllPendingDNMessages() Process any remaining queued datanode messages after entering active state.
void	processIncrementalBlockReport(DatanodeID nodeID, String poolId, ReceivedDeletedBlockInfo[] blockInfos) The given node is reporting incremental information about some blocks.
void	processMisReplicatedBlocks() For each block in the name-node verify whether it belongs to any file, over or under replicated.
void	processQueuedMessagesForBlock(Block b) Try to process any messages that were previously queued for the given block.
void	processReport(DatanodeID nodeID, String poolId, BlockListAsLongs newReport) The given datanode is reporting all its blocks.
void	removeBlock(Block block)
void	removeBlockFromMap(Block block)
void	removeStoredBlock(Block block, DatanodeDescriptor node) Modify (block-->datanode) map.
void	setBlockPlacementPolicy(BlockPlacementPolicy newpolicy) Set BlockPlacementPolicy
void	setBlockPoolId(String blockPoolId)
void	setBlockToken(LocatedBlock b, BlockTokenSecretManager.AccessMode mode) Generate a block token for the located block.
void	setPostponeBlocksFromFuture(boolean postpone)
void	setReplication(short oldRepl, short newRepl, String src, Block... blocks) Set replication for the blocks.
void	shutdown()
void	verifyReplication(String src, short replication, String clientName) Check whether the replication parameter is within the range determined by system configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

blockLog

public static final org.apache.commons.logging.Log blockLog

DEFAULT_MAP_LOAD_FACTOR

public static final float DEFAULT_MAP_LOAD_FACTOR

Default load factor of map

See Also:

Constant Field Values

excessReplicateMap

public final Map<String,LightWeightLinkedSet<Block>> excessReplicateMap

Maps a StorageID to the set of blocks that are "extra" for this DataNode. We'll eventually remove these extras.

neededReplications

public final org.apache.hadoop.hdfs.server.blockmanagement.UnderReplicatedBlocks neededReplications

Store set of Blocks that need to be replicated 1 or more times. We also store pending replication-orders.

maxReplication

public final short maxReplication

The maximum number of replicas allowed for a block

minReplication

public final short minReplication

Minimum copies needed or else write is disallowed

defaultReplication

public final int defaultReplication

Default number of replicas

Constructor Detail

BlockManager

public BlockManager(Namesystem namesystem,
            FSClusterStats stats,
            Configuration conf)
             throws IOException

Throws:

IOException

Method Detail

getPendingReplicationBlocksCount

public long getPendingReplicationBlocksCount()

Used by metrics

getUnderReplicatedBlocksCount

public long getUnderReplicatedBlocksCount()

Used by metrics

getCorruptReplicaBlocksCount

public long getCorruptReplicaBlocksCount()

Used by metrics

getScheduledReplicationBlocksCount

public long getScheduledReplicationBlocksCount()

Used by metrics

getPendingDeletionBlocksCount

public long getPendingDeletionBlocksCount()

Used by metrics

getExcessBlocksCount

public long getExcessBlocksCount()

Used by metrics

getPostponedMisreplicatedBlocksCount

public long getPostponedMisreplicatedBlocksCount()

Used by metrics

getPendingDataNodeMessageCount

public int getPendingDataNodeMessageCount()

Used by metrics

setBlockPoolId

public void setBlockPoolId(String blockPoolId)

getBlockTokenSecretManager

public BlockTokenSecretManager getBlockTokenSecretManager()

get the BlockTokenSecretManager

activate

public void activate(Configuration conf)

close

public void close()

getDatanodeManager

public DatanodeManager getDatanodeManager()

Returns:

the datanodeManager

getBlockPlacementPolicy

public BlockPlacementPolicy getBlockPlacementPolicy()

Returns:

the BlockPlacementPolicy

setBlockPlacementPolicy

public void setBlockPlacementPolicy(BlockPlacementPolicy newpolicy)

Set BlockPlacementPolicy

metaSave

public void metaSave(PrintWriter out)

Dump meta data to out.

getMaxReplicationStreams

public int getMaxReplicationStreams()

Returns:

maxReplicationStreams

checkMinReplication

public boolean checkMinReplication(Block block)

Parameters:

block -

Returns:

true if the block has minimum replicas

commitOrCompleteLastBlock

public boolean commitOrCompleteLastBlock(MutableBlockCollection bc,
                                Block commitBlock)
                                  throws IOException

Commit the last block of the file and mark it as complete if it has meets the minimum replication requirement

Parameters:

bc - block collection

commitBlock - - contains client reported block length and generation

Returns:

true if the last block is changed to committed state.

Throws:

IOException - if the block does not have at least a minimal number of replicas reported from data-nodes.

forceCompleteBlock

public BlockInfo forceCompleteBlock(MutableBlockCollection bc,
                           BlockInfoUnderConstruction block)
                             throws IOException

Force the given block in the given file to be marked as complete, regardless of whether enough replicas are present. This is necessary when tailing edit logs as a Standby.

Throws:

IOException

convertLastBlockToUnderConstruction

public LocatedBlock convertLastBlockToUnderConstruction(MutableBlockCollection bc)
                                                 throws IOException

Convert the last block of the file to an under construction block.

The block is converted only if the file has blocks and the last one is a partial block (its size is less than the preferred block size). The converted block is returned to the client. The client uses the returned block locations to form the data pipeline for this block.
The methods returns null if there is no partial block at the end. The client is supposed to allocate a new block with the next call.

Parameters:

bc - file

Returns:

the last block locations if the block is partial or null otherwise

Throws:

IOException

createLocatedBlocks

public LocatedBlocks createLocatedBlocks(BlockInfo[] blocks,
                                long fileSizeExcludeBlocksUnderConstruction,
                                boolean isFileUnderConstruction,
                                long offset,
                                long length,
                                boolean needBlockToken,
                                boolean inSnapshot)
                                  throws IOException

Create a LocatedBlocks.

Throws:

IOException

getBlockKeys

public ExportedBlockKeys getBlockKeys()

Returns:

current access keys.

setBlockToken

public void setBlockToken(LocatedBlock b,
                 BlockTokenSecretManager.AccessMode mode)
                   throws IOException

Generate a block token for the located block.

Throws:

IOException

generateDataEncryptionKey

public DataEncryptionKey generateDataEncryptionKey()

adjustReplication

public short adjustReplication(short replication)

Clamp the specified replication between the minimum and the maximum replication levels.

verifyReplication

public void verifyReplication(String src,
                     short replication,
                     String clientName)
                       throws IOException

Check whether the replication parameter is within the range determined by system configuration.

Throws:

IOException

getBlocks

public BlocksWithLocations getBlocks(DatanodeID datanode,
                            long size)
                              throws IOException

return a list of blocks & their locations on datanode whose total size is size

Parameters:

datanode - on which blocks are located

size - total size of blocks

Throws:

IOException

findAndMarkBlockAsCorrupt

public void findAndMarkBlockAsCorrupt(ExtendedBlock blk,
                             DatanodeInfo dn,
                             String reason)
                               throws IOException

Mark the block belonging to datanode as corrupt

Parameters:

blk - Block to be marked as corrupt

dn - Datanode which holds the corrupt replica

reason - a textual reason why the block should be marked corrupt, for logging purposes

Throws:

IOException

setPostponeBlocksFromFuture

public void setPostponeBlocksFromFuture(boolean postpone)

getUnderReplicatedNotMissingBlocks

public int getUnderReplicatedNotMissingBlocks()

Return number of under-replicated but not missing blocks

chooseTarget

public DatanodeDescriptor[] chooseTarget(String src,
                                int numOfReplicas,
                                DatanodeDescriptor client,
                                HashMap<Node,Node> excludedNodes,
                                long blocksize,
                                List<String> favoredNodes)
                                  throws IOException

Choose target datanodes according to the replication policy.

Throws:

IOException - if the number of targets < minimum replication.

See Also:

BlockPlacementPolicy.chooseTarget(String, int, DatanodeDescriptor, List, boolean, HashMap, long)

processReport

public void processReport(DatanodeID nodeID,
                 String poolId,
                 BlockListAsLongs newReport)
                   throws IOException

The given datanode is reporting all its blocks. Update the (machine-->blocklist) and (block-->machinelist) maps.

Throws:

IOException

processQueuedMessagesForBlock

public void processQueuedMessagesForBlock(Block b)
                                   throws IOException

Try to process any messages that were previously queued for the given block. This is called from FSEditLogLoader whenever a block's state in the namespace has changed or a new block has been created.

Throws:

IOException

processAllPendingDNMessages

public void processAllPendingDNMessages()
                                 throws IOException

Process any remaining queued datanode messages after entering active state. At this point they will not be re-queued since we are the definitive master node and thus should be up-to-date with the namespace information.

Throws:

IOException

processMisReplicatedBlocks

public void processMisReplicatedBlocks()

For each block in the name-node verify whether it belongs to any file, over or under replicated. Place it into the respective queue.

setReplication

public void setReplication(short oldRepl,
                  short newRepl,
                  String src,
                  Block... blocks)

Set replication for the blocks.

removeStoredBlock

public void removeStoredBlock(Block block,
                     DatanodeDescriptor node)

Modify (block-->datanode) map. Possibly generate replication tasks, if the removed block is still valid.

processIncrementalBlockReport

public void processIncrementalBlockReport(DatanodeID nodeID,
                                 String poolId,
                                 ReceivedDeletedBlockInfo[] blockInfos)
                                   throws IOException

The given node is reporting incremental information about some blocks. This includes blocks that are starting to be received, completed being received, or deleted. This method must be called with FSNamesystem lock held.

Throws:

IOException

countNodes

public NumberReplicas countNodes(Block b)

Return the number of nodes hosting a given block, grouped by the state of those replicas.

getActiveBlockCount

public int getActiveBlockCount()

getNodes

public DatanodeDescriptor[] getNodes(BlockInfo block)

getTotalBlocks

public int getTotalBlocks()

removeBlock

public void removeBlock(Block block)

getStoredBlock

public BlockInfo getStoredBlock(Block block)

checkReplication

public void checkReplication(BlockCollection bc)

Check replication of the blocks in the collection. If any block is needed replication, insert it into the replication queue. Otherwise, if the block is more than the expected replication factor, process it as an over replicated block.

getMissingBlocksCount

public long getMissingBlocksCount()

addBlockCollection

public BlockInfo addBlockCollection(BlockInfo block,
                           BlockCollection bc)

getBlockCollection

public BlockCollection getBlockCollection(Block b)

datanodeIterator

public Iterator<DatanodeDescriptor> datanodeIterator(Block block)

Returns:

an iterator of the datanodes.

numCorruptReplicas

public int numCorruptReplicas(Block block)

removeBlockFromMap

public void removeBlockFromMap(Block block)

getCapacity

public int getCapacity()

getCorruptReplicaBlockIds

public long[] getCorruptReplicaBlockIds(int numExpectedBlocks,
                               Long startingBlockId)

Return a range of corrupt replica block ids. Up to numExpectedBlocks blocks starting at the next block after startingBlockId are returned (fewer if numExpectedBlocks blocks are unavailable). If startingBlockId is null, up to numExpectedBlocks blocks are returned from the beginning. If startingBlockId cannot be found, null is returned.

Parameters:

numExpectedBlocks - Number of block ids to return. 0 <= numExpectedBlocks <= 100

startingBlockId - Block id from which to start. If null, start at beginning.

Returns:

Up to numExpectedBlocks blocks from startingBlockId if it exists

getCorruptReplicaBlockIterator

public Iterator<Block> getCorruptReplicaBlockIterator()

Return an iterator over the set of blocks for which there are no replicas.

numOfUnderReplicatedBlocks

public int numOfUnderReplicatedBlocks()

Returns:

the size of UnderReplicatedBlocks

clearQueues

public void clearQueues()

Clear all queues that hold decisions previously made by this NameNode.

shutdown

public void shutdown()