org.elasticsearch.repositories.blobstore

Class BlobStoreRepository

java.lang.Object

org.elasticsearch.common.component.AbstractComponent

org.elasticsearch.common.component.AbstractLifecycleComponent<Repository>

org.elasticsearch.repositories.blobstore.BlobStoreRepository

All Implemented Interfaces:

AutoCloseable, LifecycleComponent<Repository>, Releasable, BlobStoreIndexShardRepository.RateLimiterListener, Repository

Direct Known Subclasses:

FsRepository, URLRepository

public abstract class BlobStoreRepository
extends AbstractLifecycleComponent<Repository>
implements Repository, BlobStoreIndexShardRepository.RateLimiterListener

BlobStore - based implementation of Snapshot Repository

This repository works with any BlobStore implementation. The blobStore should be initialized in the derived class before doStart() is called.

BlobStoreRepository maintains the following structure in the blob store

 
   STORE_ROOT
   |- index             - list of all snapshot name as JSON array
   |- snapshot-20131010 - JSON serialized BlobStoreSnapshot for snapshot "20131010"
   |- metadata-20131010 - JSON serialized MetaData for snapshot "20131010" (includes only global metadata)
   |- snapshot-20131011 - JSON serialized BlobStoreSnapshot for snapshot "20131011"
   |- metadata-20131011 - JSON serialized MetaData for snapshot "20131011"
   .....
   |- indices/ - data for all indices
      |- foo/ - data for index "foo"
      |  |- snapshot-20131010 - JSON Serialized IndexMetaData for index "foo"
      |  |- 0/ - data for shard "0" of index "foo"
      |  |  |- __1 \
      |  |  |- __2 |
      |  |  |- __3 |- files from different segments see snapshot-* for their mappings to real segment files
      |  |  |- __4 |
      |  |  |- __5 /
      |  |  .....
      |  |  |- snapshot-20131010 - JSON serialized BlobStoreIndexShardSnapshot for snapshot "20131010"
      |  |  |- snapshot-20131011 - JSON serialized BlobStoreIndexShardSnapshot for snapshot "20131011"
      |  |
      |  |- 1/ - data for shard "1" of index "foo"
      |  |  |- __1
      |  |  .....
      |  |
      |  |-2/
      |  ......
      |
      |- bar/ - data for index bar
      ......
 
 

Field Summary

Fields

Modifier and Type	Field and Description
protected String	repositoryName

Fields inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent

lifecycle

Fields inherited from class org.elasticsearch.common.component.AbstractComponent

componentSettings, logger, settings

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BlobStoreRepository(String repositoryName, RepositorySettings repositorySettings, IndexShardRepository indexShardRepository) Constructs new BlobStoreRepository

Method Summary

Modifier and Type	Method and Description
protected abstract BlobPath	basePath() Returns base path of the repository
protected abstract BlobStore	blobStore() Returns initialized and ready to use BlobStore This method is first called in the doStart() method.
protected ByteSizeValue	chunkSize() Returns data file chunk size.
void	deleteSnapshot(SnapshotId snapshotId) Deletes snapshot
protected void	doClose()
protected void	doStart()
protected void	doStop()
void	endVerification(String seed) Called at the end of repository verification process.
Snapshot	finalizeSnapshot(SnapshotId snapshotId, String failure, int totalShards, com.google.common.collect.ImmutableList<SnapshotShardFailure> shardFailures) Finalizes snapshotting process This method is called on master after all shards are snapshotted.
void	initializeSnapshot(SnapshotId snapshotId, com.google.common.collect.ImmutableList<String> indices, MetaData metaData) Starts snapshotting process
protected boolean	isCompress() Returns true if metadata and snapshot files should be compressed
void	onRestorePause(long nanos)
void	onSnapshotPause(long nanos)
Snapshot	readSnapshot(SnapshotId snapshotId) Reads snapshot description from repository.
protected com.google.common.collect.ImmutableList<SnapshotId>	readSnapshotList() Reads snapshot index file This file can be used by read-only repositories that are unable to list files in the repository
MetaData	readSnapshotMetaData(SnapshotId snapshotId, com.google.common.collect.ImmutableList<String> indices) Returns global metadata associate with the snapshot.
long	restoreThrottleTimeInNanos() Returns restore throttle time in nanoseconds
com.google.common.collect.ImmutableList<SnapshotId>	snapshots() Returns the list of snapshots currently stored in the repository
long	snapshotThrottleTimeInNanos() Returns snapshot throttle time in nanoseconds
String	startVerification() Verifies repository on the master node and returns the verification token.
static String	testBlobPrefix(String seed)
protected void	writeSnapshotList(com.google.common.collect.ImmutableList<SnapshotId> snapshots) Writes snapshot index file This file can be used by read-only repositories that are unable to list files in the repository

Methods inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent

addLifecycleListener, close, lifecycleState, removeLifecycleListener, start, stop

Methods inherited from class org.elasticsearch.common.component.AbstractComponent

nodeName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.common.component.LifecycleComponent

addLifecycleListener, lifecycleState, removeLifecycleListener, start, stop

Methods inherited from interface org.elasticsearch.common.lease.Releasable

close

Field Detail

repositoryName

protected final String repositoryName

Constructor Detail

BlobStoreRepository

protected BlobStoreRepository(String repositoryName,
                              RepositorySettings repositorySettings,
                              IndexShardRepository indexShardRepository)

Constructs new BlobStoreRepository

Parameters:

repositoryName - repository name

repositorySettings - repository settings

indexShardRepository - an instance of IndexShardRepository

Method Detail

doStart

protected void doStart()
                throws ElasticsearchException

Specified by:

doStart in class AbstractLifecycleComponent<Repository>

Throws:

ElasticsearchException

doStop

protected void doStop()
               throws ElasticsearchException

Specified by:

doStop in class AbstractLifecycleComponent<Repository>

Throws:

ElasticsearchException

doClose

protected void doClose()
                throws ElasticsearchException

Specified by:

doClose in class AbstractLifecycleComponent<Repository>

Throws:

ElasticsearchException

blobStore

protected abstract BlobStore blobStore()

Returns initialized and ready to use BlobStore

This method is first called in the doStart() method.

Returns:

blob store

basePath

protected abstract BlobPath basePath()

Returns base path of the repository

isCompress

protected boolean isCompress()

Returns true if metadata and snapshot files should be compressed

Returns:

true if compression is needed

chunkSize

protected ByteSizeValue chunkSize()

Returns data file chunk size.

This method should return null if no chunking is needed.

Returns:

chunk size

initializeSnapshot

public void initializeSnapshot(SnapshotId snapshotId,
                               com.google.common.collect.ImmutableList<String> indices,
                               MetaData metaData)

Starts snapshotting process

Specified by:

initializeSnapshot in interface Repository

Parameters:

snapshotId - snapshot id

indices - list of indices to be snapshotted

metaData - cluster metadata

deleteSnapshot

public void deleteSnapshot(SnapshotId snapshotId)

Deletes snapshot

Specified by:

deleteSnapshot in interface Repository

Parameters:

snapshotId - snapshot id

finalizeSnapshot

public Snapshot finalizeSnapshot(SnapshotId snapshotId,
                                 String failure,
                                 int totalShards,
                                 com.google.common.collect.ImmutableList<SnapshotShardFailure> shardFailures)

Finalizes snapshotting process

This method is called on master after all shards are snapshotted.

Specified by:

finalizeSnapshot in interface Repository

Parameters:

snapshotId - snapshot id

failure - global failure reason or null

totalShards - total number of shards

shardFailures - list of shard failures

Returns:

snapshot description

snapshots

public com.google.common.collect.ImmutableList<SnapshotId> snapshots()

Returns the list of snapshots currently stored in the repository

Specified by:

snapshots in interface Repository

Returns:

snapshot list

readSnapshotMetaData

public MetaData readSnapshotMetaData(SnapshotId snapshotId,
                                     com.google.common.collect.ImmutableList<String> indices)

Returns global metadata associate with the snapshot.

The returned meta data contains global metadata as well as metadata for all indices listed in the indices parameter.

Specified by:

readSnapshotMetaData in interface Repository

Parameters:

snapshotId - snapshot ID

indices - list of indices

Returns:

information about snapshot

readSnapshot

public Snapshot readSnapshot(SnapshotId snapshotId)

Reads snapshot description from repository.

Specified by:

readSnapshot in interface Repository

Parameters:

snapshotId - snapshot ID

Returns:

information about snapshot

writeSnapshotList

protected void writeSnapshotList(com.google.common.collect.ImmutableList<SnapshotId> snapshots)
                          throws IOException

Writes snapshot index file

This file can be used by read-only repositories that are unable to list files in the repository

Parameters:

snapshots - list of snapshot ids

Throws:

IOException - I/O errors

readSnapshotList

protected com.google.common.collect.ImmutableList<SnapshotId> readSnapshotList()
                                                                        throws IOException

Reads snapshot index file

This file can be used by read-only repositories that are unable to list files in the repository

Returns:

list of snapshots in the repository

Throws:

IOException - I/O errors

onRestorePause

public void onRestorePause(long nanos)

Specified by:

onRestorePause in interface BlobStoreIndexShardRepository.RateLimiterListener

onSnapshotPause

public void onSnapshotPause(long nanos)

Specified by:

onSnapshotPause in interface BlobStoreIndexShardRepository.RateLimiterListener

snapshotThrottleTimeInNanos

public long snapshotThrottleTimeInNanos()

Description copied from interface: Repository

Returns snapshot throttle time in nanoseconds

Specified by:

snapshotThrottleTimeInNanos in interface Repository

restoreThrottleTimeInNanos

public long restoreThrottleTimeInNanos()

Description copied from interface: Repository

Returns restore throttle time in nanoseconds

Specified by:

restoreThrottleTimeInNanos in interface Repository

startVerification

public String startVerification()

Description copied from interface: Repository

Verifies repository on the master node and returns the verification token. If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required

Specified by:

startVerification in interface Repository

Returns:

verification token that should be passed to all Index Shard Repositories for additional verification or null

endVerification

public void endVerification(String seed)

Description copied from interface: Repository

Called at the end of repository verification process. This method should perform all necessary cleanup of the temporary files created in the repository

Specified by:

endVerification in interface Repository

Parameters:

seed - verification request generated by Repository.startVerification() command

testBlobPrefix

public static String testBlobPrefix(String seed)