Package org.yamcs.yarch.rocksdb

Class Tablespace

java.lang.Object
org.yamcs.yarch.rocksdb.Tablespace
public class Tablespace extends Object
Tablespaces are used to store data by the RdbStorageEngine. Each tablespace can store data from one or more Yamcs instances.

Tablespaces are rocksdb databases normally stored in the yamcs data directory (/storage/yamcs-data). Each corresponds to a directory <tablespace-name>.rdb and has a definition file tablespace-name.tbs.

Tablespaces can also have time based partitions in different RocksDB databases in sub-directories such as <tablespace-name>.rdb/YYYY/

There are four column families in the main database:

  • _metadata_ - contains metadata - tables definition, tbsIndices,...
  • rt_data - stores data for tm, pp and events tables.
  • parameter_archive - stores data for parameter archive
  • the default column family - contains other data (alarms, activities, timeline...)

The data is partitioned by the first 4 bytes of the key which we call tbsIndex.

One tbsIndex corresponds to a so called tablespace record. For example tbsIndex=5 can correspond to all telemetry packets for packet XYZ.

Except for the 4 bytes tbsIndex, the rest of the key and value are completely dependent on the data type. For example for yarch table data, the rest of key following the 4 bytes tbsIndex represents the key of the row in the table.

The metadata contains three types of records, identified by the first byte of the key:

  • key: 0x01

    value: 1 byte version number (0x1), 4 bytes max tbsIndex

    used to store the max tbsIndex and also stores a version number in case the format will change in the future

  • key: 0x02, 1 byte record type, 4 bytes tbsIndex

    value: protobuf encoded TablespaceRecord

    Used to store the information corresponding to the given tbsIndex. The record type corresponds to the Type enumerations from tablespace.proto

  • key: 0x03, 1 byte record type, sequence name encoded in UTF8

    value: last sequence number 8 bytes big endian

  • Field Details

  • Constructor Details

    • Tablespace

      public Tablespace(String name)

  • Method Details

    • loadDb

      public void loadDb(boolean readonly) throws IOException
      Throws:
      IOException

    • getName

      public String getName()

    • getTablePartitions

      public List<Tablespace.TablespaceRecord> getTablePartitions(String instanceName, String tableName) throws org.rocksdb.RocksDBException, IOException
      Returns a list of all records of type TABLE_PARTITION for a given instance and table If instanceName = tablespace name, it returns also records which do not have an instanceName specified.
      Throws:
      org.rocksdb.RocksDBException
      IOException

    • getTableHistograms

      public List<Tablespace.TablespaceRecord> getTableHistograms(String instanceName, String tableName) throws org.rocksdb.RocksDBException, IOException
      Returns a list of all records of type HISTOGRAM for a given instance and table If instanceName = tablespace name, it returns also records which do not have an instanceName specified.
      Throws:
      org.rocksdb.RocksDBException
      IOException

    • filter

      public List<Tablespace.TablespaceRecord> filter(Tablespace.TablespaceRecord.Type type, String instanceName, Predicate<Tablespace.TablespaceRecord.Builder> p) throws YarchException, DatabaseCorruptionException
      Throws:
      YarchException
      DatabaseCorruptionException

    • createMetadataRecord

      public Tablespace.TablespaceRecord createMetadataRecord(String yamcsInstance, Tablespace.TablespaceRecord.Builder trb) throws org.rocksdb.RocksDBException
      Creates a new tablespace record and adds it to the metadata database
      Parameters:
      trb - the builder has to have all fields set except for the tbsIndex which will be assigned by this method
      Returns:
      a fully built
      Throws:
      org.rocksdb.RocksDBException

    • updateRecord

      public Tablespace.TablespaceRecord updateRecord(String yamcsInstance, Tablespace.TablespaceRecord.Builder trb) throws org.rocksdb.RocksDBException
      Throws:
      org.rocksdb.RocksDBException

    • getCustomDataDir

      public String getCustomDataDir()

    • getRdb

      public YRDB getRdb(String partitionDir, boolean readOnly)
      (Creates) and returns a database in the given partition directory. If the directory is null, return then main tablespace db
      Parameters:
      partitionDir -
      readOnly -

    • getRdb

      public YRDB getRdb(String relativePath)

    • getRdb

      public YRDB getRdb()
      Get the main database of the tablespace

    • dispose

      public void dispose(YRDB rdb)

    • setCustomDataDir

      public void setCustomDataDir(String dataDir)

    • getDataDir

      public String getDataDir()

    • removeTbsIndex

      public void removeTbsIndex(Tablespace.TablespaceRecord.Type type, int tbsIndex) throws org.rocksdb.RocksDBException
      Removes the tbsIndex from the metadata and all the associated data from the main db (data might still be present in the partitions)
      Parameters:
      type -
      tbsIndex -
      Throws:
      org.rocksdb.RocksDBException

    • removeTbsIndices

      public void removeTbsIndices(Tablespace.TablespaceRecord.Type type, IntArray tbsIndexArray) throws org.rocksdb.RocksDBException
      Removes the tbs indices with ALL the associated data from the main db (data might still be present in the partitions)
      Parameters:
      type -
      tbsIndexArray -
      Throws:
      org.rocksdb.RocksDBException

    • removeMetadataRecords

      public void removeMetadataRecords(Tablespace.TablespaceRecord.Type type) throws org.rocksdb.RocksDBException
      Removes all metadata records of a given type
      Parameters:
      type -
      Throws:
      org.rocksdb.RocksDBException

    • getRdbFactory

      public RDBFactory getRdbFactory()

    • putData

      public void putData(byte[] key, byte[] value) throws org.rocksdb.RocksDBException
      inserts data into the main partition
      Parameters:
      key -
      value -
      Throws:
      org.rocksdb.RocksDBException

    • getData

      public byte[] getData(byte[] key) throws org.rocksdb.RocksDBException
      Throws:
      org.rocksdb.RocksDBException

    • remove

      public void remove(byte[] key) throws org.rocksdb.RocksDBException
      Throws:
      org.rocksdb.RocksDBException

    • createTable

      public void createTable(String yamcsInstance, TableDefinition tblDef) throws YarchException, org.rocksdb.RocksDBException
      Throws:
      YarchException
      org.rocksdb.RocksDBException

    • getTable

      public RdbTable getTable(TableDefinition tblDef)
      returns the table associated to this definition or null if this table is not known.
      Parameters:
      tblDef -
      Returns:

    • newTableWalker

      public TableWalker newTableWalker(ExecutionContext ctx, TableDefinition tblDef, boolean ascending, boolean follow)

    • newTableWriter

      public RdbTableWriter newTableWriter(YarchDatabaseInstance ydb, TableDefinition tblDef, TableWriter.InsertMode insertMode)

    • close

      public void close()

    • getSequence

      public Sequence getSequence(String name, boolean create) throws YarchException, org.rocksdb.RocksDBException
      Throws:
      YarchException
      org.rocksdb.RocksDBException

    • renameTable

      public void renameTable(String yamcsInstance, TableDefinition tblDef, String newName) throws org.rocksdb.RocksDBException
      Throws:
      org.rocksdb.RocksDBException

    • newSecondaryIndexTableWalker

      public TableWalker newSecondaryIndexTableWalker(YarchDatabaseInstance ydb, TableDefinition tblDef, boolean ascending, boolean follow)

    • getSequencesInfo

      public List<SequenceInfo> getSequencesInfo()