marius_config

marius.tools.configuration.marius_config.get_model_dir_path(dataset_dir)
class marius.tools.configuration.marius_config.NeighborSamplingConfig(type='ALL', options=NeighborSamplingOptions(), use_incoming_nbrs=True, use_outgoing_nbrs=True, use_hashmap_sets=True)
Parameters
Return type

None

type: str = 'ALL'
options: marius.tools.configuration.datatypes.NeighborSamplingOptions = NeighborSamplingOptions()
use_incoming_nbrs: bool = True
use_outgoing_nbrs: bool = True
use_hashmap_sets: bool = True
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0))
Parameters
Return type

None

type: str = 'ADAGRAD'
options: marius.tools.configuration.datatypes.OptimizerOptions = AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.InitConfig(type='GLOROT_UNIFORM', options=InitOptions())
Parameters
Return type

None

type: str = 'GLOROT_UNIFORM'
options: marius.tools.configuration.datatypes.InitOptions = InitOptions()
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.LossConfig(type='SOFTMAX_CE', options=LossOptions(reduction='SUM'))
Parameters
Return type

None

type: str = 'SOFTMAX_CE'
options: marius.tools.configuration.datatypes.LossOptions = LossOptions(reduction='SUM')
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.LayerConfig(type=None, options=LayerOptions(), input_dim=- 1, output_dim=- 1, init=InitConfig(type='GLOROT_UNIFORM', options=InitOptions()), optimizer=OptimizerConfig(type='DEFAULT', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)), bias=False, bias_init=InitConfig(type='ZEROS', options=InitOptions()), activation='NONE')
Parameters
Return type

None

type: str = None
options: marius.tools.configuration.datatypes.LayerOptions = LayerOptions()
input_dim: int = -1
output_dim: int = -1
init: marius.tools.configuration.marius_config.InitConfig = InitConfig(type='GLOROT_UNIFORM', options=InitOptions())
optimizer: marius.tools.configuration.marius_config.OptimizerConfig = OptimizerConfig(type='DEFAULT', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0))
bias: bool = False
bias_init: marius.tools.configuration.marius_config.InitConfig = InitConfig(type='ZEROS', options=InitOptions())
activation: str = 'NONE'
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.EncoderConfig(layers=<factory>, train_neighbor_sampling=<factory>, eval_neighbor_sampling=<factory>, embedding_dim=-1)
Parameters
Return type

None

layers: List[List[marius.tools.configuration.marius_config.LayerConfig]]
train_neighbor_sampling: List[marius.tools.configuration.marius_config.NeighborSamplingConfig]
eval_neighbor_sampling: List[marius.tools.configuration.marius_config.NeighborSamplingConfig]
embedding_dim: int = -1
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.DecoderConfig(type='DISTMULT', options=DecoderOptions(), optimizer=OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)))
Parameters
Return type

None

type: str = 'DISTMULT'
options: marius.tools.configuration.datatypes.DecoderOptions = DecoderOptions()
optimizer: marius.tools.configuration.marius_config.OptimizerConfig = OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0))
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.ModelConfig(random_seed='???', learning_task='???', encoder='???', decoder='???', loss='???', dense_optimizer=OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)), sparse_optimizer=OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)))
Parameters
Return type

None

random_seed: int = '???'
learning_task: str = '???'
encoder: marius.tools.configuration.marius_config.EncoderConfig = '???'
decoder: marius.tools.configuration.marius_config.DecoderConfig = '???'
loss: marius.tools.configuration.marius_config.LossConfig = '???'
dense_optimizer: marius.tools.configuration.marius_config.OptimizerConfig = OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0))
sparse_optimizer: marius.tools.configuration.marius_config.OptimizerConfig = OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0))
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return:

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float'))
Parameters
Return type

None

type: str = 'DEVICE_MEMORY'
options: marius.tools.configuration.datatypes.StorageOptions = StorageOptions(dtype='float')
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.DatasetConfig(dataset_dir='???', num_edges='???', num_nodes='???', num_relations=1, num_train='???', num_valid=- 1, num_test=- 1, node_feature_dim=- 1, rel_feature_dim=- 1, num_classes=- 1, initialized=False)
Parameters

  • dataset_dir (str) –

  • num_edges (int) –

  • num_nodes (int) –

  • num_relations (int) –

  • num_train (int) –

  • num_valid (int) –

  • num_test (int) –

  • node_feature_dim (int) –

  • rel_feature_dim (int) –

  • num_classes (int) –

  • initialized (bool) –

Return type

None

dataset_dir: str = '???'
num_edges: int = '???'
num_nodes: int = '???'
num_relations: int = 1
num_train: int = '???'
num_valid: int = -1
num_test: int = -1
node_feature_dim: int = -1
rel_feature_dim: int = -1
num_classes: int = -1
initialized: bool = False
populate_dataset_stats()
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.StorageConfig(device_type='cpu', device_ids=<factory>, dataset=DatasetConfig(dataset_dir='???', num_edges='???', num_nodes='???', num_relations=1, num_train='???', num_valid=-1, num_test=-1, node_feature_dim=-1, rel_feature_dim=-1, num_classes=-1, initialized=False), edges=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int')), nodes=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int')), embeddings=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float')), features=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float')), prefetch=True, shuffle_input=True, full_graph_evaluation=True, export_encoded_nodes=False, model_dir='???', log_level='info')
Parameters
Return type

None

device_type: str = 'cpu'
device_ids: List[int]
dataset: marius.tools.configuration.marius_config.DatasetConfig = DatasetConfig(dataset_dir='???', num_edges='???', num_nodes='???', num_relations=1, num_train='???', num_valid=-1, num_test=-1, node_feature_dim=-1, rel_feature_dim=-1, num_classes=-1, initialized=False)
edges: marius.tools.configuration.marius_config.StorageBackendConfig = StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int'))
nodes: marius.tools.configuration.marius_config.StorageBackendConfig = StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int'))
embeddings: marius.tools.configuration.marius_config.StorageBackendConfig = StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float'))
features: marius.tools.configuration.marius_config.StorageBackendConfig = StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float'))
prefetch: bool = True
shuffle_input: bool = True
full_graph_evaluation: bool = True
export_encoded_nodes: bool = False
model_dir: str = '???'
log_level: str = 'info'
SUPPORTED_EMBEDDING_BACKENDS = ['PARTITION_BUFFER', 'DEVICE_MEMORY', 'HOST_MEMORY']
SUPPORTED_EDGE_BACKENDS = ['FLAT_FILE', 'DEVICE_MEMORY', 'HOST_MEMORY']
SUPPORTED_NODE_BACKENDS = ['DEVICE_MEMORY', 'HOST_MEMORY']
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.NegativeSamplingConfig(num_chunks=1, negatives_per_positive=1000, degree_fraction=0, filtered=False, local_filter_mode='DEG')
Parameters

  • num_chunks (int) –

  • negatives_per_positive (int) –

  • degree_fraction (float) –

  • filtered (bool) –

  • local_filter_mode (str) –

Return type

None

num_chunks: int = 1
negatives_per_positive: int = 1000
degree_fraction: float = 0
filtered: bool = False
local_filter_mode: str = 'DEG'
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.CheckpointConfig(save_best=False, interval=- 1, save_state=False)
Parameters

  • save_best (bool) –

  • interval (int) –

  • save_state (bool) –

Return type

None

save_best: bool = False
interval: int = -1
save_state: bool = False
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4)
Parameters

  • sync (bool) –

  • gpu_sync_interval (int) –

  • gpu_model_average (bool) –

  • staleness_bound (int) –

  • batch_host_queue_size (int) –

  • batch_device_queue_size (int) –

  • gradients_device_queue_size (int) –

  • gradients_host_queue_size (int) –

  • batch_loader_threads (int) –

  • batch_transfer_threads (int) –

  • compute_threads (int) –

  • gradient_transfer_threads (int) –

  • gradient_update_threads (int) –

Return type

None

sync: bool = True
gpu_sync_interval: int = 16
gpu_model_average: bool = True
staleness_bound: int = 16
batch_host_queue_size: int = 4
batch_device_queue_size: int = 4
gradients_device_queue_size: int = 4
gradients_host_queue_size: int = 4
batch_loader_threads: int = 4
batch_transfer_threads: int = 2
compute_threads: int = 1
gradient_transfer_threads: int = 2
gradient_update_threads: int = 4
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.TrainingConfig(batch_size=1000, negative_sampling='???', num_epochs=10, pipeline=PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4), epochs_per_shuffle=1, logs_per_epoch=10, save_model=True, checkpoint=CheckpointConfig(save_best=False, interval=- 1, save_state=False), resume_training=False, resume_from_checkpoint='')
Parameters
Return type

None

batch_size: int = 1000
negative_sampling: marius.tools.configuration.marius_config.NegativeSamplingConfig = '???'
num_epochs: int = 10
pipeline: marius.tools.configuration.marius_config.PipelineConfig = PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4)
epochs_per_shuffle: int = 1
logs_per_epoch: int = 10
save_model: bool = True
checkpoint: marius.tools.configuration.marius_config.CheckpointConfig = CheckpointConfig(save_best=False, interval=-1, save_state=False)
resume_training: bool = False
resume_from_checkpoint: str = ''
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.EvaluationConfig(batch_size=1000, negative_sampling='???', pipeline=PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4), epochs_per_eval=1, checkpoint_dir='')
Parameters
Return type

None

batch_size: int = 1000
negative_sampling: marius.tools.configuration.marius_config.NegativeSamplingConfig = '???'
pipeline: marius.tools.configuration.marius_config.PipelineConfig = PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4)
epochs_per_eval: int = 1
checkpoint_dir: str = ''
merge(input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters

input_config (omegaconf.dictconfig.DictConfig) –

class marius.tools.configuration.marius_config.MariusConfig
model: marius.tools.configuration.marius_config.ModelConfig = ModelConfig(random_seed=1062056610952888119, learning_task='???', encoder='???', decoder='???', loss='???', dense_optimizer=OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)), sparse_optimizer=OptimizerConfig(type='ADAGRAD', options=AdagradOptions(learning_rate=0.1, eps=1e-10, init_value=0, lr_decay=0, weight_decay=0)))
storage: marius.tools.configuration.marius_config.StorageConfig = StorageConfig(device_type='cpu', device_ids=[], dataset=DatasetConfig(dataset_dir='???', num_edges='???', num_nodes='???', num_relations=1, num_train='???', num_valid=-1, num_test=-1, node_feature_dim=-1, rel_feature_dim=-1, num_classes=-1, initialized=False), edges=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int')), nodes=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='int')), embeddings=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float')), features=StorageBackendConfig(type='DEVICE_MEMORY', options=StorageOptions(dtype='float')), prefetch=True, shuffle_input=True, full_graph_evaluation=True, export_encoded_nodes=False, model_dir='???', log_level='info')
training: marius.tools.configuration.marius_config.TrainingConfig = TrainingConfig(batch_size=1000, negative_sampling='???', num_epochs=10, pipeline=PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4), epochs_per_shuffle=1, logs_per_epoch=10, save_model=True, checkpoint=CheckpointConfig(save_best=False, interval=-1, save_state=False), resume_training=False, resume_from_checkpoint='')
evaluation: marius.tools.configuration.marius_config.EvaluationConfig = EvaluationConfig(batch_size=1000, negative_sampling='???', pipeline=PipelineConfig(sync=True, gpu_sync_interval=16, gpu_model_average=True, staleness_bound=16, batch_host_queue_size=4, batch_device_queue_size=4, gradients_device_queue_size=4, gradients_host_queue_size=4, batch_loader_threads=4, batch_transfer_threads=2, compute_threads=1, gradient_transfer_threads=2, gradient_update_threads=4), epochs_per_eval=1, checkpoint_dir='')
marius.tools.configuration.marius_config.type_safe_merge(base_config, input_config)

Merges under specified dictionary config into the current configuration object :type input_config: DictConfig :param input_config: The input configuration dictionary :return: Structured output config

Parameters
marius.tools.configuration.marius_config.initialize_model_dir(output_config)
marius.tools.configuration.marius_config.infer_model_dir(output_config)
marius.tools.configuration.marius_config.load_config(input_config_path, save=False)

This function loads an input user specified configuration file and creates a full configuration file with all defaults set based on the input :param input_config_path: path to the input configuration file :save If true, the full configuration file will be saved to <dir_of_input_config>/full_config.yaml :return: config dict object