Fusion Configurations

Configuration classes for combining multiple input modalities.

Base Fusion Configuration

class eir.setup.schemas.FusionConfig(
model_type: Literal['mlp-residual', 'mlp-residual-sum', 'identity', 'mgmoe', 'pass-through', 'attention'],
model_config: ResidualMLPConfig | IdentityConfig | MGMoEModelConfig | AttentionFusionConfig,
tensor_broker_config: None | TensorBrokerConfig = None,
)
Parameters:

  • model_type – Which type of fusion model to use.

  • model_config – Fusion model configuration.

Fusion Module Configurations

class eir.models.fusion.fusion_default.ResidualMLPConfig(
layers: list[int] = <factory>,
fc_task_dim: int = 256,
rb_do: float = 0.1,
fc_do: float = 0.1,
stochastic_depth_p: float = 0.1,
)
Parameters:

  • layers – Number of residual MLP layers to use in for each output predictor after fusing.

  • fc_task_dim – Number of hidden nodes in each MLP residual block.

  • rb_do – Dropout in each MLP residual block.

  • fc_do – Dropout before final layer.

  • stochastic_depth_p – Probability of dropping input.

class eir.models.fusion.fusion_mgmoe.MGMoEModelConfig(
layers: Sequence[int] = <factory>,
fc_task_dim: int = 64,
mg_num_experts: int = 8,
rb_do: float = 0.0,
fc_do: float = 0.0,
stochastic_depth_p: float = 0.0,
)

Note that this module by default uses sum fusion with input projection.

Parameters:

  • layers – A sequence of two int values controlling the number of residual MLP blocks in the network. The first item (i.e. layers[0]) refers to the number of blocks in the expert branches. The second item (i.e. layers[1]) refers to the number of blocks in the predictor branches.

  • fc_task_dim – Number of hidden nodes in all residual blocks (both expert and predictor) of the network.

  • mg_num_experts – Number of multi gate experts to use.

  • rb_do – Dropout in all MLP residual blocks (both expert and predictor).

  • fc_do – Dropout before the last FC layer.

  • stochastic_depth_p – Probability of dropping input.

class eir.models.fusion.fusion_identity.IdentityConfig
class eir.models.fusion.fusion_attention.AttentionFusionConfig(
n_layers: int = 2,
common_embedding_dim: int = 512,
n_heads: int = 8,
dim_feedforward: int | Literal['auto'] = 'auto',
dropout: float = 0.1,
attention_pooling: bool = True,
)