iamtatsuki05/MIREI
MIREI is a research workspace that builds encoder/decoder text-embedding models under matched conditions, tracks shared training pipelines, and benchmarks their performance differences.
What's novel
MIREI is a research workspace that builds encoder/decoder text-embedding models under matched conditions, tracks shared training pipelines, and benchmarks their performance differences.
Code Analysis
8 files read · 3 roundsA research workspace that converts decoder-only LLMs (Llama, Mistral, Qwen2) into bidirectional sentence-embedding encoders, with full training pipelines (CLM/MLM/MNTP pre-training, sentence-transformer fine-tuning), dataset cleaning/splitting, tokenizer training, and benchmarking infrastructure for
Strengths
Genuinely non-trivial implementation of bidirectional attention masking for three different LLM architectures, sophisticated document-boundary-aware sequence packing with FlashAttention varlen support, and a rich analytical metrics suite (CKA, effective rank, kNN overlap, attention entropy) that goes well beyond standard evaluation. The packing module in particular shows deep understanding of how different attention implementations handle packed sequences.
Weaknesses
Significant code duplication: the SentenceEncoderMixin (~200 lines) and bidirectional mask logic are copy-pasted verbatim across all three model files (Llama, Mistral, Qwen2) with comments acknowledging the duplication. The package name 'constract_llm' is a typo of 'construct_llm', and the test suite (28 files) appears thin relative to the 90 source files, suggesting limited edge-case coverage for the complex packing and attention mask logic.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
140
Contributors
1
Files
215
Active weeks
9
Repository
Language
Python
Stars
1
Forks
0
License
—