This project provides custom FTS5 tokenizers for SQLite that use the International Components for Unicode (ICU) library to provide robust word segmentation for various languages. The project supports ...

We present RobusTok, a new image tokenizer with a two-stage training scheme: Main training → constructs a robust latent space. Post-training → aligns the generator’s latent distribution with its image ...