Corpus-Dependent Subcharacter Encoding via HMM -Guided Code Assignment
Tatsuya Hiraoka
摘要
We propose a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes for characters directly from text, which we refer to as Latom (Learned Atom-based Encoding).We instantiate this framework by training an HMM on N-repeated character sequences to estimate “atom” poster...