R: Byte Pair Encoding Text Tokenization

# Byte Pair Encoding Text Tokenization

## Documentation for package ‘tokenizers.bpe’ version 0.1.6

- DESCRIPTION file
- Package NEWS
- Package README

## Help Pages

belgium_parliament | Dataset from 2017 with Questions asked in the Belgium Federal Parliament
bpe | Construct a Byte Pair Encoding model
bpe_decode | Decode Byte Pair Encoding sequences to text
bpe_encode | Tokenise text alongside a Byte Pair Encoding model
bpe_load_model | Load a Byte Pair Encoding model