API Usage

High Level API

This is an example of using ailia Tokenizer. By creating an instance and giving a UTF8 string as an argument, a token can be obtained.

AiliaTokenizerModel model = new AiliaTokenizerModel();
model.Create(AiliaTokenizer.AILIA_TOKENIZER_TYPE_CLIP, AiliaTokenizer.AILIA_TOKENIZER_FLAG_UTF8_SAFE);
string text = "ハードウェア ソフトウェア";
int [] tokens = model.Encode(text);
string decoded = model.Decode(tokens);
model.Close();

For tokenizers that require a model, load the model with the Open API immediately after Create.

string model_path = "AiliaTokenizer/sentencepiece.bpe.model";
string asset_path = Application.streamingAssetsPath;
 
#if UNITY_ANDROID
  CopyModelToTemporaryCachePath(model_path);
  asset_path=Application.temporaryCachePath;
#endif
model.Open(model_path = asset_path+"/"+model_path);