ailiaTokenizer.AiliaTokenizerModel Class Reference

Inheritance diagram for ailiaTokenizer.AiliaTokenizerModel:

Collaboration diagram for ailiaTokenizer.AiliaTokenizerModel:

Public Member Functions
bool	Create (int type, int flag)
	Create a instance. More...
bool	Open (string model_path=null, string dictionary_path=null, string vocab_path=null)
	Open a model. More...
virtual void	Close ()
	Destroys instance. More...
virtual void	Dispose ()
	Release resources. More...
int[]	Encode (string utf8)
	Perform encode. More...
int[]	EncodeWithSpecialTokens (string utf8)
	Perform encode with special tokens. More...
string	Decode (int[] tokens)
	Perform decode @pram tokens Input tokens. More...
string	DecodeWithSpecialTokens (int[] tokens)
	Perform decode with special tokens @pram tokens Input tokens. More...
int	GetVocabSize ()
	Gets the size of vocab. More...
string	GetVocab (int token)
	Perform encode. More...

Protected Member Functions
virtual void	Dispose (bool disposing)

Member Function Documentation

Close()

virtual void ailiaTokenizer.AiliaTokenizerModel.Close ( )

inlinevirtual

Destroys instance.

Destroys and initializes the instance.

    {
        if (net != IntPtr.Zero){
            AiliaTokenizer.ailiaTokenizerDestroy(net);
            net = IntPtr.Zero;
        }
    }

Create()

bool ailiaTokenizer.AiliaTokenizerModel.Create ( int  type, int  flag  )

inline

Create a instance.

Parameters

type	Type (AiliaTokenizer..AILIA_TOKENIZER_TYPE_*)
flag	OR of flags (AiliaTokenizer..AILIA_TOKENIZER_FLAG_*)

Returns

If this function is successful, it returns true , or false otherwise.

                                          {
        if (net != null){
            Close();
        }
 
        int status = AiliaTokenizer.ailiaTokenizerCreate(ref net, type, flag);
        if (status != 0){
            return false;
        }
 
        return true;
    }

Decode()

string ailiaTokenizer.AiliaTokenizerModel.Decode ( int[]  tokens )

inline

Perform decode @pram tokens Input tokens.

Returns

If this function is successful, it returns string , or empty string otherwise.

    {
        return DecodeCore(tokens, false);
    }

DecodeWithSpecialTokens()

string ailiaTokenizer.AiliaTokenizerModel.DecodeWithSpecialTokens ( int[]  tokens )

inline

Perform decode with special tokens @pram tokens Input tokens.

Returns

If this function is successful, it returns string , or empty string otherwise.

    {
        return DecodeCore(tokens, true);
    }

Dispose() [1/2]

virtual void ailiaTokenizer.AiliaTokenizerModel.Dispose ( )

inlinevirtual

Release resources.

    {
        Dispose(true);
    }

Dispose() [2/2]

virtual void ailiaTokenizer.AiliaTokenizerModel.Dispose ( bool  disposing )

inlineprotectedvirtual

    {
        if (disposing){
            // release managed resource
        }
        Close(); // release unmanaged resource
    }

Encode()

int [] ailiaTokenizer.AiliaTokenizerModel.Encode ( string  utf8 )

inline

Perform encode.

Parameters

utf8	Input string

Returns

If this function is successful, it returns array of tokens , or empty array otherwise.

    {
        return EncodeCore(utf8, false);
    }

EncodeWithSpecialTokens()

int [] ailiaTokenizer.AiliaTokenizerModel.EncodeWithSpecialTokens ( string  utf8 )

inline

Perform encode with special tokens.

Parameters

utf8	Input string

Returns

If this function is successful, it returns array of tokens , or empty array otherwise.

    {
        return EncodeCore(utf8, true);
    }

GetVocab()

string ailiaTokenizer.AiliaTokenizerModel.GetVocab ( int  token )

inline

Perform encode.

Parameters

token

Token

Returns

If this function is successful, it returns string , or null otherwise.

    {
        IntPtr ptr = IntPtr.Zero;
        int status = AiliaTokenizer.ailiaTokenizerGetVocab(net, token, ref ptr);
        if (status != 0){
            return null;
        }
        return Marshal.PtrToStringAnsi(ptr);
    }

GetVocabSize()

int ailiaTokenizer.AiliaTokenizerModel.GetVocabSize ( )

inline

Gets the size of vocab.

Returns

If this function is successful, it returns the size of vocab , or -1 otherwise.

    {
        uint len = 0;
        int status = AiliaTokenizer.ailiaTokenizerGetVocabSize(net, ref len);
        if (status != 0){
            return -1;
        }
        return (int)len;
    }

Open()

bool ailiaTokenizer.AiliaTokenizerModel.Open ( string  model_path = null, string  dictionary_path = null, string  vocab_path = null  )

inline

Open a model.

Parameters

model_path	Path for model (don't load if null)
dictionary_path	Path for dictionary (don't load if null)
vocab_path	Path for vocab (don't load if null)

Returns

If this function is successful, it returns true , or false otherwise.

                                                                                                       {
        if (net == null){
            return false;
        }
 
        int status = 0;
        
        if (model_path != null){
            status = AiliaTokenizer.ailiaTokenizerOpenModelFile(net, model_path);
            if (status != 0){
                return false;
            }
        }
        if (dictionary_path != null){
            status = AiliaTokenizer.ailiaTokenizerOpenDictionaryFile(net, dictionary_path);
            if (status != 0){
                return false;
            }
        }
        if (vocab_path != null){
            status = AiliaTokenizer.ailiaTokenizerOpenVocabFile(net, vocab_path);
            if (status != 0){
                return false;
            }
        }
 
        return true;
    }

---

The documentation for this class was generated from the following file:

• AiliaTokenizerModel.cs