Home
last modified time | relevance | path

Searched refs:byte_to_codepoint_offset (Results 1 – 1 of 1) sorted by relevance

/aosp_15_r20/external/libtextclassifier/native/utils/
H A Dbert_tokenizer.cc115 int byte_to_codepoint_offset = 0; in TokenizeIntoWordpieces() local
119 byte_to_codepoint_offset++); in TokenizeIntoWordpieces()
121 byte_to_codepoint_offsets.push_back(byte_to_codepoint_offset); in TokenizeIntoWordpieces()