Searched refs:byte_to_codepoint_offsets (Results 1 – 1 of 1) sorted by relevance
114 std::vector<int> byte_to_codepoint_offsets; in TokenizeIntoWordpieces() local117 byte_to_codepoint_offsets.resize( in TokenizeIntoWordpieces()121 byte_to_codepoint_offsets.push_back(byte_to_codepoint_offset); in TokenizeIntoWordpieces()125 token.start + SafeLookup(byte_to_codepoint_offsets, offset)); in TokenizeIntoWordpieces()129 token.start + SafeLookup(byte_to_codepoint_offsets, offset)); in TokenizeIntoWordpieces()