1*61c4878aSAndroid Build Coastguard Worker:tocdepth: 2 2*61c4878aSAndroid Build Coastguard Worker 3*61c4878aSAndroid Build Coastguard Worker.. _module-pw_tokenizer-api: 4*61c4878aSAndroid Build Coastguard Worker 5*61c4878aSAndroid Build Coastguard Worker========================== 6*61c4878aSAndroid Build Coastguard Workerpw_tokenizer API reference 7*61c4878aSAndroid Build Coastguard Worker========================== 8*61c4878aSAndroid Build Coastguard Worker.. pigweed-module-subpage:: 9*61c4878aSAndroid Build Coastguard Worker :name: pw_tokenizer 10*61c4878aSAndroid Build Coastguard Worker 11*61c4878aSAndroid Build Coastguard Worker.. _module-pw_tokenizer-api-configuration: 12*61c4878aSAndroid Build Coastguard Worker 13*61c4878aSAndroid Build Coastguard Worker------------- 14*61c4878aSAndroid Build Coastguard WorkerConfiguration 15*61c4878aSAndroid Build Coastguard Worker------------- 16*61c4878aSAndroid Build Coastguard Worker.. tab-set:: 17*61c4878aSAndroid Build Coastguard Worker 18*61c4878aSAndroid Build Coastguard Worker .. tab-item:: C++ / C 19*61c4878aSAndroid Build Coastguard Worker :sync: cpp 20*61c4878aSAndroid Build Coastguard Worker 21*61c4878aSAndroid Build Coastguard Worker .. doxygenfile:: pw_tokenizer/config.h 22*61c4878aSAndroid Build Coastguard Worker :sections: define 23*61c4878aSAndroid Build Coastguard Worker 24*61c4878aSAndroid Build Coastguard Worker------------ 25*61c4878aSAndroid Build Coastguard WorkerTokenization 26*61c4878aSAndroid Build Coastguard Worker------------ 27*61c4878aSAndroid Build Coastguard Worker.. tab-set:: 28*61c4878aSAndroid Build Coastguard Worker 29*61c4878aSAndroid Build Coastguard Worker .. tab-item:: C++ / C 30*61c4878aSAndroid Build Coastguard Worker :sync: cpp 31*61c4878aSAndroid Build Coastguard Worker 32*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw::tokenizer::EncodeArgs 33*61c4878aSAndroid Build Coastguard Worker .. doxygenclass:: pw::tokenizer::EncodedMessage 34*61c4878aSAndroid Build Coastguard Worker :members: 35*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw::tokenizer::MinEncodingBufferSizeBytes 36*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw::tokenizer::EnumToToken 37*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw::tokenizer::EnumToString 38*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKEN_FMT 39*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_ENUM 40*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_ENUM_CUSTOM 41*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING 42*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING_ANY_ARG_COUNT 43*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING 44*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN 45*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN_EXPR 46*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING_EXPR 47*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING_MASK 48*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_STRING_MASK_EXPR 49*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_TO_BUFFER 50*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_DOMAIN 51*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_MASK 52*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZER_REPLACE_FORMAT_STRING 53*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZER_ARG_TYPES 54*61c4878aSAndroid Build Coastguard Worker .. doxygendefine:: PW_TOKENIZER_DEFINE_TOKEN 55*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw_tokenizer_EncodeArgs 56*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw_tokenizer_EncodeInt 57*61c4878aSAndroid Build Coastguard Worker .. doxygenfunction:: pw_tokenizer_EncodeInt64 58*61c4878aSAndroid Build Coastguard Worker .. doxygentypedef:: pw_tokenizer_Token 59*61c4878aSAndroid Build Coastguard Worker 60*61c4878aSAndroid Build Coastguard Worker .. tab-item:: Python 61*61c4878aSAndroid Build Coastguard Worker :sync: py 62*61c4878aSAndroid Build Coastguard Worker 63*61c4878aSAndroid Build Coastguard Worker .. autofunction:: pw_tokenizer.encode.encode_token_and_args 64*61c4878aSAndroid Build Coastguard Worker .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash 65*61c4878aSAndroid Build Coastguard Worker 66*61c4878aSAndroid Build Coastguard Worker .. tab-item:: Rust 67*61c4878aSAndroid Build Coastguard Worker :sync: rs 68*61c4878aSAndroid Build Coastguard Worker 69*61c4878aSAndroid Build Coastguard Worker See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_. 70*61c4878aSAndroid Build Coastguard Worker 71*61c4878aSAndroid Build Coastguard Worker.. _module-pw_tokenizer-api-token-databases: 72*61c4878aSAndroid Build Coastguard Worker 73*61c4878aSAndroid Build Coastguard Worker--------------- 74*61c4878aSAndroid Build Coastguard WorkerToken databases 75*61c4878aSAndroid Build Coastguard Worker--------------- 76*61c4878aSAndroid Build Coastguard Worker.. tab-set:: 77*61c4878aSAndroid Build Coastguard Worker 78*61c4878aSAndroid Build Coastguard Worker .. tab-item:: C++ / C 79*61c4878aSAndroid Build Coastguard Worker :sync: cpp 80*61c4878aSAndroid Build Coastguard Worker 81*61c4878aSAndroid Build Coastguard Worker .. doxygenclass:: pw::tokenizer::TokenDatabase 82*61c4878aSAndroid Build Coastguard Worker :members: 83*61c4878aSAndroid Build Coastguard Worker 84*61c4878aSAndroid Build Coastguard Worker.. _module-pw_tokenizer-api-detokenization: 85*61c4878aSAndroid Build Coastguard Worker 86*61c4878aSAndroid Build Coastguard Worker-------------- 87*61c4878aSAndroid Build Coastguard WorkerDetokenization 88*61c4878aSAndroid Build Coastguard Worker-------------- 89*61c4878aSAndroid Build Coastguard Worker.. tab-set:: 90*61c4878aSAndroid Build Coastguard Worker 91*61c4878aSAndroid Build Coastguard Worker .. tab-item:: C++ / C 92*61c4878aSAndroid Build Coastguard Worker :sync: cpp 93*61c4878aSAndroid Build Coastguard Worker 94*61c4878aSAndroid Build Coastguard Worker .. doxygengroup:: pw_tokenizer_detokenize 95*61c4878aSAndroid Build Coastguard Worker :content-only: 96*61c4878aSAndroid Build Coastguard Worker :members: 97*61c4878aSAndroid Build Coastguard Worker 98*61c4878aSAndroid Build Coastguard Worker .. tab-item:: Python 99*61c4878aSAndroid Build Coastguard Worker :sync: py 100*61c4878aSAndroid Build Coastguard Worker 101*61c4878aSAndroid Build Coastguard Worker .. automodule:: pw_tokenizer.detokenize 102*61c4878aSAndroid Build Coastguard Worker :members: 103*61c4878aSAndroid Build Coastguard Worker 104*61c4878aSAndroid Build Coastguard Worker .. automodule:: pw_tokenizer.proto 105*61c4878aSAndroid Build Coastguard Worker :members: 106