1:tocdepth: 2 2 3.. _module-pw_tokenizer-api: 4 5========================== 6pw_tokenizer API reference 7========================== 8.. pigweed-module-subpage:: 9 :name: pw_tokenizer 10 11.. _module-pw_tokenizer-api-configuration: 12 13------------- 14Configuration 15------------- 16.. tab-set:: 17 18 .. tab-item:: C++ / C 19 :sync: cpp 20 21 .. doxygenfile:: pw_tokenizer/config.h 22 :sections: define 23 24------------ 25Tokenization 26------------ 27.. tab-set:: 28 29 .. tab-item:: C++ / C 30 :sync: cpp 31 32 .. doxygenfunction:: pw::tokenizer::EncodeArgs 33 .. doxygenclass:: pw::tokenizer::EncodedMessage 34 :members: 35 .. doxygenfunction:: pw::tokenizer::MinEncodingBufferSizeBytes 36 .. doxygenfunction:: pw::tokenizer::EnumToToken 37 .. doxygenfunction:: pw::tokenizer::EnumToString 38 .. doxygendefine:: PW_TOKEN_FMT 39 .. doxygendefine:: PW_TOKENIZE_ENUM 40 .. doxygendefine:: PW_TOKENIZE_ENUM_CUSTOM 41 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING 42 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING_ANY_ARG_COUNT 43 .. doxygendefine:: PW_TOKENIZE_STRING 44 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN 45 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN_EXPR 46 .. doxygendefine:: PW_TOKENIZE_STRING_EXPR 47 .. doxygendefine:: PW_TOKENIZE_STRING_MASK 48 .. doxygendefine:: PW_TOKENIZE_STRING_MASK_EXPR 49 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER 50 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_DOMAIN 51 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_MASK 52 .. doxygendefine:: PW_TOKENIZER_REPLACE_FORMAT_STRING 53 .. doxygendefine:: PW_TOKENIZER_ARG_TYPES 54 .. doxygendefine:: PW_TOKENIZER_DEFINE_TOKEN 55 .. doxygenfunction:: pw_tokenizer_EncodeArgs 56 .. doxygenfunction:: pw_tokenizer_EncodeInt 57 .. doxygenfunction:: pw_tokenizer_EncodeInt64 58 .. doxygentypedef:: pw_tokenizer_Token 59 60 .. tab-item:: Python 61 :sync: py 62 63 .. autofunction:: pw_tokenizer.encode.encode_token_and_args 64 .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash 65 66 .. tab-item:: Rust 67 :sync: rs 68 69 See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_. 70 71.. _module-pw_tokenizer-api-token-databases: 72 73--------------- 74Token databases 75--------------- 76.. tab-set:: 77 78 .. tab-item:: C++ / C 79 :sync: cpp 80 81 .. doxygenclass:: pw::tokenizer::TokenDatabase 82 :members: 83 84.. _module-pw_tokenizer-api-detokenization: 85 86-------------- 87Detokenization 88-------------- 89.. tab-set:: 90 91 .. tab-item:: C++ / C 92 :sync: cpp 93 94 .. doxygengroup:: pw_tokenizer_detokenize 95 :content-only: 96 :members: 97 98 .. tab-item:: Python 99 :sync: py 100 101 .. automodule:: pw_tokenizer.detokenize 102 :members: 103 104 .. automodule:: pw_tokenizer.proto 105 :members: 106