1.. _module-pw_log_tokenized: 2 3---------------- 4pw_log_tokenized 5---------------- 6The ``pw_log_tokenized`` module contains utilities for tokenized logging. It 7connects ``pw_log`` to ``pw_tokenizer`` and supports 8:ref:`module-pw_log-tokenized-args`. 9 10C++ backend 11=========== 12``pw_log_tokenized`` provides a backend for ``pw_log`` that tokenizes log 13messages with the ``pw_tokenizer`` module. The log level, 16-bit tokenized 14module name, and flags bits are passed through the payload argument. The macro 15eventually passes logs to the :c:func:`pw_log_tokenized_HandleLog` function, 16which must be implemented by the application. 17 18.. doxygenfunction:: pw_log_tokenized_HandleLog 19 20Example implementation: 21 22.. code-block:: cpp 23 24 extern "C" void pw_log_tokenized_HandleLog( 25 uint32_t payload, const uint8_t message[], size_t size) { 26 // The metadata object provides the log level, module token, and flags. 27 // These values can be recorded and used for runtime filtering. 28 pw::log_tokenized::Metadata metadata(payload); 29 30 if (metadata.level() < current_log_level) { 31 return; 32 } 33 34 if (metadata.flags() & HIGH_PRIORITY_LOG != 0) { 35 EmitHighPriorityLog(metadata.module(), message, size); 36 } else { 37 EmitLowPriorityLog(metadata.module(), message, size); 38 } 39 } 40 41See the documentation for :ref:`module-pw_tokenizer` for further details. 42 43Metadata in the format string 44----------------------------- 45With tokenized logging, the log format string is converted to a 32-bit token. 46Regardless of how long the format string is, it's always represented by a 32-bit 47token. Because of this, metadata can be packed into the tokenized string with 48no cost. 49 50``pw_log_tokenized`` uses a simple key-value format to encode metadata in a 51format string. Each field starts with the ``■`` (U+25A0 "Black Square") 52character, followed by the key name, the ``♦`` (U+2666 "Black Diamond Suit") 53character, and then the value. The string is encoded as UTF-8. Key names are 54comprised of alphanumeric ASCII characters and underscore and start with a 55letter. 56 57.. code-block:: 58 59 "■key1♦contents1■key2♦contents2■key3♦contents3" 60 61This format makes the message easily machine parseable and human readable. It is 62extremely unlikely to conflict with log message contents due to the characters 63used. 64 65``pw_log_tokenized`` uses three fields: ``msg``, ``module``, and ``file``. 66Implementations may add other fields, but they will be ignored by the 67``pw_log_tokenized`` tooling. 68 69.. code-block:: 70 71 "■msg♦Hyperdrive %d set to %f■module♦engine■file♦propulsion/hyper.cc" 72 73Using key-value pairs allows placing the fields in any order. 74``pw_log_tokenized`` places the message first. This is prefered when tokenizing 75C code because the tokenizer only hashes a fixed number of characters. If the 76file were first, the long path might take most of the hashed characters, 77increasing the odds of a collision with other strings in that file. In C++, all 78characters in the string are hashed, so the order is not important. 79 80The format string is created by the :c:macro:`PW_LOG_TOKENIZED_FORMAT_STRING` 81macro. 82 83.. doxygendefine:: PW_LOG_TOKENIZED_FORMAT_STRING 84 85Metadata in the tokenizer payload argument 86------------------------------------------- 87``pw_log_tokenized`` packs runtime-accessible metadata into a 32-bit integer 88which is passed as the "payload" argument for ``pw_log_tokenizer``'s global 89handler with payload facade. Packing this metadata into a single word rather 90than separate arguments reduces the code size significantly. 91 92Four items are packed into the payload argument: 93 94- Log level -- Used for runtime log filtering by level. 95- Line number -- Used to track where a log message originated. 96- Log flags -- Implementation-defined log flags. 97- Tokenized :c:macro:`PW_LOG_MODULE_NAME` -- Used for runtime log filtering by 98 module. 99 100Configuring metadata bit fields 101^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 102The number of bits to use for each metadata field is configurable through macros 103in ``pw_log/config.h``. The field widths must sum to 32 bits. A field with zero 104bits allocated is excluded from the log metadata. 105 106.. doxygendefine:: PW_LOG_TOKENIZED_LEVEL_BITS 107.. doxygendefine:: PW_LOG_TOKENIZED_LINE_BITS 108.. doxygendefine:: PW_LOG_TOKENIZED_FLAG_BITS 109.. doxygendefine:: PW_LOG_TOKENIZED_MODULE_BITS 110 111Creating and reading Metadata payloads 112^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 113``pw_log_tokenized`` provides a C++ class to facilitate the creation and 114interpretation of packed log metadata payloads. 115 116.. doxygenclass:: pw::log_tokenized::GenericMetadata 117.. doxygentypedef:: pw::log_tokenized::Metadata 118 119The following example shows that a ``Metadata`` object can be created from a 120``uint32_t`` log metadata payload. 121 122.. code-block:: cpp 123 124 extern "C" void pw_log_tokenized_HandleLog( 125 uint32_t payload, 126 const uint8_t message[], 127 size_t size_bytes) { 128 pw::log_tokenized::Metadata metadata = payload; 129 // Check the log level to see if this log is a crash. 130 if (metadata.level() == PW_LOG_LEVEL_FATAL) { 131 HandleCrash(metadata, pw::ConstByteSpan( 132 reinterpret_cast<const std::byte*>(message), size_bytes)); 133 PW_UNREACHABLE; 134 } 135 // ... 136 } 137 138It's also possible to get a ``uint32_t`` representation of a ``Metadata`` 139object: 140 141.. code-block:: cpp 142 143 // Logs an explicitly created string token. 144 void LogToken(uint32_t token, int level, int line_number, int module) { 145 const uint32_t payload = 146 log_tokenized::Metadata( 147 level, module, PW_LOG_FLAGS, line_number) 148 .value(); 149 std::array<std::byte, sizeof(token)> token_buffer = 150 pw::bytes::CopyInOrder(endian::little, token); 151 152 pw_log_tokenized_HandleLog( 153 payload, 154 reinterpret_cast<const uint8_t*>(token_buffer.data()), 155 token_buffer.size()); 156 } 157 158The binary tokenized message may be encoded in the :ref:`prefixed Base64 format 159<module-pw_tokenizer-base64-format>` with the following function: 160 161.. doxygenfunction:: PrefixedBase64Encode(span<const std::byte>) 162 163Build targets 164------------- 165The GN build for ``pw_log_tokenized`` has two targets: ``pw_log_tokenized`` and 166``log_backend``. The ``pw_log_tokenized`` target provides the 167``pw_log_tokenized/log_tokenized.h`` header. The ``log_backend`` target 168implements the backend for the ``pw_log`` facade. ``pw_log_tokenized`` invokes 169the ``pw_log_tokenized:handler`` facade, which must be implemented by the user 170of ``pw_log_tokenized``. 171 172GCC has a bug resulting in section attributes of templated functions being 173ignored. This in turn means that log tokenization cannot work for templated 174functions, because the token database entries are lost at build time. 175For more information see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70435. 176If you are using GCC, the ``gcc_partially_tokenized`` target can be used as a 177backend for the ``pw_log`` facade instead which tokenizes as much as possible 178and uses the ``pw_log_string:handler`` for the rest using string logging. 179 180Python package 181============== 182``pw_log_tokenized`` includes a Python package for decoding tokenized logs. 183 184pw_log_tokenized 185---------------- 186.. automodule:: pw_log_tokenized 187 :members: 188 :undoc-members: 189