1*22dc650dSSadaf Ebrahimi<html> 2*22dc650dSSadaf Ebrahimi<head> 3*22dc650dSSadaf Ebrahimi<title>pcre2jit specification</title> 4*22dc650dSSadaf Ebrahimi</head> 5*22dc650dSSadaf Ebrahimi<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6*22dc650dSSadaf Ebrahimi<h1>pcre2jit man page</h1> 7*22dc650dSSadaf Ebrahimi<p> 8*22dc650dSSadaf EbrahimiReturn to the <a href="index.html">PCRE2 index page</a>. 9*22dc650dSSadaf Ebrahimi</p> 10*22dc650dSSadaf Ebrahimi<p> 11*22dc650dSSadaf EbrahimiThis page is part of the PCRE2 HTML documentation. It was generated 12*22dc650dSSadaf Ebrahimiautomatically from the original man page. If there is any nonsense in it, 13*22dc650dSSadaf Ebrahimiplease consult the man page, in case the conversion went wrong. 14*22dc650dSSadaf Ebrahimi<br> 15*22dc650dSSadaf Ebrahimi<ul> 16*22dc650dSSadaf Ebrahimi<li><a name="TOC1" href="#SEC1">PCRE2 JUST-IN-TIME COMPILER SUPPORT</a> 17*22dc650dSSadaf Ebrahimi<li><a name="TOC2" href="#SEC2">AVAILABILITY OF JIT SUPPORT</a> 18*22dc650dSSadaf Ebrahimi<li><a name="TOC3" href="#SEC3">SIMPLE USE OF JIT</a> 19*22dc650dSSadaf Ebrahimi<li><a name="TOC4" href="#SEC4">MATCHING SUBJECTS CONTAINING INVALID UTF</a> 20*22dc650dSSadaf Ebrahimi<li><a name="TOC5" href="#SEC5">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a> 21*22dc650dSSadaf Ebrahimi<li><a name="TOC6" href="#SEC6">RETURN VALUES FROM JIT MATCHING</a> 22*22dc650dSSadaf Ebrahimi<li><a name="TOC7" href="#SEC7">CONTROLLING THE JIT STACK</a> 23*22dc650dSSadaf Ebrahimi<li><a name="TOC8" href="#SEC8">JIT STACK FAQ</a> 24*22dc650dSSadaf Ebrahimi<li><a name="TOC9" href="#SEC9">FREEING JIT SPECULATIVE MEMORY</a> 25*22dc650dSSadaf Ebrahimi<li><a name="TOC10" href="#SEC10">EXAMPLE CODE</a> 26*22dc650dSSadaf Ebrahimi<li><a name="TOC11" href="#SEC11">JIT FAST PATH API</a> 27*22dc650dSSadaf Ebrahimi<li><a name="TOC12" href="#SEC12">SEE ALSO</a> 28*22dc650dSSadaf Ebrahimi<li><a name="TOC13" href="#SEC13">AUTHOR</a> 29*22dc650dSSadaf Ebrahimi<li><a name="TOC14" href="#SEC14">REVISION</a> 30*22dc650dSSadaf Ebrahimi</ul> 31*22dc650dSSadaf Ebrahimi<br><a name="SEC1" href="#TOC1">PCRE2 JUST-IN-TIME COMPILER SUPPORT</a><br> 32*22dc650dSSadaf Ebrahimi<P> 33*22dc650dSSadaf EbrahimiJust-in-time compiling is a heavyweight optimization that can greatly speed up 34*22dc650dSSadaf Ebrahimipattern matching. However, it comes at the cost of extra processing before the 35*22dc650dSSadaf Ebrahimimatch is performed, so it is of most benefit when the same pattern is going to 36*22dc650dSSadaf Ebrahimibe matched many times. This does not necessarily mean many calls of a matching 37*22dc650dSSadaf Ebrahimifunction; if the pattern is not anchored, matching attempts may take place many 38*22dc650dSSadaf Ebrahimitimes at various positions in the subject, even for a single call. Therefore, 39*22dc650dSSadaf Ebrahimiif the subject string is very long, it may still pay to use JIT even for 40*22dc650dSSadaf Ebrahimione-off matches. JIT support is available for all of the 8-bit, 16-bit and 41*22dc650dSSadaf Ebrahimi32-bit PCRE2 libraries. 42*22dc650dSSadaf Ebrahimi</P> 43*22dc650dSSadaf Ebrahimi<P> 44*22dc650dSSadaf EbrahimiJIT support applies only to the traditional Perl-compatible matching function. 45*22dc650dSSadaf EbrahimiIt does not apply when the DFA matching function is being used. The code for 46*22dc650dSSadaf EbrahimiJIT support was written by Zoltan Herczeg. 47*22dc650dSSadaf Ebrahimi</P> 48*22dc650dSSadaf Ebrahimi<br><a name="SEC2" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br> 49*22dc650dSSadaf Ebrahimi<P> 50*22dc650dSSadaf EbrahimiJIT support is an optional feature of PCRE2. The "configure" option 51*22dc650dSSadaf Ebrahimi--enable-jit (or equivalent CMake option) must be set when PCRE2 is built if 52*22dc650dSSadaf Ebrahimiyou want to use JIT. The support is limited to the following hardware 53*22dc650dSSadaf Ebrahimiplatforms: 54*22dc650dSSadaf Ebrahimi<pre> 55*22dc650dSSadaf Ebrahimi ARM 32-bit (v7, and Thumb2) 56*22dc650dSSadaf Ebrahimi ARM 64-bit 57*22dc650dSSadaf Ebrahimi IBM s390x 64 bit 58*22dc650dSSadaf Ebrahimi Intel x86 32-bit and 64-bit 59*22dc650dSSadaf Ebrahimi LoongArch 64 bit 60*22dc650dSSadaf Ebrahimi MIPS 32-bit and 64-bit 61*22dc650dSSadaf Ebrahimi Power PC 32-bit and 64-bit 62*22dc650dSSadaf Ebrahimi RISC-V 32-bit and 64-bit 63*22dc650dSSadaf Ebrahimi</pre> 64*22dc650dSSadaf EbrahimiIf --enable-jit is set on an unsupported platform, compilation fails. 65*22dc650dSSadaf Ebrahimi</P> 66*22dc650dSSadaf Ebrahimi<P> 67*22dc650dSSadaf EbrahimiA client program can tell if JIT support is available by calling 68*22dc650dSSadaf Ebrahimi<b>pcre2_config()</b> with the PCRE2_CONFIG_JIT option. The result is one if 69*22dc650dSSadaf EbrahimiPCRE2 was built with JIT support, and zero otherwise. However, having the JIT 70*22dc650dSSadaf Ebrahimicode available does not guarantee that it will be used for any particular 71*22dc650dSSadaf Ebrahimimatch. One reason for this is that there are a number of options and pattern 72*22dc650dSSadaf Ebrahimiitems that are 73*22dc650dSSadaf Ebrahimi<a href="#unsupported">not supported by JIT</a> 74*22dc650dSSadaf Ebrahimi(see below). Another reason is that in some environments JIT is unable to get 75*22dc650dSSadaf Ebrahimimemory in which to build its compiled code. The only guarantee from 76*22dc650dSSadaf Ebrahimi<b>pcre2_config()</b> is that if it returns zero, JIT will definitely <i>not</i> 77*22dc650dSSadaf Ebrahimibe used. 78*22dc650dSSadaf Ebrahimi</P> 79*22dc650dSSadaf Ebrahimi<P> 80*22dc650dSSadaf EbrahimiA simple program does not need to check availability in order to use JIT when 81*22dc650dSSadaf Ebrahimipossible. The API is implemented in a way that falls back to the interpretive 82*22dc650dSSadaf Ebrahimicode if JIT is not available or cannot be used for a given match. For programs 83*22dc650dSSadaf Ebrahimithat need the best possible performance, there is a 84*22dc650dSSadaf Ebrahimi<a href="#fastpath">"fast path"</a> 85*22dc650dSSadaf EbrahimiAPI that is JIT-specific. 86*22dc650dSSadaf Ebrahimi</P> 87*22dc650dSSadaf Ebrahimi<br><a name="SEC3" href="#TOC1">SIMPLE USE OF JIT</a><br> 88*22dc650dSSadaf Ebrahimi<P> 89*22dc650dSSadaf EbrahimiTo make use of the JIT support in the simplest way, all you have to do is to 90*22dc650dSSadaf Ebrahimicall <b>pcre2_jit_compile()</b> after successfully compiling a pattern with 91*22dc650dSSadaf Ebrahimi<b>pcre2_compile()</b>. This function has two arguments: the first is the 92*22dc650dSSadaf Ebrahimicompiled pattern pointer that was returned by <b>pcre2_compile()</b>, and the 93*22dc650dSSadaf Ebrahimisecond is zero or more of the following option bits: PCRE2_JIT_COMPLETE, 94*22dc650dSSadaf EbrahimiPCRE2_JIT_PARTIAL_HARD, or PCRE2_JIT_PARTIAL_SOFT. 95*22dc650dSSadaf Ebrahimi</P> 96*22dc650dSSadaf Ebrahimi<P> 97*22dc650dSSadaf EbrahimiIf JIT support is not available, a call to <b>pcre2_jit_compile()</b> does 98*22dc650dSSadaf Ebrahiminothing and returns PCRE2_ERROR_JIT_BADOPTION. Otherwise, the compiled pattern 99*22dc650dSSadaf Ebrahimiis passed to the JIT compiler, which turns it into machine code that executes 100*22dc650dSSadaf Ebrahimimuch faster than the normal interpretive code, but yields exactly the same 101*22dc650dSSadaf Ebrahimiresults. The returned value from <b>pcre2_jit_compile()</b> is zero on success, 102*22dc650dSSadaf Ebrahimior a negative error code. 103*22dc650dSSadaf Ebrahimi</P> 104*22dc650dSSadaf Ebrahimi<P> 105*22dc650dSSadaf EbrahimiThere is a limit to the size of pattern that JIT supports, imposed by the size 106*22dc650dSSadaf Ebrahimiof machine stack that it uses. The exact rules are not documented because they 107*22dc650dSSadaf Ebrahimimay change at any time, in particular, when new optimizations are introduced. 108*22dc650dSSadaf EbrahimiIf a pattern is too big, a call to <b>pcre2_jit_compile()</b> returns 109*22dc650dSSadaf EbrahimiPCRE2_ERROR_NOMEMORY. 110*22dc650dSSadaf Ebrahimi</P> 111*22dc650dSSadaf Ebrahimi<P> 112*22dc650dSSadaf EbrahimiPCRE2_JIT_COMPLETE requests the JIT compiler to generate code for complete 113*22dc650dSSadaf Ebrahimimatches. If you want to run partial matches using the PCRE2_PARTIAL_HARD or 114*22dc650dSSadaf EbrahimiPCRE2_PARTIAL_SOFT options of <b>pcre2_match()</b>, you should set one or both 115*22dc650dSSadaf Ebrahimiof the other options as well as, or instead of PCRE2_JIT_COMPLETE. The JIT 116*22dc650dSSadaf Ebrahimicompiler generates different optimized code for each of the three modes 117*22dc650dSSadaf Ebrahimi(normal, soft partial, hard partial). When <b>pcre2_match()</b> is called, the 118*22dc650dSSadaf Ebrahimiappropriate code is run if it is available. Otherwise, the pattern is matched 119*22dc650dSSadaf Ebrahimiusing interpretive code. 120*22dc650dSSadaf Ebrahimi</P> 121*22dc650dSSadaf Ebrahimi<P> 122*22dc650dSSadaf EbrahimiYou can call <b>pcre2_jit_compile()</b> multiple times for the same compiled 123*22dc650dSSadaf Ebrahimipattern. It does nothing if it has previously compiled code for any of the 124*22dc650dSSadaf Ebrahimioption bits. For example, you can call it once with PCRE2_JIT_COMPLETE and 125*22dc650dSSadaf Ebrahimi(perhaps later, when you find you need partial matching) again with 126*22dc650dSSadaf EbrahimiPCRE2_JIT_COMPLETE and PCRE2_JIT_PARTIAL_HARD. This time it will ignore 127*22dc650dSSadaf EbrahimiPCRE2_JIT_COMPLETE and just compile code for partial matching. If 128*22dc650dSSadaf Ebrahimi<b>pcre2_jit_compile()</b> is called with no option bits set, it immediately 129*22dc650dSSadaf Ebrahimireturns zero. This is an alternative way of testing whether JIT is available. 130*22dc650dSSadaf Ebrahimi</P> 131*22dc650dSSadaf Ebrahimi<P> 132*22dc650dSSadaf EbrahimiAt present, it is not possible to free JIT compiled code except when the entire 133*22dc650dSSadaf Ebrahimicompiled pattern is freed by calling <b>pcre2_code_free()</b>. 134*22dc650dSSadaf Ebrahimi</P> 135*22dc650dSSadaf Ebrahimi<P> 136*22dc650dSSadaf EbrahimiIn some circumstances you may need to call additional functions. These are 137*22dc650dSSadaf Ebrahimidescribed in the section entitled 138*22dc650dSSadaf Ebrahimi<a href="#stackcontrol">"Controlling the JIT stack"</a> 139*22dc650dSSadaf Ebrahimibelow. 140*22dc650dSSadaf Ebrahimi</P> 141*22dc650dSSadaf Ebrahimi<P> 142*22dc650dSSadaf EbrahimiThere are some <b>pcre2_match()</b> options that are not supported by JIT, and 143*22dc650dSSadaf Ebrahimithere are also some pattern items that JIT cannot handle. Details are given 144*22dc650dSSadaf Ebrahimi<a href="#unsupported">below.</a> 145*22dc650dSSadaf EbrahimiIn both cases, matching automatically falls back to the interpretive code. If 146*22dc650dSSadaf Ebrahimiyou want to know whether JIT was actually used for a particular match, you 147*22dc650dSSadaf Ebrahimishould arrange for a JIT callback function to be set up as described in the 148*22dc650dSSadaf Ebrahimisection entitled 149*22dc650dSSadaf Ebrahimi<a href="#stackcontrol">"Controlling the JIT stack"</a> 150*22dc650dSSadaf Ebrahimibelow, even if you do not need to supply a non-default JIT stack. Such a 151*22dc650dSSadaf Ebrahimicallback function is called whenever JIT code is about to be obeyed. If the 152*22dc650dSSadaf Ebrahimimatch-time options are not right for JIT execution, the callback function is 153*22dc650dSSadaf Ebrahiminot obeyed. 154*22dc650dSSadaf Ebrahimi</P> 155*22dc650dSSadaf Ebrahimi<P> 156*22dc650dSSadaf EbrahimiIf the JIT compiler finds an unsupported item, no JIT data is generated. You 157*22dc650dSSadaf Ebrahimican find out if JIT compilation was successful for a compiled pattern by 158*22dc650dSSadaf Ebrahimicalling <b>pcre2_pattern_info()</b> with the PCRE2_INFO_JITSIZE option. A 159*22dc650dSSadaf Ebrahiminon-zero result means that JIT compilation was successful. A result of 0 means 160*22dc650dSSadaf Ebrahimithat JIT support is not available, or the pattern was not processed by 161*22dc650dSSadaf Ebrahimi<b>pcre2_jit_compile()</b>, or the JIT compiler was not able to handle the 162*22dc650dSSadaf Ebrahimipattern. Successful JIT compilation does not, however, guarantee the use of JIT 163*22dc650dSSadaf Ebrahimiat match time because there are some match time options that are not supported 164*22dc650dSSadaf Ebrahimiby JIT. 165*22dc650dSSadaf Ebrahimi</P> 166*22dc650dSSadaf Ebrahimi<br><a name="SEC4" href="#TOC1">MATCHING SUBJECTS CONTAINING INVALID UTF</a><br> 167*22dc650dSSadaf Ebrahimi<P> 168*22dc650dSSadaf EbrahimiWhen a pattern is compiled with the PCRE2_UTF option, subject strings are 169*22dc650dSSadaf Ebrahiminormally expected to be a valid sequence of UTF code units. By default, this is 170*22dc650dSSadaf Ebrahimichecked at the start of matching and an error is generated if invalid UTF is 171*22dc650dSSadaf Ebrahimidetected. The PCRE2_NO_UTF_CHECK option can be passed to <b>pcre2_match()</b> to 172*22dc650dSSadaf Ebrahimiskip the check (for improved performance) if you are sure that a subject string 173*22dc650dSSadaf Ebrahimiis valid. If this option is used with an invalid string, the result is 174*22dc650dSSadaf Ebrahimiundefined. The calling program may crash or loop or otherwise misbehave. 175*22dc650dSSadaf Ebrahimi</P> 176*22dc650dSSadaf Ebrahimi<P> 177*22dc650dSSadaf EbrahimiHowever, a way of running matches on strings that may contain invalid UTF 178*22dc650dSSadaf Ebrahimisequences is available. Calling <b>pcre2_compile()</b> with the 179*22dc650dSSadaf EbrahimiPCRE2_MATCH_INVALID_UTF option has two effects: it tells the interpreter in 180*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b> to support invalid UTF, and, if <b>pcre2_jit_compile()</b> 181*22dc650dSSadaf Ebrahimiis subsequently called, the compiled JIT code also supports invalid UTF. 182*22dc650dSSadaf EbrahimiDetails of how this support works, in both the JIT and the interpretive cases, 183*22dc650dSSadaf Ebrahimiis given in the 184*22dc650dSSadaf Ebrahimi<a href="pcre2unicode.html"><b>pcre2unicode</b></a> 185*22dc650dSSadaf Ebrahimidocumentation. 186*22dc650dSSadaf Ebrahimi</P> 187*22dc650dSSadaf Ebrahimi<P> 188*22dc650dSSadaf EbrahimiThere is also an obsolete option for <b>pcre2_jit_compile()</b> called 189*22dc650dSSadaf EbrahimiPCRE2_JIT_INVALID_UTF, which currently exists only for backward compatibility. 190*22dc650dSSadaf EbrahimiIt is superseded by the <b>pcre2_compile()</b> option PCRE2_MATCH_INVALID_UTF 191*22dc650dSSadaf Ebrahimiand should no longer be used. It may be removed in future. 192*22dc650dSSadaf Ebrahimi<a name="unsupported"></a></P> 193*22dc650dSSadaf Ebrahimi<br><a name="SEC5" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br> 194*22dc650dSSadaf Ebrahimi<P> 195*22dc650dSSadaf EbrahimiThe <b>pcre2_match()</b> options that are supported for JIT matching are 196*22dc650dSSadaf EbrahimiPCRE2_COPY_MATCHED_SUBJECT, PCRE2_NOTBOL, PCRE2_NOTEOL, PCRE2_NOTEMPTY, 197*22dc650dSSadaf EbrahimiPCRE2_NOTEMPTY_ATSTART, PCRE2_NO_UTF_CHECK, PCRE2_PARTIAL_HARD, and 198*22dc650dSSadaf EbrahimiPCRE2_PARTIAL_SOFT. The PCRE2_ANCHORED and PCRE2_ENDANCHORED options are not 199*22dc650dSSadaf Ebrahimisupported at match time. 200*22dc650dSSadaf Ebrahimi</P> 201*22dc650dSSadaf Ebrahimi<P> 202*22dc650dSSadaf EbrahimiIf the PCRE2_NO_JIT option is passed to <b>pcre2_match()</b> it disables the 203*22dc650dSSadaf Ebrahimiuse of JIT, forcing matching by the interpreter code. 204*22dc650dSSadaf Ebrahimi</P> 205*22dc650dSSadaf Ebrahimi<P> 206*22dc650dSSadaf EbrahimiThe only unsupported pattern items are \C (match a single data unit) when 207*22dc650dSSadaf Ebrahimirunning in a UTF mode, and a callout immediately before an assertion condition 208*22dc650dSSadaf Ebrahimiin a conditional group. 209*22dc650dSSadaf Ebrahimi</P> 210*22dc650dSSadaf Ebrahimi<br><a name="SEC6" href="#TOC1">RETURN VALUES FROM JIT MATCHING</a><br> 211*22dc650dSSadaf Ebrahimi<P> 212*22dc650dSSadaf EbrahimiWhen a pattern is matched using JIT, the return values are the same as those 213*22dc650dSSadaf Ebrahimigiven by the interpretive <b>pcre2_match()</b> code, with the addition of one 214*22dc650dSSadaf Ebrahiminew error code: PCRE2_ERROR_JIT_STACKLIMIT. This means that the memory used for 215*22dc650dSSadaf Ebrahimithe JIT stack was insufficient. See 216*22dc650dSSadaf Ebrahimi<a href="#stackcontrol">"Controlling the JIT stack"</a> 217*22dc650dSSadaf Ebrahimibelow for a discussion of JIT stack usage. 218*22dc650dSSadaf Ebrahimi</P> 219*22dc650dSSadaf Ebrahimi<P> 220*22dc650dSSadaf EbrahimiThe error code PCRE2_ERROR_MATCHLIMIT is returned by the JIT code if searching 221*22dc650dSSadaf Ebrahimia very large pattern tree goes on for too long, as it is in the same 222*22dc650dSSadaf Ebrahimicircumstance when JIT is not used, but the details of exactly what is counted 223*22dc650dSSadaf Ebrahimiare not the same. The PCRE2_ERROR_DEPTHLIMIT error code is never returned 224*22dc650dSSadaf Ebrahimiwhen JIT matching is used. 225*22dc650dSSadaf Ebrahimi<a name="stackcontrol"></a></P> 226*22dc650dSSadaf Ebrahimi<br><a name="SEC7" href="#TOC1">CONTROLLING THE JIT STACK</a><br> 227*22dc650dSSadaf Ebrahimi<P> 228*22dc650dSSadaf EbrahimiWhen the compiled JIT code runs, it needs a block of memory to use as a stack. 229*22dc650dSSadaf EbrahimiBy default, it uses 32KiB on the machine stack. However, some large or 230*22dc650dSSadaf Ebrahimicomplicated patterns need more than this. The error PCRE2_ERROR_JIT_STACKLIMIT 231*22dc650dSSadaf Ebrahimiis given when there is not enough stack. Three functions are provided for 232*22dc650dSSadaf Ebrahimimanaging blocks of memory for use as JIT stacks. There is further discussion 233*22dc650dSSadaf Ebrahimiabout the use of JIT stacks in the section entitled 234*22dc650dSSadaf Ebrahimi<a href="#stackfaq">"JIT stack FAQ"</a> 235*22dc650dSSadaf Ebrahimibelow. 236*22dc650dSSadaf Ebrahimi</P> 237*22dc650dSSadaf Ebrahimi<P> 238*22dc650dSSadaf EbrahimiThe <b>pcre2_jit_stack_create()</b> function creates a JIT stack. Its arguments 239*22dc650dSSadaf Ebrahimiare a starting size, a maximum size, and a general context (for memory 240*22dc650dSSadaf Ebrahimiallocation functions, or NULL for standard memory allocation). It returns a 241*22dc650dSSadaf Ebrahimipointer to an opaque structure of type <b>pcre2_jit_stack</b>, or NULL if there 242*22dc650dSSadaf Ebrahimiis an error. The <b>pcre2_jit_stack_free()</b> function is used to free a stack 243*22dc650dSSadaf Ebrahimithat is no longer needed. If its argument is NULL, this function returns 244*22dc650dSSadaf Ebrahimiimmediately, without doing anything. (For the technically minded: the address 245*22dc650dSSadaf Ebrahimispace is allocated by mmap or VirtualAlloc.) A maximum stack size of 512KiB to 246*22dc650dSSadaf Ebrahimi1MiB should be more than enough for any pattern. 247*22dc650dSSadaf Ebrahimi</P> 248*22dc650dSSadaf Ebrahimi<P> 249*22dc650dSSadaf EbrahimiThe <b>pcre2_jit_stack_assign()</b> function specifies which stack JIT code 250*22dc650dSSadaf Ebrahimishould use. Its arguments are as follows: 251*22dc650dSSadaf Ebrahimi<pre> 252*22dc650dSSadaf Ebrahimi pcre2_match_context *mcontext 253*22dc650dSSadaf Ebrahimi pcre2_jit_callback callback 254*22dc650dSSadaf Ebrahimi void *data 255*22dc650dSSadaf Ebrahimi</pre> 256*22dc650dSSadaf EbrahimiThe first argument is a pointer to a match context. When this is subsequently 257*22dc650dSSadaf Ebrahimipassed to a matching function, its information determines which JIT stack is 258*22dc650dSSadaf Ebrahimiused. If this argument is NULL, the function returns immediately, without doing 259*22dc650dSSadaf Ebrahimianything. There are three cases for the values of the other two options: 260*22dc650dSSadaf Ebrahimi<pre> 261*22dc650dSSadaf Ebrahimi (1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32KiB block 262*22dc650dSSadaf Ebrahimi on the machine stack is used. This is the default when a match 263*22dc650dSSadaf Ebrahimi context is created. 264*22dc650dSSadaf Ebrahimi 265*22dc650dSSadaf Ebrahimi (2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be 266*22dc650dSSadaf Ebrahimi a pointer to a valid JIT stack, the result of calling 267*22dc650dSSadaf Ebrahimi <b>pcre2_jit_stack_create()</b>. 268*22dc650dSSadaf Ebrahimi 269*22dc650dSSadaf Ebrahimi (3) If <i>callback</i> is not NULL, it must point to a function that is 270*22dc650dSSadaf Ebrahimi called with <i>data</i> as an argument at the start of matching, in 271*22dc650dSSadaf Ebrahimi order to set up a JIT stack. If the return from the callback 272*22dc650dSSadaf Ebrahimi function is NULL, the internal 32KiB stack is used; otherwise the 273*22dc650dSSadaf Ebrahimi return value must be a valid JIT stack, the result of calling 274*22dc650dSSadaf Ebrahimi <b>pcre2_jit_stack_create()</b>. 275*22dc650dSSadaf Ebrahimi</pre> 276*22dc650dSSadaf EbrahimiA callback function is obeyed whenever JIT code is about to be run; it is not 277*22dc650dSSadaf Ebrahimiobeyed when <b>pcre2_match()</b> is called with options that are incompatible 278*22dc650dSSadaf Ebrahimifor JIT matching. A callback function can therefore be used to determine 279*22dc650dSSadaf Ebrahimiwhether a match operation was executed by JIT or by the interpreter. 280*22dc650dSSadaf Ebrahimi</P> 281*22dc650dSSadaf Ebrahimi<P> 282*22dc650dSSadaf EbrahimiYou may safely use the same JIT stack for more than one pattern (either by 283*22dc650dSSadaf Ebrahimiassigning directly or by callback), as long as the patterns are matched 284*22dc650dSSadaf Ebrahimisequentially in the same thread. Currently, the only way to set up 285*22dc650dSSadaf Ebrahiminon-sequential matches in one thread is to use callouts: if a callout function 286*22dc650dSSadaf Ebrahimistarts another match, that match must use a different JIT stack to the one used 287*22dc650dSSadaf Ebrahimifor currently suspended match(es). 288*22dc650dSSadaf Ebrahimi</P> 289*22dc650dSSadaf Ebrahimi<P> 290*22dc650dSSadaf EbrahimiIn a multithread application, if you do not specify a JIT stack, or if you 291*22dc650dSSadaf Ebrahimiassign or pass back NULL from a callback, that is thread-safe, because each 292*22dc650dSSadaf Ebrahimithread has its own machine stack. However, if you assign or pass back a 293*22dc650dSSadaf Ebrahiminon-NULL JIT stack, this must be a different stack for each thread so that the 294*22dc650dSSadaf Ebrahimiapplication is thread-safe. 295*22dc650dSSadaf Ebrahimi</P> 296*22dc650dSSadaf Ebrahimi<P> 297*22dc650dSSadaf EbrahimiStrictly speaking, even more is allowed. You can assign the same non-NULL stack 298*22dc650dSSadaf Ebrahimito a match context that is used by any number of patterns, as long as they are 299*22dc650dSSadaf Ebrahiminot used for matching by multiple threads at the same time. For example, you 300*22dc650dSSadaf Ebrahimicould use the same stack in all compiled patterns, with a global mutex in the 301*22dc650dSSadaf Ebrahimicallback to wait until the stack is available for use. However, this is an 302*22dc650dSSadaf Ebrahimiinefficient solution, and not recommended. 303*22dc650dSSadaf Ebrahimi</P> 304*22dc650dSSadaf Ebrahimi<P> 305*22dc650dSSadaf EbrahimiThis is a suggestion for how a multithreaded program that needs to set up 306*22dc650dSSadaf Ebrahiminon-default JIT stacks might operate: 307*22dc650dSSadaf Ebrahimi<pre> 308*22dc650dSSadaf Ebrahimi During thread initialization 309*22dc650dSSadaf Ebrahimi thread_local_var = pcre2_jit_stack_create(...) 310*22dc650dSSadaf Ebrahimi 311*22dc650dSSadaf Ebrahimi During thread exit 312*22dc650dSSadaf Ebrahimi pcre2_jit_stack_free(thread_local_var) 313*22dc650dSSadaf Ebrahimi 314*22dc650dSSadaf Ebrahimi Use a one-line callback function 315*22dc650dSSadaf Ebrahimi return thread_local_var 316*22dc650dSSadaf Ebrahimi</pre> 317*22dc650dSSadaf EbrahimiAll the functions described in this section do nothing if JIT is not available. 318*22dc650dSSadaf Ebrahimi<a name="stackfaq"></a></P> 319*22dc650dSSadaf Ebrahimi<br><a name="SEC8" href="#TOC1">JIT STACK FAQ</a><br> 320*22dc650dSSadaf Ebrahimi<P> 321*22dc650dSSadaf Ebrahimi(1) Why do we need JIT stacks? 322*22dc650dSSadaf Ebrahimi<br> 323*22dc650dSSadaf Ebrahimi<br> 324*22dc650dSSadaf EbrahimiPCRE2 (and JIT) is a recursive, depth-first engine, so it needs a stack where 325*22dc650dSSadaf Ebrahimithe local data of the current node is pushed before checking its child nodes. 326*22dc650dSSadaf EbrahimiAllocating real machine stack on some platforms is difficult. For example, the 327*22dc650dSSadaf Ebrahimistack chain needs to be updated every time if we extend the stack on PowerPC. 328*22dc650dSSadaf EbrahimiAlthough it is possible, its updating time overhead decreases performance. So 329*22dc650dSSadaf Ebrahimiwe do the recursion in memory. 330*22dc650dSSadaf Ebrahimi</P> 331*22dc650dSSadaf Ebrahimi<P> 332*22dc650dSSadaf Ebrahimi(2) Why don't we simply allocate blocks of memory with <b>malloc()</b>? 333*22dc650dSSadaf Ebrahimi<br> 334*22dc650dSSadaf Ebrahimi<br> 335*22dc650dSSadaf EbrahimiModern operating systems have a nice feature: they can reserve an address space 336*22dc650dSSadaf Ebrahimiinstead of allocating memory. We can safely allocate memory pages inside this 337*22dc650dSSadaf Ebrahimiaddress space, so the stack could grow without moving memory data (this is 338*22dc650dSSadaf Ebrahimiimportant because of pointers). Thus we can allocate 1MiB address space, and 339*22dc650dSSadaf Ebrahimiuse only a single memory page (usually 4KiB) if that is enough. However, we can 340*22dc650dSSadaf Ebrahimistill grow up to 1MiB anytime if needed. 341*22dc650dSSadaf Ebrahimi</P> 342*22dc650dSSadaf Ebrahimi<P> 343*22dc650dSSadaf Ebrahimi(3) Who "owns" a JIT stack? 344*22dc650dSSadaf Ebrahimi<br> 345*22dc650dSSadaf Ebrahimi<br> 346*22dc650dSSadaf EbrahimiThe owner of the stack is the user program, not the JIT studied pattern or 347*22dc650dSSadaf Ebrahimianything else. The user program must ensure that if a stack is being used by 348*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b>, (that is, it is assigned to a match context that is passed 349*22dc650dSSadaf Ebrahimito the pattern currently running), that stack must not be used by any other 350*22dc650dSSadaf Ebrahimithreads (to avoid overwriting the same memory area). The best practice for 351*22dc650dSSadaf Ebrahimimultithreaded programs is to allocate a stack for each thread, and return this 352*22dc650dSSadaf Ebrahimistack through the JIT callback function. 353*22dc650dSSadaf Ebrahimi</P> 354*22dc650dSSadaf Ebrahimi<P> 355*22dc650dSSadaf Ebrahimi(4) When should a JIT stack be freed? 356*22dc650dSSadaf Ebrahimi<br> 357*22dc650dSSadaf Ebrahimi<br> 358*22dc650dSSadaf EbrahimiYou can free a JIT stack at any time, as long as it will not be used by 359*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b> again. When you assign the stack to a match context, only a 360*22dc650dSSadaf Ebrahimipointer is set. There is no reference counting or any other magic. You can free 361*22dc650dSSadaf Ebrahimicompiled patterns, contexts, and stacks in any order, anytime. 362*22dc650dSSadaf EbrahimiJust <i>do not</i> call <b>pcre2_match()</b> with a match context pointing to an 363*22dc650dSSadaf Ebrahimialready freed stack, as that will cause SEGFAULT. (Also, do not free a stack 364*22dc650dSSadaf Ebrahimicurrently used by <b>pcre2_match()</b> in another thread). You can also replace 365*22dc650dSSadaf Ebrahimithe stack in a context at any time when it is not in use. You should free the 366*22dc650dSSadaf Ebrahimiprevious stack before assigning a replacement. 367*22dc650dSSadaf Ebrahimi</P> 368*22dc650dSSadaf Ebrahimi<P> 369*22dc650dSSadaf Ebrahimi(5) Should I allocate/free a stack every time before/after calling 370*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b>? 371*22dc650dSSadaf Ebrahimi<br> 372*22dc650dSSadaf Ebrahimi<br> 373*22dc650dSSadaf EbrahimiNo, because this is too costly in terms of resources. However, you could 374*22dc650dSSadaf Ebrahimiimplement some clever idea which release the stack if it is not used in let's 375*22dc650dSSadaf Ebrahimisay two minutes. The JIT callback can help to achieve this without keeping a 376*22dc650dSSadaf Ebrahimilist of patterns. 377*22dc650dSSadaf Ebrahimi</P> 378*22dc650dSSadaf Ebrahimi<P> 379*22dc650dSSadaf Ebrahimi(6) OK, the stack is for long term memory allocation. But what happens if a 380*22dc650dSSadaf Ebrahimipattern causes stack overflow with a stack of 1MiB? Is that 1MiB kept until the 381*22dc650dSSadaf Ebrahimistack is freed? 382*22dc650dSSadaf Ebrahimi<br> 383*22dc650dSSadaf Ebrahimi<br> 384*22dc650dSSadaf EbrahimiEspecially on embedded systems, it might be a good idea to release memory 385*22dc650dSSadaf Ebrahimisometimes without freeing the stack. There is no API for this at the moment. 386*22dc650dSSadaf EbrahimiProbably a function call which returns with the currently allocated memory for 387*22dc650dSSadaf Ebrahimiany stack and another which allows releasing memory (shrinking the stack) would 388*22dc650dSSadaf Ebrahimibe a good idea if someone needs this. 389*22dc650dSSadaf Ebrahimi</P> 390*22dc650dSSadaf Ebrahimi<P> 391*22dc650dSSadaf Ebrahimi(7) This is too much of a headache. Isn't there any better solution for JIT 392*22dc650dSSadaf Ebrahimistack handling? 393*22dc650dSSadaf Ebrahimi<br> 394*22dc650dSSadaf Ebrahimi<br> 395*22dc650dSSadaf EbrahimiNo, thanks to Windows. If POSIX threads were used everywhere, we could throw 396*22dc650dSSadaf Ebrahimiout this complicated API. 397*22dc650dSSadaf Ebrahimi</P> 398*22dc650dSSadaf Ebrahimi<br><a name="SEC9" href="#TOC1">FREEING JIT SPECULATIVE MEMORY</a><br> 399*22dc650dSSadaf Ebrahimi<P> 400*22dc650dSSadaf Ebrahimi<b>void pcre2_jit_free_unused_memory(pcre2_general_context *<i>gcontext</i>);</b> 401*22dc650dSSadaf Ebrahimi</P> 402*22dc650dSSadaf Ebrahimi<P> 403*22dc650dSSadaf EbrahimiThe JIT executable allocator does not free all memory when it is possible. It 404*22dc650dSSadaf Ebrahimiexpects new allocations, and keeps some free memory around to improve 405*22dc650dSSadaf Ebrahimiallocation speed. However, in low memory conditions, it might be better to free 406*22dc650dSSadaf Ebrahimiall possible memory. You can cause this to happen by calling 407*22dc650dSSadaf Ebrahimipcre2_jit_free_unused_memory(). Its argument is a general context, for custom 408*22dc650dSSadaf Ebrahimimemory management, or NULL for standard memory management. 409*22dc650dSSadaf Ebrahimi</P> 410*22dc650dSSadaf Ebrahimi<br><a name="SEC10" href="#TOC1">EXAMPLE CODE</a><br> 411*22dc650dSSadaf Ebrahimi<P> 412*22dc650dSSadaf EbrahimiThis is a single-threaded example that specifies a JIT stack without using a 413*22dc650dSSadaf Ebrahimicallback. A real program should include error checking after all the function 414*22dc650dSSadaf Ebrahimicalls. 415*22dc650dSSadaf Ebrahimi<pre> 416*22dc650dSSadaf Ebrahimi int rc; 417*22dc650dSSadaf Ebrahimi pcre2_code *re; 418*22dc650dSSadaf Ebrahimi pcre2_match_data *match_data; 419*22dc650dSSadaf Ebrahimi pcre2_match_context *mcontext; 420*22dc650dSSadaf Ebrahimi pcre2_jit_stack *jit_stack; 421*22dc650dSSadaf Ebrahimi 422*22dc650dSSadaf Ebrahimi re = pcre2_compile(pattern, PCRE2_ZERO_TERMINATED, 0, 423*22dc650dSSadaf Ebrahimi &errornumber, &erroffset, NULL); 424*22dc650dSSadaf Ebrahimi rc = pcre2_jit_compile(re, PCRE2_JIT_COMPLETE); 425*22dc650dSSadaf Ebrahimi mcontext = pcre2_match_context_create(NULL); 426*22dc650dSSadaf Ebrahimi jit_stack = pcre2_jit_stack_create(32*1024, 512*1024, NULL); 427*22dc650dSSadaf Ebrahimi pcre2_jit_stack_assign(mcontext, NULL, jit_stack); 428*22dc650dSSadaf Ebrahimi match_data = pcre2_match_data_create(re, 10); 429*22dc650dSSadaf Ebrahimi rc = pcre2_match(re, subject, length, 0, 0, match_data, mcontext); 430*22dc650dSSadaf Ebrahimi /* Process result */ 431*22dc650dSSadaf Ebrahimi 432*22dc650dSSadaf Ebrahimi pcre2_code_free(re); 433*22dc650dSSadaf Ebrahimi pcre2_match_data_free(match_data); 434*22dc650dSSadaf Ebrahimi pcre2_match_context_free(mcontext); 435*22dc650dSSadaf Ebrahimi pcre2_jit_stack_free(jit_stack); 436*22dc650dSSadaf Ebrahimi 437*22dc650dSSadaf Ebrahimi<a name="fastpath"></a></PRE> 438*22dc650dSSadaf Ebrahimi</P> 439*22dc650dSSadaf Ebrahimi<br><a name="SEC11" href="#TOC1">JIT FAST PATH API</a><br> 440*22dc650dSSadaf Ebrahimi<P> 441*22dc650dSSadaf EbrahimiBecause the API described above falls back to interpreted matching when JIT is 442*22dc650dSSadaf Ebrahiminot available, it is convenient for programs that are written for general use 443*22dc650dSSadaf Ebrahimiin many environments. However, calling JIT via <b>pcre2_match()</b> does have a 444*22dc650dSSadaf Ebrahimiperformance impact. Programs that are written for use where JIT is known to be 445*22dc650dSSadaf Ebrahimiavailable, and which need the best possible performance, can instead use a 446*22dc650dSSadaf Ebrahimi"fast path" API to call JIT matching directly instead of calling 447*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b> (obviously only for patterns that have been successfully 448*22dc650dSSadaf Ebrahimiprocessed by <b>pcre2_jit_compile()</b>). 449*22dc650dSSadaf Ebrahimi</P> 450*22dc650dSSadaf Ebrahimi<P> 451*22dc650dSSadaf EbrahimiThe fast path function is called <b>pcre2_jit_match()</b>, and it takes exactly 452*22dc650dSSadaf Ebrahimithe same arguments as <b>pcre2_match()</b>. However, the subject string must be 453*22dc650dSSadaf Ebrahimispecified with a length; PCRE2_ZERO_TERMINATED is not supported. Unsupported 454*22dc650dSSadaf Ebrahimioption bits (for example, PCRE2_ANCHORED and PCRE2_ENDANCHORED) are ignored, as 455*22dc650dSSadaf Ebrahimiis the PCRE2_NO_JIT option. The return values are also the same as for 456*22dc650dSSadaf Ebrahimi<b>pcre2_match()</b>, plus PCRE2_ERROR_JIT_BADOPTION if a matching mode (partial 457*22dc650dSSadaf Ebrahimior complete) is requested that was not compiled. 458*22dc650dSSadaf Ebrahimi</P> 459*22dc650dSSadaf Ebrahimi<P> 460*22dc650dSSadaf EbrahimiWhen you call <b>pcre2_match()</b>, as well as testing for invalid options, a 461*22dc650dSSadaf Ebrahiminumber of other sanity checks are performed on the arguments. For example, if 462*22dc650dSSadaf Ebrahimithe subject pointer is NULL but the length is non-zero, an immediate error is 463*22dc650dSSadaf Ebrahimigiven. Also, unless PCRE2_NO_UTF_CHECK is set, a UTF subject string is tested 464*22dc650dSSadaf Ebrahimifor validity. In the interests of speed, these checks do not happen on the JIT 465*22dc650dSSadaf Ebrahimifast path. If invalid UTF data is passed when PCRE2_MATCH_INVALID_UTF was not 466*22dc650dSSadaf Ebrahimiset for <b>pcre2_compile()</b>, the result is undefined. The program may crash 467*22dc650dSSadaf Ebrahimior loop or give wrong results. In the absence of PCRE2_MATCH_INVALID_UTF you 468*22dc650dSSadaf Ebrahimishould call <b>pcre2_jit_match()</b> in UTF mode only if you are sure the 469*22dc650dSSadaf Ebrahimisubject is valid. 470*22dc650dSSadaf Ebrahimi</P> 471*22dc650dSSadaf Ebrahimi<P> 472*22dc650dSSadaf EbrahimiBypassing the sanity checks and the <b>pcre2_match()</b> wrapping can give 473*22dc650dSSadaf Ebrahimispeedups of more than 10%. 474*22dc650dSSadaf Ebrahimi</P> 475*22dc650dSSadaf Ebrahimi<br><a name="SEC12" href="#TOC1">SEE ALSO</a><br> 476*22dc650dSSadaf Ebrahimi<P> 477*22dc650dSSadaf Ebrahimi<b>pcre2api</b>(3), <b>pcre2unicode</b>(3) 478*22dc650dSSadaf Ebrahimi</P> 479*22dc650dSSadaf Ebrahimi<br><a name="SEC13" href="#TOC1">AUTHOR</a><br> 480*22dc650dSSadaf Ebrahimi<P> 481*22dc650dSSadaf EbrahimiPhilip Hazel (FAQ by Zoltan Herczeg) 482*22dc650dSSadaf Ebrahimi<br> 483*22dc650dSSadaf EbrahimiRetired from University Computing Service 484*22dc650dSSadaf Ebrahimi<br> 485*22dc650dSSadaf EbrahimiCambridge, England. 486*22dc650dSSadaf Ebrahimi<br> 487*22dc650dSSadaf Ebrahimi</P> 488*22dc650dSSadaf Ebrahimi<br><a name="SEC14" href="#TOC1">REVISION</a><br> 489*22dc650dSSadaf Ebrahimi<P> 490*22dc650dSSadaf EbrahimiLast updated: 21 February 2024 491*22dc650dSSadaf Ebrahimi<br> 492*22dc650dSSadaf EbrahimiCopyright © 1997-2024 University of Cambridge. 493*22dc650dSSadaf Ebrahimi<br> 494*22dc650dSSadaf Ebrahimi<p> 495*22dc650dSSadaf EbrahimiReturn to the <a href="index.html">PCRE2 index page</a>. 496*22dc650dSSadaf Ebrahimi</p> 497