1<?xml version="1.0" standalone="no"?> 2<!DOCTYPE s1 SYSTEM "sbk:/style/dtd/document.dtd"> 3<!-- 4 * Licensed to the Apache Software Foundation (ASF) under one 5 * or more contributor license agreements. See the NOTICE file 6 * distributed with this work for additional information 7 * regarding copyright ownership. The ASF licenses this file 8 * to you under the Apache License, Version 2.0 (the "License"); 9 * you may not use this file except in compliance with the License. 10 * You may obtain a copy of the License at 11 * 12 * http://www.apache.org/licenses/LICENSE-2.0 13 * 14 * Unless required by applicable law or agreed to in writing, software 15 * distributed under the License is distributed on an "AS IS" BASIS, 16 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 * See the License for the specific language governing permissions and 18 * limitations under the License. 19--> 20<s1 title="Running Tests"> 21<ul> 22<li><link anchor="how-to-run">How-to: Run Xalan-J tests</link></li> 23<li><link anchor="how-to-view-results">How-to: View Test Results</link></li> 24<li><link anchor="test-options">Common Test Options</link></li> 25<li><link anchor="how-to-run-c">How-to: Run Xalan-C tests</link></li> 26</ul> 27 28 <anchor name="how-to-run"/> 29 <s2 title="How-to: Run tests"> 30 <p>Nearly all tests for Xalan are independent Java classes built 31 into testxsl.jar that 32 can be run either individually on the command line, programmatically 33 from your application or our handy Ant build.xml file, or in batches from 34 <jump href="apidocs/org/apache/qetest/xsl/XSLTestHarness.html">XSLTestHarness</jump>. 35 There really isn't any magic to them: you can just set your classpath and 36 execute java.exe to run them; some Tests and Testlets currently provide defaults 37 for their inputs, so you can run them without any setup at all. 38 However we have provided a couple of more 39 convenient ways to run the most common tests:</p> 40 <note>If you need to debug into the tests themselves, 41 you may want to use the <link idref="faq" anchor="debug">debug*.bat files</link></note> 42 <p>Of course, first <link idref="getstarted" anchor="how-to-build">Build a fresh copy of testxsl.jar.</link> 43 </p> 44 <p>cd xml-xalan\test<br/></p> 45 <p>You can either: use the Ant build.xml script; run a convenience batch file; or execute java.exe yourself.</p> 46 47 <p> 48 <code>build conf [<link anchor="test-options">Ant-prefixed options</link>]</code> 49 <br/>(runs StylesheetTestletDriver over tests\conf test tree using the default StylesheetTestlet)<br/><br/> 50 <code>build perf [<link anchor="test-options">Ant-prefixed options</link>]</code> 51 <br/>(runs StylesheetTestletDriver over tests\perf test tree using the default PerformanceTestlet)<br/><br/> 52 <code>build api -DtestClass=TRAXAPITestClassName [<link anchor="test-options">Ant-prefixed options</link>]</code> 53 <br/>(runs TRAX interface tests with Xalan-J 2.x<br/> 54 </p> 55 <p>Alternately: some convenience batch files are provided for the most common 56 tests - these simply turn around and call build.bat/.sh for you.<br/> 57 (Namely conf.bat, perf.bat, and the like)</p> 58 <p>Alternately: Run java.exe and the desired test class yourself:<br/></p> 59 <note>Running tests with alternate JAXP parsers: all org.apache.qetest.trax.* 60 tests can be run with Xalan-J 2.x and any JAXP 1.1 compatible parser, like 61 crimson.jar. Be sure to manually set the appropriate system properties to use 62 your parser instead of xerces.jar, which is the default for Xalan-J. Tests will 63 also run on any JAXP 1.1 compatible xslt processor, namely either our default 64 xalan one or our new xsltc one.</note> 65 </s2> 66 67 <anchor name="how-to-view-results"/> 68 <s2 title="How-to: View Test Results"> 69 <p>Most tests both send basic results to System.out, as well as 70 writing a full set of output results to their <code><link anchor="test-options-logfile">logFile</link></code>, as 71 set from a .properties file or the command line. Generally the 72 output results file is easier to deal with. The basic format is 73 fairly simple and you can certainly read it as-is. Also, many tests 74 send only summary results to the console, but full output to the results file.</p> 75 <p>To 'pretty-print' results or produce reports, please use the 76 viewResults.xsl stylesheet and associated batch/shell files, like so:<br/> 77 <code>cd \xml-xalan\test</code><br/><br/> 78 <code>build conf -DoptionName=optionValue <link anchor="test-options-logfile">-Dqetest.logFile results/MyResults.xml</link></code><br/><br/> 79 <code>viewResults.bat results/MyResults.xml results/MyPrettyResults.html [options]</code><br/><br/> 80 These options are passed to Xalan's command line to transform the 81 xml formatted output results into a nicer-looking html file. The most 82 common option would be <code>-param loggingLevel 99</code> to get 83 more output shown in the results (higher numbers up to 99 show more details, 84 lower numbers, down to 0, show fewer details). 85 </p> 86 <p>Alternatively, the tableResults.xsl stylesheet can be used to pretty-print 87 results from the conformance test for multiple TRAX flavours:<br/><br/> 88 <code>cd \xml-xalan\test</code><br/><br/> 89 <code>build alltest.conf</code><br/><br/> 90 <code>set RESULTSCANNER=tableResults.xsl</code><br/><br/> 91 <code>viewResults.bat results-alltest/conf/sax/results.xml results.html [options]</code><br/><br/> 92 This will generate a pretty-printed HTML table in results.html, listing all 93 flavours as well as the results of individual test cases for each category. 94 The options can be passed to tableResults.xsl to generate conformance reports 95 on previously run tests, or to compare two runs of the conformance test suite. 96 </p> 97 <p>Possible options that can be passed to tableResults.xsl: 98 <ul> 99 <li> 100 <code>-param resultsDir /path/to/result/results-alltest/conf</code><br/><br/> 101 Passing the path to the <code>results-alltest/conf</code> directory generates 102 a report based on the results in that directory. Defaults to <code>./results-alltest/conf</code><br/><br/> 103 </li> 104 <li> 105 <code>-param compareAgainst /path/to/other/results-alltest/conf</code><br/><br/> 106 Passing the path to anothe <code>results-alltest/conf</code> directory compares 107 that result against the result pointed to by resultsDir.<br/><br/> 108 </li> 109 <li> 110 <code>-param resultsDir results-alltest.xsltc/conf</code><br/><br/> 111 Generates a report on the results of an xsltc test (with the <code>alltest.conf.xsltc</code>) target. 112 </li> 113 </ul> 114 </p> 115 <p>Future work includes greatly updated results analysis stylesheets. 116 See FailScanner.xsl and PerfScanner.xsl for ideas. An important design 117 principle in the tests is that at runtime the tests merely output 118 whatever data they can as results are generated; afterwards, we then 119 post-process the results into whatever presentation format is needed, 120 including perhaps re-calculating overall results (example: if we have 121 a list of known fails correlated to JIRA numbers, then a stylesheet 122 could filter out these fails and report them as known bugs instead. 123 </p> 124 </s2> 125 <anchor name="test-options"/> 126 <s2 title="Common Test Options"> 127 <note>Section needs updating to reflect the fact that while the options the 128 tests see remain as below, the user must now prefix all optionNames with 129 'qetest.' or 'conf.', etc. when passing the options on the command line or 130 via my.test.properties or test.properties -sc</note> 131 <p>Most tests can either accept options from a properties file, via:<br/> 132 <code> TestName -load file.properties</code><br/><br/> 133 or simply from the command line (which overrides the properties file) like:<br/> 134 <code> TestName -arg1 value1 -arg2 -arg3 value3</code><br/><br/></p> 135 <p>To see all options, call a test with an illegal argument to force it 136 to print out it's .usage(). You may mix setting options from a properties 137 file and from the command line; command line options will take precedence.</p> 138 <p>For another description of options, see <br/><code>xml-xalan\test\test.properties</code>,<br/> 139 which describes most of them with comments. Remember that the prefixes 140 'qetest.', 'conf.' etc. are used by the Ant build.xml file to manage which 141 properties are used for different kinds of tests, and are ripped off before 142 being passed to the Java test script itself. Thus qetest.loggingLevel=99 in 143 the test.properties file becomes just loggingLevel of 99 when passed to the test.</p> 144 <note>Path-like options set in a properties file generally should use 145 forward slashes (legal in URL's), even on Windows platforms.</note> 146 <p>Quick list of options</p> 147 <anchor name="test-options-logfile"/> 148 <gloss> 149 <label>-logFile <ref>resultsFileName.xml</ref></label> 150 <item>sends test results to an XML-based results file</item> 151 <label>-loggingLevel <ref>nn</ref></label> 152 <item>determines how much information is sent to your logFile, 0=very little, 99=lots</item> 153 <label>-ConsoleLogger.loggingLevel <ref>nn</ref></label> 154 <item>determines how much information is sent just to the default ConsoleLogger: 155 since often you won't be watching the console as the test is running, you can set this 156 lower than your loggingLevel to speed the tests up a little</item> 157 <label>-inputDir <ref>path/to/tests</ref></label> 158 <item>path to a directory tree of input *.xml/*.xsl files, using your system's separator</item> 159 <label>-outputDir <ref>path/to/output/area</ref></label> 160 <item>where all output is sent</item> 161 <label>-goldDir <ref>path/to/gold</ref></label> 162 <item>path to a directory tree of reference output - this tree should be 163 a parallel structure to the inputDir tree</item> 164 <label>-category <ref>dirName</ref></label> 165 <item>only run this single named subdir within inputDir</item> 166 <label>-excludes <ref>'test1.xsl;test2.xsl'</ref></label> 167 <item>will skip running any specifically named tests; do not use any path elements here</item> 168 <label>-flavor <ref>xalan|trax|trax.d2d</ref></label> 169 <item>which kind/flavor of Processor to test; see 170 <jump href="apidocs/org/apache/qetest/xslwrapper/ProcessorWrapper.html">ProcessorWrapper.java</jump> </item> 171 <label>-testlet <ref>TestletClassname</ref></label> 172 <item>For StylesheetTestletDriver, use a different class for the testing algorithim</item> 173 <label>-load <ref>file.properties</ref></label> 174 <item>(read in a .properties file, that can set any/all of the other opts) 175 This option is automatically used by the Ant build.xml file, so you normally 176 would not specify it yourself.</item> 177 </gloss> 178 <p>Note that most options work equivalently with either Xalan-J or Xalan-C tests.</p> 179 <p>When running tests using Ant, the <xalantest> task actually 180 marshalls various Ant variables and uses the precompiled org.apache.qetest.xsl.XSLTestAntTask 181 to actually execute the test (either in the same JVM or forked depending on 182 ${fork-tests}. It simply strips the appropriate set of prefixes 183 off of required Ant variables and dumps them into an XSLTestAntTask.properties 184 file on disk, which it tells the test executing to -load.</p> 185 </s2> 186 187 <anchor name="how-to-run-c"/> 188 <s2 title="How-to: Run Xalan-C tests"> 189 <p>In progress. A few C++ API tests are checked into the <code>xml-xalan/c/Tests</code> 190 repository area already. To execute any set of 'conformance' tests with the 191 Xalan-C processor, we currently use the 192 org.apache.qetest.xsl.<jump href="apidocs/org/apache/qetest/xsl/XalanCTestlet.html">XalanCTestlet</jump> 193 driver. This is written in Java to take advantage of the framework and 194 results reporting, but basically constructs a command line for each test 195 and then shells out to <code>TestXSLT.exe -in file.xsl...</code> to run the test; 196 it then uses the same validation routines as the Java ConformanceTest.</p> 197 </s2> 198</s1> 199