1<?xml version="1.0" standalone="no"?> 2<!DOCTYPE s1 SYSTEM "sbk:/style/dtd/document.dtd"> 3<!-- 4 * Licensed to the Apache Software Foundation (ASF) under one 5 * or more contributor license agreements. See the NOTICE file 6 * distributed with this work for additional information 7 * regarding copyright ownership. The ASF licenses this file 8 * to you under the Apache License, Version 2.0 (the "License"); 9 * you may not use this file except in compliance with the License. 10 * You may obtain a copy of the License at 11 * 12 * http://www.apache.org/licenses/LICENSE-2.0 13 * 14 * Unless required by applicable law or agreed to in writing, software 15 * distributed under the License is distributed on an "AS IS" BASIS, 16 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 * See the License for the specific language governing permissions and 18 * limitations under the License. 19--> 20<s1 title="Overview"> 21<ul> 22<li><link anchor="purpose">Purpose of these tests</link></li> 23<li><link anchor="brief-overview">Brief overview of testing methods</link></li> 24<li><link anchor="dir-map">Directory Structure</link></li> 25<li><link anchor="test-map">Listing of Java tests and drivers</link></li> 26<li><link anchor="credits">Credits for the tests</link></li> 27</ul> 28 29 <anchor name="purpose"/> 30 <s2 title="Purpose of these tests"> 31 <p>These tests are provided for Xalan contributors to evaluate the impact of code changes. 32 Run the tests on the unchanged code, make the change and rebuild. Then run the tests again 33 and compare the results. Results are automatically compared to the files in the "-gold" 34 directory trees. Even though not all tests have "gold" files, it's still valuable to run 35 the tests before and after a code change. That way you can at least ensure that 36 your changes didn't cause any regressions to the code before you check your 37 changes in. In the future, we hope to provide a tool to compare test results from 38 one run to another (without necessarily having to re-run the test) to make this process even simpler.</p> 39 </s2> 40 41 <anchor name="brief-overview"/> 42 <s2 title="Brief overview of testing methods"> 43 <p>The Xalan tests include a richly featured and fully automated testing framework 44 that the great majority of the tests use. The org.apache.qetest package provides 45 an independent testing and automation harness for Xalan, including logging and 46 reporting mechanisims. The basic framework is easily used in testing 47 other programs as well since dependencies on Xalan and an XML parser are 48 clearly compartmented.</p> 49 <p>Nearly all tests are automated, run without any user interaction and with a minimum amount of setup, and produce 50 a rolled-up report of their pass/fail/other status. Our existing testing library includes 51 a wide array of tests, from XSLT conformance to detailed API tests, and welcomes 52 user-submitted tests as well. The 'smoketest' target (a subset of the most important 53 tests) are also integrated into the 54 <jump href="http://vmgump.apache.org/gump/public/xml-xalan/xml-xalan2-smoketest/index.html">GUMP nightly build system</jump>, 55 and thus ensure a 56 minimum baseline of functionality on a daily basis. Developers can run the smoketest 57 and ensure it passes before checking in code changes.</p> 58 </s2> 59 60 <anchor name="dir-map"/> 61 <s2 title="Directory Structure"> 62 <gloss> 63 <label>Brief overview of directory structure:</label> 64 <label><code>xml-xalan/test</code></label> 65 <item>Top level dir for all Xalan versions/products tests</item> 66 <label></label> 67 <label><code>xml-xalan/test/tools</code></label> 68 <item>Tools required by the test harness, such as JTidy. Note that all .jars 69 required to run Xalan, including Ant, the Xerces parser, etc, are 70 included in the lib and tools directories under xml-xalan/java. 71 </item> 72 <label><code>xml-xalan/test/java/src</code></label> 73 <item>Java test automation source tree - this includes 74 a generic testing framework as well as specific API tests for parts of Xalan 75 and several test drivers for testing conformance / performance / etc. over a large 76 number of xsl test stylesheets. 77 <br/>Primary packages are:<br/> 78 <jump href="apidocs/org/apache/qetest/package-summary.html">org.apache.qetest</jump><br/> 79 <jump href="apidocs/org/apache/qetest/xsl/package-summary.html">org.apache.qetest.xsl</jump><br/> 80 <jump href="apidocs/org/apache/qetest/trax/package-summary.html">org.apache.qetest.trax</jump><br/> 81 <jump href="apidocs/org/apache/qetest/trax/dom/package-summary.html">org.apache.qetest.trax.dom</jump><br/> 82 <jump href="apidocs/org/apache/qetest/trax/stream/package-summary.html">org.apache.qetest.trax.stream</jump><br/> 83 <jump href="apidocs/org/apache/qetest/trax/sax/package-summary.html">org.apache.qetest.trax.sax</jump><br/> 84 <jump href="apidocs/org/apache/qetest/xalanj2/package-summary.html">org.apache.qetest.xalanj2</jump><br/> 85 <jump href="apidocs/org/apache/qetest/dtm/package-summary.html">org.apache.qetest.dtm</jump><br/> 86 <br/></item> 87 <label><code>xml-xalan/test/tests</code></label><item>Top level for XSLT stylesheet trees and special API tests</item> 88 <label><code>xml-xalan/test/tests/conf</code></label><item>Directory tree of specific conformance testing stylesheets</item> 89 <label><code>xml-xalan/test/tests/conf-gold</code></label><item>Directory tree of specific conformance testing stylesheets gold 90 output reference files (this tree should mirror the structure of contrib)<br/></item> 91 <label><code>xml-xalan/test/tests/contrib</code></label><item>Directory tree of user-contributed stylesheets</item> 92 <label><code>xml-xalan/test/tests/contrib-gold</code></label><item>Directory tree of user-contributed stylesheets gold 93 output reference files (this tree should mirror the structure of contrib)<br/></item> 94 <label><code>xml-xalan/test/tests/api</code></label><item>Directory tree for stylesheets used in Java API tests</item> 95 <label><code>xml-xalan/test/tests/api/trax</code></label><item>Stylesheets used in Java API tests in 96 <jump href="apidocs/org/apache/qetest/trax/package-summary.html">org.apache.qetest.trax</jump></item> 97 <label><code>xml-xalan/test/tests/api/trax/dom</code></label><item>Stylesheets used in Java API tests in 98 <jump href="apidocs/org/apache/qetest/trax/dom/package-summary.html">org.apache.qetest.trax.dom</jump></item> 99 <label></label><item>etc. - often the directory tree in the stylesheet area 100 will match the Java sources directory/package tree.</item> 101 <label><code>xml-xalan/test/tests/api-gold</code></label><item>Matching Directory tree of gold files for Java API tests<br/></item> 102 <label><code>xml-xalan/test/tests/extensions</code></label><item>Directory tree for stylesheets used in Xalan-specific extension tests</item> 103 <label><code>xml-xalan/test/tests/extensions/java</code></label><item>Tests for extensions written in Java</item> 104 <label><code>xml-xalan/test/tests/extensions/javascript</code></label><item>Tests for extensions written in Javascript</item> 105 <label><code>xml-xalan/test/tests/extension-gold</code></label><item>Matching Directory tree of gold files for extensions tests<br/></item> 106 <label><code>xml-xalan/test/tests/bugzilla</code></label><item>Special directory of stylesheets and automated Testlets reproducing selected Bugzilla bug reports</item> 107 </gloss> 108 </s2> 109 110 <anchor name="test-map"/> 111 <s2 title="Listing of Java tests and drivers"> 112 <note>This section is a sort of catalog of existing tests. Beginning users 113 will probably want to see <link idref="run" anchor="how-to-run">how to run tests</link> as well.</note> 114<p>Java Test Drivers (data driven testing)</p> 115<p>A Java Test Driver executes a test for each xml/xsl file pair in 116the specified directory tree or each pair in the specified fileList. 117The driver iterates over the inputDir tree or list of files 118and asks a Testlet to execute() a test on each one. This is also similar to 119data driven testing, where a common algorithim is defined for a test case, and 120then a large number of data points (in this case, the xml/xsl file pairs) are run through the test case in order. 121The best example is <jump href="apidocs/org/apache/qetest/xsl/StylesheetTestletDriver.html">StylesheetTestletDriver</jump>. 122Another generic example is <jump href="apidocs/org/apache/qetest/FileTestletDriver.html">FileTestletDriver</jump>.</p> 123<p>The Test Drivers rely on various Testlet implementations 124to define the actual testing algorithim to apply to each xml/xsl 125file pair. This defines any options to be used when processing the 126file as well as logging out information about the test in progress. 127Examples include 128<jump href="apidocs/org/apache/qetest/xsl/StylesheetTestlet.html">StylesheetTestlet</jump> and 129<jump href="apidocs/org/apache/qetest/xsl/PerformanceTestlet.html">PerformanceTestlet</jump></p> 130<p>The Testlets rely on <jump href="apidocs/org/apache/qetest/xslwrapper/TransformWrapper.html">TransformWrapper</jump> 131subclasses to perform the actual test of processing or transformation 132of the xml/xsl file pair into the output file. We can then plug 133in different TransformWrapper "flavors" easily. Different 134TransformWrapper can process or transform in various ways, like 135using DOM trees, SAX events, or input/output streams.</p> 136<p>The three levels of iteration, test algorithim, and 137processor flavor are all independently changeable, so we can 138easily try out different kinds of tests. This technique is used to 139run the full sets of 'conf' (conformance), 'perf' (performance), 140extensions, and 'contrib' (user contributed stylesheets) tests.</p> 141<gloss> 142<label>org.apache.qetest.xsl.<link idref="run" anchor="how-to-run-c">XalanCTestlet</link></label> 143<item>This is similar to the StylesheetTestlet, but for Xalan-C. 144It simply shells 145out to a command prompt to run each stylesheet through the TestXSLT.exe program 146from Xalan-C.</item> 147</gloss> 148 149<p>Java API tests for the TRAX (or javax.xml.transform) interface, that 150Xalan-J 2.x implements.<br/> 151All in package: org.apache.qetest.trax</p> 152<note>(This Section needs updating: many new tests have been added; see the Javadoc for a list -sc)</note> 153<gloss> 154<label>REPLACE_template_for_new_tests.java</label> 155<item>a template for creating new TRAX API tests, see <link idref="submit" anchor="write-API-tests">Submitting New Tests</link></item> 156<label>LoggingErrorListener.java</label> 157<item><ref>utility:</ref> wraps javax.xml.transform.ErrorListener, and logs info; 158this class also supports setting expected errors to trap, and it will call 159logger.checkPass/checkFail for you when it gets an expected or unexpected event. 160This allows us to write very detailed negative tests and have them be 161fully automated.</item> 162<label>LoggingURIResolver.java</label> 163<item><ref>utility:</ref> wraps javax.xml.transform.URIResolver, and logs info</item> 164<label>ExamplesTest.java</label> 165<item>A testing version of samples/trax/Examples.java, a sample file 166provided in Xalan-J 2.x showing various uses of the TRAX or 167javax.xml.transform API to process stylesheets.</item> 168<label>TransformerAPITest.java</label> 169<item>API coverage tests for javax.xml.transform.Transformer</item> 170<label>TransformerFactoryAPITest.java</label> 171<item>API coverage tests for javax.xml.transform.TransformerFactory</item> 172<label>TemplatesAPITest.java</label> 173<item>API coverage tests for javax.xml.transform.Templates</item> 174 175<label>EmbeddedStylesheetTest.java</label> 176<item>Testing various types and kinds of stylesheets embedded with the xml-stylesheet PI</item> 177<label>ErrorListenerAPITest.java</label> 178<item>API Coverage test for ErrorListener</item> 179<label>ErrorListenerTest.java</label> 180<item>Functionality test of error listeners when using illegal stylesheets</item> 181<label>OutputPropertiesTest.java</label> 182<item>Various tests of programmatic access and changing of output properties</item> 183<label>SystemIdImpInclTest.java</label> 184<item>Testing various forms of URLs in setSystemID with imported and included stylesheets</item> 185<label>SystemIdTest.java</label> 186<item>Testing various forms of URLs in setSystemID</item> 187 188 189<label>TestThreads.java</label> 190<item>MANUALLY executed test for running multiple threads 191and transforming multiple stylesheets simultaneously. An updated and automated 192test is now available, org.apache.qetest.xsl.ThreadedTestletDriver, which 193should be used instead. </item> 194</gloss> 195 196<p>All in subpackages of: org.apache.qetest.trax</p> 197<gloss> 198<label>stream.StreamSourceAPITest.java</label> 199<item>API coverage tests for javax.xml.transform.stream.StreamSource</item> 200<label>stream.StreamResultAPITest.java</label> 201<item>API coverage tests for javax.xml.transform.stream.StreamResult</item> 202 203<label>etc.</label> 204<item>API coverage tests are available with a similar naming scheme for most javax.xml.transform.* classes</item> 205</gloss> 206 207 208<p>A few tests are ones that Xalan does not currently pass due to Bugzilla reports, but we know the 209correct ("gold") result by analysis or by trying the test on other processors. 210A number of tests may also be missing matching "gold" files, if we haven't 211yet had time to confirm the correct output. It's still useful to run these 212tests (although the test driver will report an AMBG or 'Ambiguous' 213result) because you can still see if the output looks basically correct, and 214compare the output to previous test runs before you submit your code changes, etc.</p> 215<p>The tests have several different types of results beyond just pass or fail, 216which are documented in <jump href="apidocs/org/apache/qetest/Logger.html#PASS_RESULT">org.apache.qetest.Logger</jump>. 217</p> 218 219 </s2> 220 221 <anchor name="credits"/> 222 <s2 title="Credits for the tests"> 223 <ul> 224 <li><jump href="mailto:[email protected]">Shane Curcuru</jump></li> 225 <li><jump href="mailto:[email protected]">Paul Dick</jump></li> 226 <li><jump href="mailto:[email protected]">David Marston</jump></li> 227 <li><jump href="mailto:[email protected]">Tom Amiro</jump></li> 228 <li><jump href="mailto:[email protected]">Gary L Peskin</jump></li> 229 <li>Many other <jump href="http://xml.apache.org/mail.html">xalan-dev</jump> subscribers</li> 230 <li>Test cases written by Carmelo Montanez at <jump href="http://www.nist.gov/xml/">NIST</jump> for general use</li> 231 <li>Many other helpers who we still need to credit! Sorry!</li> 232 </ul> 233 </s2> 234 235</s1> 236