1Tests naming convention 2----------------------- 3Names of tests (suffixes of control.* files) are build from 2 or 3 words 4separated by an underscore character. The first one is 'core' or 'ext', what 5defines the set of PPD files to test. These two are described in the section 6below. The second word of the name is either 'test' or 'dump'. The third and 7optional word is 'debug'. All these three words influence the testing 8procedure used by the autotest, their meanings are described in the next 9paragraphs. 10 11Input sets of PPD files ('core' and 'ext') 12------------------------------------------ 13There are two input sets of PPD files to test: 'core' and 'ext'. 'core' is the 14smaller one and represents clusters defined in the file large_clusters.txt; it 15is built by taking the first element (PPD filename) from every line of this 16file. The set 'ext' is built in similar way by taking the first element (PPD 17filename) from every line of small_clusters.txt file; however the elements 18already present in 'core' are omitted. Currently, 'core' contains around ~130 19PPD files; the exact size equals the number of lines in large_clusters.txt. 20The set 'ext' contains around ~1400 PPD files; the exact size equals the number 21of lines in small_clusters.txt minus the number of lines in large_clusters.txt. 22 23Overall testing procedure 24------------------------- 25The purpose of this autotest is to verify that given subset of PPD files work 26in ChromeOS. Each PPD file is tested with the following procedure: 271. A printer driver is added to CUPS server. 282. Test documents are printed on the configured printer. 293. Raw output from the CUPS server is intercepted by, so called, FakePrinter. 304. CUPS logs are parsed to make sure that no errors occured. 315. Obtained outputs are verified (see below) - test only. 326. A printing pipeline used by CUPS is rerun in shell (see below) - debug only. 337. All obtained outputs & logs are saved on the device (see below) - dump only. 348. The printer driver is removed from CUPS server. 35This procedure is repeated for every PPD file. The number of PPD files may be 36large (~2K files). To decrease amount of time needed by the autotest, several 37PPD files are tested simultaneously in parallel threads. Autotest always run 38the procedure for all given PPD files and print a summary report at the end. 39If at least one of PPD files fails, whole autotest is finished with failure 40(but always all PPD files are processed). 41 42Output verification ('test') 43---------------------------- 44Intercepted output is verified by comparision with the previous results 45obtained for the same PPD. We cannot store outputs directly, because their 46total size may have hundreds of megabytes. Instead of that short digest is 47calculated for each obtained document and it is used for comparison. 48A function for digests calculation is in the 'helpers.py' file. Not all 49outputs can be tested this way because for some PPD files produced contents 50differ between runs. List of PPD files for which we cannot calculate 51constant digest is saved in the file digests_denylist.txt. Files with 52expected digests for every test document are stored in the directory "digests". 53If a digests for given pair (test document, PPD file) is missing, the test 54checks only check if the output is not empty (or not too short). 55 56Rerun printing pipeline ('debug') 57--------------------------------- 58Every PPD file defines printing pipeline used by CUPS, consisting of external 59binaries (so called filters). To make sure that every part of the printing 60pipeline works correctly, it can be run outside CUPS in Linux shell. In tests 61with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In 62this mode every document is processed twice: the first time by CUPS and the 63second time by extracted pipeline run as bash script. Additionally, each 64pipeline's step is verified by checking its return code (it is supposed to be 65equal 0) and the final output is compared with the output returned by CUPS 66server. If the test is also defined as 'dump', each intermediate content is 67saved to the directory given in path_outputs parameter. 68This mode requires more disk space on tested device (~2GB in /tmp) and takes 69more execution time (~2 times more). All information needed to reconstruct 70a pipeline used by CUPS can are extracted from CUPS logs after the job is 71completed. To do that, CUPS configuration must be modified: the LogLevel 72option in /etc/cups/cupsd.conf must be set to "debug". This modification is 73done when the autotest is initialized; the original setting is restored in 74the cleaup stage. To make these modifications, 'debug' tests remount the root 75partition in read-write mode. The partition is NOT remounted back to read-only 76mode. 77 78Save outputs and logs ('dump') 79------------------------------ 80All obtained outputs and logs are saved on the device in the directory 81/tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix 82are grouped together and stored in single archive to limit usage of disk space. 83 84Test parameters 85--------------- 86path_docs - path to directory with test documents (PDF files) 87path_ppds - path to directory with PPD files, it is supposed to be compressed 88 as .tar.xz files (with a command "tar cJf ...") 89path_digests - path to directory with files containing digests for 90 verification, if not set then outputs are not verified 91path_outputs - if set, then all outputs are dumped there (given directory is 92 deleted if already exists); also all digests files are recalculated 93 and saved in the same directory 94debug_mode - True or False, if True then for every test the printing pipeline 95 is extracted from CUPS logs and executed again outside CUPS 96 97Generating new digests 98---------------------- 99The following procedure can be used to update digests: 1001. Run the test defined in control.all_dump: 101 test_that <device IP> platform_PrinterPpds.all_dump 1022. Download generated files with digests to your workstation 103 rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir> 1043. Replace the files from the "digests" directory and commit changes 105 106Updating the archives with PPD files 107------------------------------------ 108Currently, all tests are based on PPD files stored in local directories. The 109autotest can download all PPD files by itself, but we do not use this option 110to limit the number of possible points of failures during testing. Archives 111with PPD files are prepared with ppdTool.go: 1121. Delete old files: 113 rm ppds_core.tar.xz ppds_ext.tar.xz large_clusters.txt small_clusters.txt 1142. Download all PPD files to ppds_all directory: 115 go run ppdTool.go download 1163. Calculate new clusters: 117 go run ppdTool.go compare 118 If it fails with an error "too many open files" you have to increase the 119 soft limit of open files with ulimit, e.g.: 120 ulimit -Sn 10123 1214. Compress new directories with PPD files: 122 tar cJf ppds_core.tar.xz ppds_core 123 tar cJf ppds_ext.tar.xz ppds_ext 124 125Others 126------ 127* How to get comma-separated list of all used ghostscript devices? 128 1. Go to the directory generated by all_dump_debug 129 2. Unpack all tar.xz archives (not on the device) 130 for d in *.pdf; 131 do 132 cd $d; 133 for f in *.tar.xz; do tar xf $f; done ; 134 cd ..; 135 done 136 3. Run the following 137 grep -o sDEVICE=[^\ ]*\ ./*.pdf/*.err[12345] | cut -d \= -f 2 | sort | uniq | tr -d ' ' | tr '\n' ',' 138 4. Add to the list uniprint (it is not caught by the command above) 139* How to get a list of all used cups filters? 140 1. Go to the directory generated by all_dump_debug 141 2. Unpack all tar.xz archives (not on the device) 142 for d in *.pdf; 143 do 144 cd $d; 145 for f in *.tar.xz; do tar xf $f; done ; 146 cd ..; 147 done 148 3. Run the following 149 grep -o '(exec -a "Fake[^"]*" [^ ]* ' ./*.pdf/*.sh | cut -d \ -f 4 | sort | uniq 150