1# Distributed under the OSI-approved BSD 3-Clause License. See accompanying 2# file Copyright.txt or https://cmake.org/licensing for details. 3 4#[=======================================================================[.rst: 5ExternalData 6------------ 7 8.. only:: html 9 10 .. contents:: 11 12Manage data files stored outside source tree 13 14Introduction 15^^^^^^^^^^^^ 16 17Use this module to unambiguously reference data files stored outside 18the source tree and fetch them at build time from arbitrary local and 19remote content-addressed locations. Functions provided by this module 20recognize arguments with the syntax ``DATA{<name>}`` as references to 21external data, replace them with full paths to local copies of those 22data, and create build rules to fetch and update the local copies. 23 24For example: 25 26.. code-block:: cmake 27 28 include(ExternalData) 29 set(ExternalData_URL_TEMPLATES "file:///local/%(algo)/%(hash)" 30 "file:////host/share/%(algo)/%(hash)" 31 "http://data.org/%(algo)/%(hash)") 32 ExternalData_Add_Test(MyData 33 NAME MyTest 34 COMMAND MyExe DATA{MyInput.png} 35 ) 36 ExternalData_Add_Target(MyData) 37 38When test ``MyTest`` runs the ``DATA{MyInput.png}`` argument will be 39replaced by the full path to a real instance of the data file 40``MyInput.png`` on disk. If the source tree contains a content link 41such as ``MyInput.png.md5`` then the ``MyData`` target creates a real 42``MyInput.png`` in the build tree. 43 44Module Functions 45^^^^^^^^^^^^^^^^ 46 47.. command:: ExternalData_Expand_Arguments 48 49 The ``ExternalData_Expand_Arguments`` function evaluates ``DATA{}`` 50 references in its arguments and constructs a new list of arguments:: 51 52 ExternalData_Expand_Arguments( 53 <target> # Name of data management target 54 <outVar> # Output variable 55 [args...] # Input arguments, DATA{} allowed 56 ) 57 58 It replaces each ``DATA{}`` reference in an argument with the full path of 59 a real data file on disk that will exist after the ``<target>`` builds. 60 61.. command:: ExternalData_Add_Test 62 63 The ``ExternalData_Add_Test`` function wraps around the CMake 64 :command:`add_test` command but supports ``DATA{}`` references in 65 its arguments:: 66 67 ExternalData_Add_Test( 68 <target> # Name of data management target 69 ... # Arguments of add_test(), DATA{} allowed 70 ) 71 72 It passes its arguments through ``ExternalData_Expand_Arguments`` and then 73 invokes the :command:`add_test` command using the results. 74 75.. command:: ExternalData_Add_Target 76 77 The ``ExternalData_Add_Target`` function creates a custom target to 78 manage local instances of data files stored externally:: 79 80 ExternalData_Add_Target( 81 <target> # Name of data management target 82 [SHOW_PROGRESS <ON|OFF>] # Show progress during the download 83 ) 84 85 It creates custom commands in the target as necessary to make data 86 files available for each ``DATA{}`` reference previously evaluated by 87 other functions provided by this module. 88 Data files may be fetched from one of the URL templates specified in 89 the ``ExternalData_URL_TEMPLATES`` variable, or may be found locally 90 in one of the paths specified in the ``ExternalData_OBJECT_STORES`` 91 variable. 92 93 .. versionadded:: 3.20 94 The ``SHOW_PROGRESS`` argument may be passed to suppress progress information 95 during the download of objects. If not provided, it defaults to ``OFF`` for 96 :generator:`Ninja` and :generator:`Ninja Multi-Config` generators and ``ON`` 97 otherwise. 98 99 Typically only one target is needed to manage all external data within 100 a project. Call this function once at the end of configuration after 101 all data references have been processed. 102 103Module Variables 104^^^^^^^^^^^^^^^^ 105 106The following variables configure behavior. They should be set before 107calling any of the functions provided by this module. 108 109.. variable:: ExternalData_BINARY_ROOT 110 111 The ``ExternalData_BINARY_ROOT`` variable may be set to the directory to 112 hold the real data files named by expanded ``DATA{}`` references. The 113 default is ``CMAKE_BINARY_DIR``. The directory layout will mirror that of 114 content links under ``ExternalData_SOURCE_ROOT``. 115 116.. variable:: ExternalData_CUSTOM_SCRIPT_<key> 117 118 .. versionadded:: 3.2 119 120 Specify a full path to a ``.cmake`` custom fetch script identified by 121 ``<key>`` in entries of the ``ExternalData_URL_TEMPLATES`` list. 122 See `Custom Fetch Scripts`_. 123 124.. variable:: ExternalData_LINK_CONTENT 125 126 The ``ExternalData_LINK_CONTENT`` variable may be set to the name of a 127 supported hash algorithm to enable automatic conversion of real data 128 files referenced by the ``DATA{}`` syntax into content links. For each 129 such ``<file>`` a content link named ``<file><ext>`` is created. The 130 original file is renamed to the form ``.ExternalData_<algo>_<hash>`` to 131 stage it for future transmission to one of the locations in the list 132 of URL templates (by means outside the scope of this module). The 133 data fetch rule created for the content link will use the staged 134 object if it cannot be found using any URL template. 135 136.. variable:: ExternalData_NO_SYMLINKS 137 138 .. versionadded:: 3.3 139 140 The real data files named by expanded ``DATA{}`` references may be made 141 available under ``ExternalData_BINARY_ROOT`` using symbolic links on 142 some platforms. The ``ExternalData_NO_SYMLINKS`` variable may be set 143 to disable use of symbolic links and enable use of copies instead. 144 145.. variable:: ExternalData_OBJECT_STORES 146 147 The ``ExternalData_OBJECT_STORES`` variable may be set to a list of local 148 directories that store objects using the layout ``<dir>/%(algo)/%(hash)``. 149 These directories will be searched first for a needed object. If the 150 object is not available in any store then it will be fetched remotely 151 using the URL templates and added to the first local store listed. If 152 no stores are specified the default is a location inside the build 153 tree. 154 155.. variable:: ExternalData_SERIES_PARSE 156 ExternalData_SERIES_PARSE_PREFIX 157 ExternalData_SERIES_PARSE_NUMBER 158 ExternalData_SERIES_PARSE_SUFFIX 159 ExternalData_SERIES_MATCH 160 161 See `Referencing File Series`_. 162 163.. variable:: ExternalData_SOURCE_ROOT 164 165 The ``ExternalData_SOURCE_ROOT`` variable may be set to the highest source 166 directory containing any path named by a ``DATA{}`` reference. The 167 default is ``CMAKE_SOURCE_DIR``. ``ExternalData_SOURCE_ROOT`` and 168 ``CMAKE_SOURCE_DIR`` must refer to directories within a single source 169 distribution (e.g. they come together in one tarball). 170 171.. variable:: ExternalData_TIMEOUT_ABSOLUTE 172 173 The ``ExternalData_TIMEOUT_ABSOLUTE`` variable sets the download 174 absolute timeout, in seconds, with a default of ``300`` seconds. 175 Set to ``0`` to disable enforcement. 176 177.. variable:: ExternalData_TIMEOUT_INACTIVITY 178 179 The ``ExternalData_TIMEOUT_INACTIVITY`` variable sets the download 180 inactivity timeout, in seconds, with a default of ``60`` seconds. 181 Set to ``0`` to disable enforcement. 182 183.. variable:: ExternalData_URL_ALGO_<algo>_<key> 184 185 .. versionadded:: 3.3 186 187 Specify a custom URL component to be substituted for URL template 188 placeholders of the form ``%(algo:<key>)``, where ``<key>`` is a 189 valid C identifier, when fetching an object referenced via hash 190 algorithm ``<algo>``. If not defined, the default URL component 191 is just ``<algo>`` for any ``<key>``. 192 193.. variable:: ExternalData_URL_TEMPLATES 194 195 The ``ExternalData_URL_TEMPLATES`` may be set to provide a list 196 of URL templates using the placeholders ``%(algo)`` and ``%(hash)`` 197 in each template. Data fetch rules try each URL template in order 198 by substituting the hash algorithm name for ``%(algo)`` and the hash 199 value for ``%(hash)``. Alternatively one may use ``%(algo:<key>)`` 200 with ``ExternalData_URL_ALGO_<algo>_<key>`` variables to gain more 201 flexibility in remote URLs. 202 203Referencing Files 204^^^^^^^^^^^^^^^^^ 205 206Referencing Single Files 207"""""""""""""""""""""""" 208 209The ``DATA{}`` syntax is literal and the ``<name>`` is a full or relative path 210within the source tree. The source tree must contain either a real 211data file at ``<name>`` or a "content link" at ``<name><ext>`` containing a 212hash of the real file using a hash algorithm corresponding to ``<ext>``. 213For example, the argument ``DATA{img.png}`` may be satisfied by either a 214real ``img.png`` file in the current source directory or a ``img.png.md5`` 215file containing its MD5 sum. 216 217.. versionadded:: 3.8 218 Multiple content links of the same name with different hash algorithms 219 are supported (e.g. ``img.png.sha256`` and ``img.png.sha1``) so long as 220 they all correspond to the same real file. This allows objects to be 221 fetched from sources indexed by different hash algorithms. 222 223Referencing File Series 224""""""""""""""""""""""" 225 226The ``DATA{}`` syntax can be told to fetch a file series using the form 227``DATA{<name>,:}``, where the ``:`` is literal. If the source tree 228contains a group of files or content links named like a series then a 229reference to one member adds rules to fetch all of them. Although all 230members of a series are fetched, only the file originally named by the 231``DATA{}`` argument is substituted for it. The default configuration 232recognizes file series names ending with ``#.ext``, ``_#.ext``, ``.#.ext``, 233or ``-#.ext`` where ``#`` is a sequence of decimal digits and ``.ext`` is 234any single extension. Configure it with a regex that parses ``<number>`` 235and ``<suffix>`` parts from the end of ``<name>``:: 236 237 ExternalData_SERIES_PARSE = regex of the form (<number>)(<suffix>)$ 238 239For more complicated cases set:: 240 241 ExternalData_SERIES_PARSE = regex with at least two () groups 242 ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any 243 ExternalData_SERIES_PARSE_NUMBER = <number> regex group number 244 ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number 245 246Configure series number matching with a regex that matches the 247``<number>`` part of series members named ``<prefix><number><suffix>``:: 248 249 ExternalData_SERIES_MATCH = regex matching <number> in all series members 250 251Note that the ``<suffix>`` of a series does not include a hash-algorithm 252extension. 253 254Referencing Associated Files 255"""""""""""""""""""""""""""" 256 257The ``DATA{}`` syntax can alternatively match files associated with the 258named file and contained in the same directory. Associated files may 259be specified by options using the syntax 260``DATA{<name>,<opt1>,<opt2>,...}``. Each option may specify one file by 261name or specify a regular expression to match file names using the 262syntax ``REGEX:<regex>``. For example, the arguments:: 263 264 DATA{MyData/MyInput.mhd,MyInput.img} # File pair 265 DATA{MyData/MyFrames00.png,REGEX:MyFrames[0-9]+\\.png} # Series 266 267will pass ``MyInput.mha`` and ``MyFrames00.png`` on the command line but 268ensure that the associated files are present next to them. 269 270Referencing Directories 271""""""""""""""""""""""" 272 273The ``DATA{}`` syntax may reference a directory using a trailing slash and 274a list of associated files. The form ``DATA{<name>/,<opt1>,<opt2>,...}`` 275adds rules to fetch any files in the directory that match one of the 276associated file options. For example, the argument 277``DATA{MyDataDir/,REGEX:.*}`` will pass the full path to a ``MyDataDir`` 278directory on the command line and ensure that the directory contains 279files corresponding to every file or content link in the ``MyDataDir`` 280source directory. 281 282.. versionadded:: 3.3 283 In order to match associated files in subdirectories, 284 specify a ``RECURSE:`` option, e.g. ``DATA{MyDataDir/,RECURSE:,REGEX:.*}``. 285 286Hash Algorithms 287^^^^^^^^^^^^^^^ 288 289The following hash algorithms are supported:: 290 291 %(algo) <ext> Description 292 ------- ----- ----------- 293 MD5 .md5 Message-Digest Algorithm 5, RFC 1321 294 SHA1 .sha1 US Secure Hash Algorithm 1, RFC 3174 295 SHA224 .sha224 US Secure Hash Algorithms, RFC 4634 296 SHA256 .sha256 US Secure Hash Algorithms, RFC 4634 297 SHA384 .sha384 US Secure Hash Algorithms, RFC 4634 298 SHA512 .sha512 US Secure Hash Algorithms, RFC 4634 299 SHA3_224 .sha3-224 Keccak SHA-3 300 SHA3_256 .sha3-256 Keccak SHA-3 301 SHA3_384 .sha3-384 Keccak SHA-3 302 SHA3_512 .sha3-512 Keccak SHA-3 303 304.. versionadded:: 3.8 305 Added the ``SHA3_*`` hash algorithms. 306 307Note that the hashes are used only for unique data identification and 308download verification. 309 310.. _`ExternalData Custom Fetch Scripts`: 311 312Custom Fetch Scripts 313^^^^^^^^^^^^^^^^^^^^ 314 315.. versionadded:: 3.2 316 317When a data file must be fetched from one of the URL templates 318specified in the ``ExternalData_URL_TEMPLATES`` variable, it is 319normally downloaded using the :command:`file(DOWNLOAD)` command. 320One may specify usage of a custom fetch script by using a URL 321template of the form ``ExternalDataCustomScript://<key>/<loc>``. 322The ``<key>`` must be a C identifier, and the ``<loc>`` must 323contain the ``%(algo)`` and ``%(hash)`` placeholders. 324A variable corresponding to the key, ``ExternalData_CUSTOM_SCRIPT_<key>``, 325must be set to the full path to a ``.cmake`` script file. The script 326will be included to perform the actual fetch, and provided with 327the following variables: 328 329.. variable:: ExternalData_CUSTOM_LOCATION 330 331 When a custom fetch script is loaded, this variable is set to the 332 location part of the URL, which will contain the substituted hash 333 algorithm name and content hash value. 334 335.. variable:: ExternalData_CUSTOM_FILE 336 337 When a custom fetch script is loaded, this variable is set to the 338 full path to a file in which the script must store the fetched 339 content. The name of the file is unspecified and should not be 340 interpreted in any way. 341 342The custom fetch script is expected to store fetched content in the 343file or set a variable: 344 345.. variable:: ExternalData_CUSTOM_ERROR 346 347 When a custom fetch script fails to fetch the requested content, 348 it must set this variable to a short one-line message describing 349 the reason for failure. 350 351#]=======================================================================] 352 353function(ExternalData_add_test target) 354 # Expand all arguments as a single string to preserve escaped semicolons. 355 ExternalData_expand_arguments("${target}" testArgs "${ARGN}") 356 add_test(${testArgs}) 357endfunction() 358 359function(ExternalData_add_target target) 360 if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES) 361 message(FATAL_ERROR 362 "Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!") 363 endif() 364 if(NOT ExternalData_OBJECT_STORES) 365 set(ExternalData_OBJECT_STORES ${CMAKE_BINARY_DIR}/ExternalData/Objects) 366 endif() 367 set(_ExternalData_CONFIG_CODE "") 368 369 cmake_parse_arguments(PARSE_ARGV 1 _ExternalData_add_target 370 "" 371 "SHOW_PROGRESS" 372 "") 373 if (_ExternalData_add_target_UNPARSED_ARGUMENTS) 374 message(AUTHOR_WARNING 375 "Ignoring unrecognized arguments passed to ExternalData_add_target: " 376 "`${_ExternalData_add_target_UNPARSED_ARGUMENTS}`") 377 endif () 378 379 # Turn `SHOW_PROGRESS` into a boolean 380 if (NOT DEFINED _ExternalData_add_target_SHOW_PROGRESS) 381 # The default setting 382 if (CMAKE_GENERATOR MATCHES "Ninja") 383 set(_ExternalData_add_target_SHOW_PROGRESS OFF) 384 else () 385 set(_ExternalData_add_target_SHOW_PROGRESS ON) 386 endif () 387 elseif (_ExternalData_add_target_SHOW_PROGRESS) 388 set(_ExternalData_add_target_SHOW_PROGRESS ON) 389 else () 390 set(_ExternalData_add_target_SHOW_PROGRESS OFF) 391 endif () 392 393 # Store custom script configuration. 394 foreach(url_template IN LISTS ExternalData_URL_TEMPLATES) 395 if("${url_template}" MATCHES "^ExternalDataCustomScript://([^/]*)/(.*)$") 396 set(key "${CMAKE_MATCH_1}") 397 if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$") 398 if(ExternalData_CUSTOM_SCRIPT_${key}) 399 if(IS_ABSOLUTE "${ExternalData_CUSTOM_SCRIPT_${key}}") 400 string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n" 401 "set(ExternalData_CUSTOM_SCRIPT_${key} \"${ExternalData_CUSTOM_SCRIPT_${key}}\")") 402 else() 403 message(FATAL_ERROR 404 "No ExternalData_CUSTOM_SCRIPT_${key} is not set to a full path:\n" 405 " ${ExternalData_CUSTOM_SCRIPT_${key}}") 406 endif() 407 else() 408 message(FATAL_ERROR 409 "No ExternalData_CUSTOM_SCRIPT_${key} is set for URL template:\n" 410 " ${url_template}") 411 endif() 412 else() 413 message(FATAL_ERROR 414 "Bad ExternalDataCustomScript key '${key}' in URL template:\n" 415 " ${url_template}\n" 416 "The key must be a valid C identifier.") 417 endif() 418 endif() 419 420 # Store custom algorithm name to URL component maps. 421 if("${url_template}" MATCHES "%\\(algo:([^)]*)\\)") 422 set(key "${CMAKE_MATCH_1}") 423 if(key MATCHES "^[A-Za-z_][A-Za-z0-9_]*$") 424 string(REPLACE "|" ";" _algos "${_ExternalData_REGEX_ALGO}") 425 foreach(algo ${_algos}) 426 if(DEFINED ExternalData_URL_ALGO_${algo}_${key}) 427 string(CONCAT _ExternalData_CONFIG_CODE "${_ExternalData_CONFIG_CODE}\n" 428 "set(ExternalData_URL_ALGO_${algo}_${key} \"${ExternalData_URL_ALGO_${algo}_${key}}\")") 429 endif() 430 endforeach() 431 else() 432 message(FATAL_ERROR 433 "Bad %(algo:${key}) in URL template:\n" 434 " ${url_template}\n" 435 "The transform name must be a valid C identifier.") 436 endif() 437 endif() 438 endforeach() 439 440 # Store configuration for use by build-time script. 441 set(config ${CMAKE_CURRENT_BINARY_DIR}/${target}_config.cmake) 442 configure_file(${_ExternalData_SELF_DIR}/ExternalData_config.cmake.in ${config} @ONLY) 443 444 set(files "") 445 446 # Set a "_ExternalData_FILE_${file}" variable for each output file to avoid 447 # duplicate entries within this target. Set a directory property of the same 448 # name to avoid repeating custom commands with the same output in this directory. 449 # Repeating custom commands with the same output across directories or across 450 # targets in the same directory may be a race, but this is likely okay because 451 # we use atomic replacement of output files. 452 # 453 # Use local data first to prefer real files over content links. 454 455 # Custom commands to copy or link local data. 456 get_property(data_local GLOBAL PROPERTY _ExternalData_${target}_LOCAL) 457 foreach(entry IN LISTS data_local) 458 string(REPLACE "|" ";" tuple "${entry}") 459 list(GET tuple 0 file) 460 list(GET tuple 1 name) 461 if(NOT DEFINED "_ExternalData_FILE_${file}") 462 set("_ExternalData_FILE_${file}" 1) 463 get_property(added DIRECTORY PROPERTY "_ExternalData_FILE_${file}") 464 if(NOT added) 465 set_property(DIRECTORY PROPERTY "_ExternalData_FILE_${file}" 1) 466 add_custom_command( 467 COMMENT "Generating ${file}" 468 OUTPUT "${file}" 469 COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR} 470 -Dfile=${file} -Dname=${name} 471 -DExternalData_ACTION=local 472 -DExternalData_SHOW_PROGRESS=${_ExternalData_add_target_SHOW_PROGRESS} 473 -DExternalData_CONFIG=${config} 474 -P ${_ExternalData_SELF} 475 MAIN_DEPENDENCY "${name}" 476 ) 477 endif() 478 list(APPEND files "${file}") 479 endif() 480 endforeach() 481 482 # Custom commands to fetch remote data. 483 get_property(data_fetch GLOBAL PROPERTY _ExternalData_${target}_FETCH) 484 foreach(entry IN LISTS data_fetch) 485 string(REPLACE "|" ";" tuple "${entry}") 486 list(GET tuple 0 file) 487 list(GET tuple 1 name) 488 list(GET tuple 2 exts) 489 string(REPLACE "+" ";" exts_list "${exts}") 490 list(GET exts_list 0 first_ext) 491 set(stamp "-hash-stamp") 492 if(NOT DEFINED "_ExternalData_FILE_${file}") 493 set("_ExternalData_FILE_${file}" 1) 494 get_property(added DIRECTORY PROPERTY "_ExternalData_FILE_${file}") 495 if(NOT added) 496 set_property(DIRECTORY PROPERTY "_ExternalData_FILE_${file}" 1) 497 add_custom_command( 498 # Users care about the data file, so hide the hash/timestamp file. 499 COMMENT "Generating ${file}" 500 # The hash/timestamp file is the output from the build perspective. 501 # List the real file as a second output in case it is a broken link. 502 # The files must be listed in this order so CMake can hide from the 503 # make tool that a symlink target may not be newer than the input. 504 OUTPUT "${file}${stamp}" "${file}" 505 # Run the data fetch/update script. 506 COMMAND ${CMAKE_COMMAND} -Drelative_top=${CMAKE_BINARY_DIR} 507 -Dfile=${file} -Dname=${name} -Dexts=${exts} 508 -DExternalData_ACTION=fetch 509 -DExternalData_SHOW_PROGRESS=${_ExternalData_add_target_SHOW_PROGRESS} 510 -DExternalData_CONFIG=${config} 511 -P ${_ExternalData_SELF} 512 # Update whenever the object hash changes. 513 MAIN_DEPENDENCY "${name}${first_ext}" 514 ) 515 endif() 516 list(APPEND files "${file}${stamp}") 517 endif() 518 endforeach() 519 520 # Custom target to drive all update commands. 521 add_custom_target(${target} ALL DEPENDS ${files}) 522endfunction() 523 524function(ExternalData_expand_arguments target outArgsVar) 525 # Replace DATA{} references with real arguments. 526 set(data_regex "DATA{([^;{}\r\n]*)}") 527 set(other_regex "([^D]|D[^A]|DA[^T]|DAT[^A]|DATA[^{])+|.") 528 set(outArgs "") 529 # This list expansion un-escapes semicolons in list element values so we 530 # must re-escape them below anywhere a new list expansion will occur. 531 foreach(arg IN LISTS ARGN) 532 if("x${arg}" MATCHES "${data_regex}") 533 # Re-escape in-value semicolons before expansion in foreach below. 534 string(REPLACE ";" "\\;" tmp "${arg}") 535 # Split argument into DATA{}-pieces and other pieces. 536 string(REGEX MATCHALL "${data_regex}|${other_regex}" pieces "${tmp}") 537 # Compose output argument with DATA{}-pieces replaced. 538 set(outArg "") 539 foreach(piece IN LISTS pieces) 540 if("x${piece}" MATCHES "^x${data_regex}$") 541 # Replace this DATA{}-piece with a file path. 542 _ExternalData_arg("${target}" "${piece}" "${CMAKE_MATCH_1}" file) 543 string(APPEND outArg "${file}") 544 else() 545 # No replacement needed for this piece. 546 string(APPEND outArg "${piece}") 547 endif() 548 endforeach() 549 else() 550 # No replacements needed in this argument. 551 set(outArg "${arg}") 552 endif() 553 # Re-escape in-value semicolons in resulting list. 554 string(REPLACE ";" "\\;" outArg "${outArg}") 555 list(APPEND outArgs "${outArg}") 556 endforeach() 557 set("${outArgsVar}" "${outArgs}" PARENT_SCOPE) 558endfunction() 559 560#----------------------------------------------------------------------------- 561# Private helper interface 562 563set(_ExternalData_REGEX_ALGO "MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512") 564set(_ExternalData_REGEX_EXT "md5|sha1|sha224|sha256|sha384|sha512|sha3-224|sha3-256|sha3-384|sha3-512") 565set(_ExternalData_SELF "${CMAKE_CURRENT_LIST_FILE}") 566get_filename_component(_ExternalData_SELF_DIR "${_ExternalData_SELF}" PATH) 567 568function(_ExternalData_compute_hash var_hash algo file) 569 if("${algo}" MATCHES "^${_ExternalData_REGEX_ALGO}$") 570 file("${algo}" "${file}" hash) 571 set("${var_hash}" "${hash}" PARENT_SCOPE) 572 else() 573 message(FATAL_ERROR "Hash algorithm ${algo} unimplemented.") 574 endif() 575endfunction() 576 577function(_ExternalData_random var) 578 string(RANDOM LENGTH 6 random) 579 set("${var}" "${random}" PARENT_SCOPE) 580endfunction() 581 582function(_ExternalData_exact_regex regex_var string) 583 string(REGEX REPLACE "([][+.*()^])" "\\\\\\1" regex "${string}") 584 set("${regex_var}" "${regex}" PARENT_SCOPE) 585endfunction() 586 587function(_ExternalData_atomic_write file content) 588 _ExternalData_random(random) 589 set(tmp "${file}.tmp${random}") 590 file(WRITE "${tmp}" "${content}") 591 file(RENAME "${tmp}" "${file}") 592endfunction() 593 594function(_ExternalData_link_content name var_ext) 595 if("${ExternalData_LINK_CONTENT}" MATCHES "^(${_ExternalData_REGEX_ALGO})$") 596 set(algo "${ExternalData_LINK_CONTENT}") 597 else() 598 message(FATAL_ERROR 599 "Unknown hash algorithm specified by ExternalData_LINK_CONTENT:\n" 600 " ${ExternalData_LINK_CONTENT}") 601 endif() 602 _ExternalData_compute_hash(hash "${algo}" "${name}") 603 get_filename_component(dir "${name}" PATH) 604 set(staged "${dir}/.ExternalData_${algo}_${hash}") 605 string(TOLOWER ".${algo}" ext) 606 _ExternalData_atomic_write("${name}${ext}" "${hash}\n") 607 file(RENAME "${name}" "${staged}") 608 set("${var_ext}" "${ext}" PARENT_SCOPE) 609 610 file(RELATIVE_PATH relname "${ExternalData_SOURCE_ROOT}" "${name}${ext}") 611 message(STATUS "Linked ${relname} to ExternalData ${algo}/${hash}") 612endfunction() 613 614function(_ExternalData_arg target arg options var_file) 615 # Separate data path from the options. 616 string(REPLACE "," ";" options "${options}") 617 list(GET options 0 data) 618 list(REMOVE_AT options 0) 619 620 # Interpret trailing slashes as directories. 621 set(data_is_directory 0) 622 if("x${data}" MATCHES "^x(.*)([/\\])$") 623 set(data_is_directory 1) 624 set(data "${CMAKE_MATCH_1}") 625 endif() 626 627 # Convert to full path. 628 if(IS_ABSOLUTE "${data}") 629 set(absdata "${data}") 630 else() 631 set(absdata "${CMAKE_CURRENT_SOURCE_DIR}/${data}") 632 endif() 633 get_filename_component(absdata "${absdata}" ABSOLUTE) 634 635 # Convert to relative path under the source tree. 636 if(NOT ExternalData_SOURCE_ROOT) 637 set(ExternalData_SOURCE_ROOT "${CMAKE_SOURCE_DIR}") 638 endif() 639 set(top_src "${ExternalData_SOURCE_ROOT}") 640 file(RELATIVE_PATH reldata "${top_src}" "${absdata}") 641 if(IS_ABSOLUTE "${reldata}" OR "${reldata}" MATCHES "^\\.\\./") 642 message(FATAL_ERROR "Data file referenced by argument\n" 643 " ${arg}\n" 644 "does not lie under the top-level source directory\n" 645 " ${top_src}\n") 646 endif() 647 if(data_is_directory AND NOT IS_DIRECTORY "${top_src}/${reldata}") 648 message(FATAL_ERROR "Data directory referenced by argument\n" 649 " ${arg}\n" 650 "corresponds to source tree path\n" 651 " ${reldata}\n" 652 "that does not exist as a directory!") 653 endif() 654 if(NOT ExternalData_BINARY_ROOT) 655 set(ExternalData_BINARY_ROOT "${CMAKE_BINARY_DIR}") 656 endif() 657 set(top_bin "${ExternalData_BINARY_ROOT}") 658 659 # Handle in-source builds gracefully. 660 if("${top_src}" STREQUAL "${top_bin}") 661 if(ExternalData_LINK_CONTENT) 662 message(WARNING "ExternalData_LINK_CONTENT cannot be used in-source") 663 set(ExternalData_LINK_CONTENT 0) 664 endif() 665 set(top_same 1) 666 endif() 667 668 set(external "") # Entries external to the source tree. 669 set(internal "") # Entries internal to the source tree. 670 set(have_original ${data_is_directory}) 671 set(have_original_as_dir 0) 672 673 # Process options. 674 set(series_option "") 675 set(recurse_option "") 676 set(associated_files "") 677 set(associated_regex "") 678 foreach(opt ${options}) 679 # Regular expression to match associated files. 680 if("x${opt}" MATCHES "^xREGEX:([^:/]+)$") 681 list(APPEND associated_regex "${CMAKE_MATCH_1}") 682 elseif(opt STREQUAL ":") 683 # Activate series matching. 684 set(series_option "${opt}") 685 elseif(opt STREQUAL "RECURSE:") 686 # Activate recursive matching in directories. 687 set(recurse_option "${opt}") 688 elseif("x${opt}" MATCHES "^[^][:/*?]+$") 689 # Specific associated file. 690 list(APPEND associated_files "${opt}") 691 else() 692 message(FATAL_ERROR "Unknown option \"${opt}\" in argument\n" 693 " ${arg}\n") 694 endif() 695 endforeach() 696 697 if(series_option) 698 if(data_is_directory) 699 message(FATAL_ERROR "Series option \"${series_option}\" not allowed with directories.") 700 endif() 701 if(associated_files OR associated_regex) 702 message(FATAL_ERROR "Series option \"${series_option}\" not allowed with associated files.") 703 endif() 704 if(recurse_option) 705 message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.") 706 endif() 707 # Load a whole file series. 708 _ExternalData_arg_series() 709 elseif(data_is_directory) 710 if(associated_files OR associated_regex) 711 # Load listed/matching associated files in the directory. 712 _ExternalData_arg_associated() 713 else() 714 message(FATAL_ERROR "Data directory referenced by argument\n" 715 " ${arg}\n" 716 "must list associated files.") 717 endif() 718 else() 719 if(recurse_option) 720 message(FATAL_ERROR "Recurse option \"${recurse_option}\" allowed only with directories.") 721 endif() 722 # Load the named data file. 723 _ExternalData_arg_single() 724 if(associated_files OR associated_regex) 725 # Load listed/matching associated files. 726 _ExternalData_arg_associated() 727 endif() 728 endif() 729 730 if(NOT have_original) 731 if(have_original_as_dir) 732 set(msg_kind FATAL_ERROR) 733 set(msg "that is directory instead of a file!") 734 else() 735 set(msg_kind AUTHOR_WARNING) 736 set(msg "that does not exist as a file (with or without an extension)!") 737 endif() 738 message(${msg_kind} "Data file referenced by argument\n" 739 " ${arg}\n" 740 "corresponds to source tree path\n" 741 " ${reldata}\n" 742 "${msg}") 743 endif() 744 745 if(external) 746 # Make the series available in the build tree. 747 set_property(GLOBAL APPEND PROPERTY 748 _ExternalData_${target}_FETCH "${external}") 749 set_property(GLOBAL APPEND PROPERTY 750 _ExternalData_${target}_LOCAL "${internal}") 751 set("${var_file}" "${top_bin}/${reldata}" PARENT_SCOPE) 752 else() 753 # The whole series is in the source tree. 754 set("${var_file}" "${top_src}/${reldata}" PARENT_SCOPE) 755 endif() 756endfunction() 757 758macro(_ExternalData_arg_associated) 759 # Associated files lie in the same directory. 760 if(data_is_directory) 761 set(reldir "${reldata}") 762 else() 763 get_filename_component(reldir "${reldata}" PATH) 764 endif() 765 if(reldir) 766 string(APPEND reldir "/") 767 endif() 768 _ExternalData_exact_regex(reldir_regex "${reldir}") 769 if(recurse_option) 770 set(glob GLOB_RECURSE) 771 string(APPEND reldir_regex "(.+/)?") 772 else() 773 set(glob GLOB) 774 endif() 775 776 # Find files named explicitly. 777 foreach(file ${associated_files}) 778 _ExternalData_exact_regex(file_regex "${file}") 779 _ExternalData_arg_find_files(${glob} "${reldir}${file}" 780 "${reldir_regex}${file_regex}") 781 endforeach() 782 783 # Find files matching the given regular expressions. 784 set(all "") 785 set(sep "") 786 foreach(regex ${associated_regex}) 787 string(APPEND all "${sep}${reldir_regex}${regex}") 788 set(sep "|") 789 endforeach() 790 _ExternalData_arg_find_files(${glob} "${reldir}" "${all}") 791endmacro() 792 793macro(_ExternalData_arg_single) 794 # Match only the named data by itself. 795 _ExternalData_exact_regex(data_regex "${reldata}") 796 _ExternalData_arg_find_files(GLOB "${reldata}" "${data_regex}") 797endmacro() 798 799macro(_ExternalData_arg_series) 800 # Configure series parsing and matching. 801 set(series_parse_prefix "") 802 set(series_parse_number "\\1") 803 set(series_parse_suffix "\\2") 804 if(ExternalData_SERIES_PARSE) 805 if(ExternalData_SERIES_PARSE_NUMBER AND ExternalData_SERIES_PARSE_SUFFIX) 806 if(ExternalData_SERIES_PARSE_PREFIX) 807 set(series_parse_prefix "\\${ExternalData_SERIES_PARSE_PREFIX}") 808 endif() 809 set(series_parse_number "\\${ExternalData_SERIES_PARSE_NUMBER}") 810 set(series_parse_suffix "\\${ExternalData_SERIES_PARSE_SUFFIX}") 811 elseif(NOT "x${ExternalData_SERIES_PARSE}" MATCHES "^x\\([^()]*\\)\\([^()]*\\)\\$$") 812 message(FATAL_ERROR 813 "ExternalData_SERIES_PARSE is set to\n" 814 " ${ExternalData_SERIES_PARSE}\n" 815 "which is not of the form\n" 816 " (<number>)(<suffix>)$\n" 817 "Fix the regular expression or set variables\n" 818 " ExternalData_SERIES_PARSE_PREFIX = <prefix> regex group number, if any\n" 819 " ExternalData_SERIES_PARSE_NUMBER = <number> regex group number\n" 820 " ExternalData_SERIES_PARSE_SUFFIX = <suffix> regex group number\n" 821 ) 822 endif() 823 set(series_parse "${ExternalData_SERIES_PARSE}") 824 else() 825 set(series_parse "([0-9]*)(\\.[^./]*)$") 826 endif() 827 if(ExternalData_SERIES_MATCH) 828 set(series_match "${ExternalData_SERIES_MATCH}") 829 else() 830 set(series_match "[_.-]?[0-9]*") 831 endif() 832 833 # Parse the base, number, and extension components of the series. 834 string(REGEX REPLACE "${series_parse}" "${series_parse_prefix};${series_parse_number};${series_parse_suffix}" tuple "${reldata}") 835 list(LENGTH tuple len) 836 if(NOT "${len}" EQUAL 3) 837 message(FATAL_ERROR "Data file referenced by argument\n" 838 " ${arg}\n" 839 "corresponds to path\n" 840 " ${reldata}\n" 841 "that does not match regular expression\n" 842 " ${series_parse}") 843 endif() 844 list(GET tuple 0 relbase) 845 list(GET tuple 2 ext) 846 847 # Glob files that might match the series. 848 # Then match base, number, and extension. 849 _ExternalData_exact_regex(series_base "${relbase}") 850 _ExternalData_exact_regex(series_ext "${ext}") 851 _ExternalData_arg_find_files(GLOB "${relbase}*${ext}" 852 "${series_base}${series_match}${series_ext}") 853endmacro() 854 855function(_ExternalData_arg_find_files glob pattern regex) 856 cmake_policy(PUSH) 857 cmake_policy(SET CMP0009 NEW) 858 file(${glob} globbed RELATIVE "${top_src}" "${top_src}/${pattern}*") 859 cmake_policy(POP) 860 set(externals_count -1) 861 foreach(entry IN LISTS globbed) 862 if("x${entry}" MATCHES "^x(.*)(\\.(${_ExternalData_REGEX_EXT}))$") 863 set(relname "${CMAKE_MATCH_1}") 864 set(alg "${CMAKE_MATCH_2}") 865 else() 866 set(relname "${entry}") 867 set(alg "") 868 endif() 869 if("x${relname}" MATCHES "^x${regex}$" # matches 870 AND NOT "x${relname}" MATCHES "(^x|/)\\.ExternalData_" # not staged obj 871 ) 872 if(IS_DIRECTORY "${top_src}/${entry}") 873 if("${relname}" STREQUAL "${reldata}") 874 set(have_original_as_dir 1) 875 endif() 876 else() 877 set(name "${top_src}/${relname}") 878 set(file "${top_bin}/${relname}") 879 if(alg) 880 if(NOT "${external_${externals_count}_file_name}" STREQUAL "${file}|${name}") 881 math(EXPR externals_count "${externals_count} + 1") 882 set(external_${externals_count}_file_name "${file}|${name}") 883 endif() 884 list(APPEND external_${externals_count}_algs "${alg}") 885 elseif(ExternalData_LINK_CONTENT) 886 _ExternalData_link_content("${name}" alg) 887 list(APPEND external "${file}|${name}|${alg}") 888 elseif(NOT top_same) 889 list(APPEND internal "${file}|${name}") 890 endif() 891 if("${relname}" STREQUAL "${reldata}") 892 set(have_original 1) 893 endif() 894 endif() 895 endif() 896 endforeach() 897 if(${externals_count} GREATER -1) 898 foreach(ii RANGE ${externals_count}) 899 string(REPLACE ";" "+" algs_delim "${external_${ii}_algs}") 900 list(APPEND external "${external_${ii}_file_name}|${algs_delim}") 901 unset(external_${ii}_algs) 902 unset(external_${ii}_file_name) 903 endforeach() 904 endif() 905 set(external "${external}" PARENT_SCOPE) 906 set(internal "${internal}" PARENT_SCOPE) 907 set(have_original "${have_original}" PARENT_SCOPE) 908 set(have_original_as_dir "${have_original_as_dir}" PARENT_SCOPE) 909endfunction() 910 911#----------------------------------------------------------------------------- 912# Private script mode interface 913 914if(CMAKE_GENERATOR OR NOT ExternalData_ACTION) 915 return() 916endif() 917 918if(ExternalData_CONFIG) 919 include(${ExternalData_CONFIG}) 920endif() 921if(NOT ExternalData_URL_TEMPLATES AND NOT ExternalData_OBJECT_STORES) 922 message(FATAL_ERROR 923 "Neither ExternalData_URL_TEMPLATES nor ExternalData_OBJECT_STORES is set!") 924endif() 925 926function(_ExternalData_link_or_copy src dst) 927 # Create a temporary file first. 928 get_filename_component(dst_dir "${dst}" PATH) 929 file(MAKE_DIRECTORY "${dst_dir}") 930 _ExternalData_random(random) 931 set(tmp "${dst}.tmp${random}") 932 if(UNIX AND NOT ExternalData_NO_SYMLINKS) 933 # Create a symbolic link. 934 set(tgt "${src}") 935 if(relative_top) 936 # Use relative path if files are close enough. 937 file(RELATIVE_PATH relsrc "${relative_top}" "${src}") 938 file(RELATIVE_PATH relfile "${relative_top}" "${dst}") 939 if(NOT IS_ABSOLUTE "${relsrc}" AND NOT "${relsrc}" MATCHES "^\\.\\./" AND 940 NOT IS_ABSOLUTE "${reldst}" AND NOT "${reldst}" MATCHES "^\\.\\./") 941 file(RELATIVE_PATH tgt "${dst_dir}" "${src}") 942 endif() 943 endif() 944 # Create link (falling back to copying if there's a problem). 945 file(CREATE_LINK "${tgt}" "${tmp}" RESULT result COPY_ON_ERROR SYMBOLIC) 946 else() 947 # Create a copy. 948 file(COPY_FILE "${src}" "${tmp}" RESULT result) 949 endif() 950 if(result) 951 file(REMOVE "${tmp}") 952 message(FATAL_ERROR "Failed to create:\n \"${tmp}\"\nfrom:\n \"${obj}\"\nwith error:\n ${result}") 953 endif() 954 955 # Atomically create/replace the real destination. 956 file(RENAME "${tmp}" "${dst}") 957endfunction() 958 959function(_ExternalData_download_file url file err_var msg_var) 960 set(retry 3) 961 while(retry) 962 math(EXPR retry "${retry} - 1") 963 if(ExternalData_TIMEOUT_INACTIVITY) 964 set(inactivity_timeout INACTIVITY_TIMEOUT ${ExternalData_TIMEOUT_INACTIVITY}) 965 elseif(NOT "${ExternalData_TIMEOUT_INACTIVITY}" EQUAL 0) 966 set(inactivity_timeout INACTIVITY_TIMEOUT 60) 967 else() 968 set(inactivity_timeout "") 969 endif() 970 if(ExternalData_TIMEOUT_ABSOLUTE) 971 set(absolute_timeout TIMEOUT ${ExternalData_TIMEOUT_ABSOLUTE}) 972 elseif(NOT "${ExternalData_TIMEOUT_ABSOLUTE}" EQUAL 0) 973 set(absolute_timeout TIMEOUT 300) 974 else() 975 set(absolute_timeout "") 976 endif() 977 set(show_progress_args) 978 if (ExternalData_SHOW_PROGRESS) 979 list(APPEND show_progress_args SHOW_PROGRESS) 980 endif () 981 file(DOWNLOAD "${url}" "${file}" STATUS status LOG log ${inactivity_timeout} ${absolute_timeout} ${show_progress_args}) 982 list(GET status 0 err) 983 list(GET status 1 msg) 984 if(err) 985 if("${msg}" MATCHES "HTTP response code said error" AND 986 "${log}" MATCHES "error: 503") 987 set(msg "temporarily unavailable") 988 endif() 989 elseif("${log}" MATCHES "\nHTTP[^\n]* 503") 990 set(err TRUE) 991 set(msg "temporarily unavailable") 992 endif() 993 if(NOT err OR NOT "${msg}" MATCHES "partial|timeout|temporarily") 994 break() 995 elseif(retry) 996 message(STATUS "[download terminated: ${msg}, retries left: ${retry}]") 997 endif() 998 endwhile() 999 set("${err_var}" "${err}" PARENT_SCOPE) 1000 set("${msg_var}" "${msg}" PARENT_SCOPE) 1001endfunction() 1002 1003function(_ExternalData_custom_fetch key loc file err_var msg_var) 1004 if(NOT ExternalData_CUSTOM_SCRIPT_${key}) 1005 set(err 1) 1006 set(msg "No ExternalData_CUSTOM_SCRIPT_${key} set!") 1007 elseif(NOT EXISTS "${ExternalData_CUSTOM_SCRIPT_${key}}") 1008 set(err 1) 1009 set(msg "No '${ExternalData_CUSTOM_SCRIPT_${key}}' exists!") 1010 else() 1011 set(ExternalData_CUSTOM_LOCATION "${loc}") 1012 set(ExternalData_CUSTOM_FILE "${file}") 1013 unset(ExternalData_CUSTOM_ERROR) 1014 include("${ExternalData_CUSTOM_SCRIPT_${key}}") 1015 if(DEFINED ExternalData_CUSTOM_ERROR) 1016 set(err 1) 1017 set(msg "${ExternalData_CUSTOM_ERROR}") 1018 else() 1019 set(err 0) 1020 set(msg "no error") 1021 endif() 1022 endif() 1023 set("${err_var}" "${err}" PARENT_SCOPE) 1024 set("${msg_var}" "${msg}" PARENT_SCOPE) 1025endfunction() 1026 1027function(_ExternalData_get_from_object_store hash algo var_obj var_success) 1028 # Search all object stores for an existing object. 1029 foreach(dir ${ExternalData_OBJECT_STORES}) 1030 set(obj "${dir}/${algo}/${hash}") 1031 if(EXISTS "${obj}") 1032 message(STATUS "Found object: \"${obj}\"") 1033 set("${var_obj}" "${obj}" PARENT_SCOPE) 1034 set("${var_success}" 1 PARENT_SCOPE) 1035 return() 1036 endif() 1037 endforeach() 1038endfunction() 1039 1040function(_ExternalData_download_object name hash algo var_obj var_success var_errorMsg) 1041 # Search all object stores for an existing object. 1042 set(success 1) 1043 foreach(dir ${ExternalData_OBJECT_STORES}) 1044 set(obj "${dir}/${algo}/${hash}") 1045 if(EXISTS "${obj}") 1046 message(STATUS "Found object: \"${obj}\"") 1047 set("${var_obj}" "${obj}" PARENT_SCOPE) 1048 set("${var_success}" "${success}" PARENT_SCOPE) 1049 return() 1050 endif() 1051 endforeach() 1052 1053 # Download object to the first store. 1054 list(GET ExternalData_OBJECT_STORES 0 store) 1055 set(obj "${store}/${algo}/${hash}") 1056 1057 _ExternalData_random(random) 1058 set(tmp "${obj}.tmp${random}") 1059 set(found 0) 1060 set(tried "") 1061 foreach(url_template IN LISTS ExternalData_URL_TEMPLATES) 1062 string(REPLACE "%(hash)" "${hash}" url_tmp "${url_template}") 1063 string(REPLACE "%(algo)" "${algo}" url "${url_tmp}") 1064 if(url MATCHES "^(.*)%\\(algo:([A-Za-z_][A-Za-z0-9_]*)\\)(.*)$") 1065 set(lhs "${CMAKE_MATCH_1}") 1066 set(key "${CMAKE_MATCH_2}") 1067 set(rhs "${CMAKE_MATCH_3}") 1068 if(DEFINED ExternalData_URL_ALGO_${algo}_${key}) 1069 set(url "${lhs}${ExternalData_URL_ALGO_${algo}_${key}}${rhs}") 1070 else() 1071 set(url "${lhs}${algo}${rhs}") 1072 endif() 1073 endif() 1074 string(REGEX REPLACE "((https?|ftp)://)([^@]+@)?(.*)" "\\1\\4" secured_url "${url}") 1075 message(STATUS "Fetching \"${secured_url}\"") 1076 if(url MATCHES "^ExternalDataCustomScript://([A-Za-z_][A-Za-z0-9_]*)/(.*)$") 1077 _ExternalData_custom_fetch("${CMAKE_MATCH_1}" "${CMAKE_MATCH_2}" "${tmp}" err errMsg) 1078 else() 1079 _ExternalData_download_file("${url}" "${tmp}" err errMsg) 1080 endif() 1081 string(APPEND tried "\n ${url}") 1082 if(err) 1083 string(APPEND tried " (${errMsg})") 1084 else() 1085 # Verify downloaded object. 1086 _ExternalData_compute_hash(dl_hash "${algo}" "${tmp}") 1087 if("${dl_hash}" STREQUAL "${hash}") 1088 set(found 1) 1089 break() 1090 else() 1091 string(APPEND tried " (wrong hash ${algo}=${dl_hash})") 1092 if("$ENV{ExternalData_DEBUG_DOWNLOAD}" MATCHES ".") 1093 file(RENAME "${tmp}" "${store}/${algo}/${dl_hash}") 1094 endif() 1095 endif() 1096 endif() 1097 file(REMOVE "${tmp}") 1098 endforeach() 1099 1100 get_filename_component(dir "${name}" PATH) 1101 set(staged "${dir}/.ExternalData_${algo}_${hash}") 1102 1103 set(success 1) 1104 if(found) 1105 # Atomically create the object. If we lose a race with another process, 1106 # do not replace it. Content-addressing ensures it has what we expect. 1107 file(RENAME "${tmp}" "${obj}" NO_REPLACE RESULT result) 1108 if (result STREQUAL "NO_REPLACE") 1109 file(REMOVE "${tmp}") 1110 elseif (result) 1111 message(FATAL_ERROR "Failed to rename:\n \"${tmp}\"\nto:\n \"${obj}\"\nwith error:\n ${result}") 1112 endif() 1113 message(STATUS "Downloaded object: \"${obj}\"") 1114 elseif(EXISTS "${staged}") 1115 set(obj "${staged}") 1116 message(STATUS "Staged object: \"${obj}\"") 1117 else() 1118 if(NOT tried) 1119 set(tried "\n (No ExternalData_URL_TEMPLATES given)") 1120 endif() 1121 set(success 0) 1122 set("${var_errorMsg}" "Object ${algo}=${hash} not found at:${tried}" PARENT_SCOPE) 1123 endif() 1124 1125 set("${var_obj}" "${obj}" PARENT_SCOPE) 1126 set("${var_success}" "${success}" PARENT_SCOPE) 1127endfunction() 1128 1129if("${ExternalData_ACTION}" STREQUAL "fetch") 1130 foreach(v ExternalData_OBJECT_STORES file name exts) 1131 if(NOT DEFINED "${v}") 1132 message(FATAL_ERROR "No \"-D${v}=\" value provided!") 1133 endif() 1134 endforeach() 1135 1136 string(REPLACE "+" ";" exts_list "${exts}") 1137 set(succeeded 0) 1138 set(errorMsg "") 1139 set(hash_list ) 1140 set(algo_list ) 1141 set(hash ) 1142 set(algo ) 1143 foreach(ext ${exts_list}) 1144 file(READ "${name}${ext}" hash) 1145 string(STRIP "${hash}" hash) 1146 1147 if("${ext}" MATCHES "^\\.(${_ExternalData_REGEX_EXT})$") 1148 string(TOUPPER "${CMAKE_MATCH_1}" algo) 1149 string(REPLACE "-" "_" algo "${algo}") 1150 else() 1151 message(FATAL_ERROR "Unknown hash algorithm extension \"${ext}\"") 1152 endif() 1153 1154 list(APPEND hash_list ${hash}) 1155 list(APPEND algo_list ${algo}) 1156 endforeach() 1157 1158 list(LENGTH exts_list num_extensions) 1159 math(EXPR exts_range "${num_extensions} - 1") 1160 foreach(ii RANGE 0 ${exts_range}) 1161 list(GET hash_list ${ii} hash) 1162 list(GET algo_list ${ii} algo) 1163 _ExternalData_get_from_object_store("${hash}" "${algo}" obj succeeded) 1164 if(succeeded) 1165 break() 1166 endif() 1167 endforeach() 1168 if(NOT succeeded) 1169 foreach(ii RANGE 0 ${exts_range}) 1170 list(GET hash_list ${ii} hash) 1171 list(GET algo_list ${ii} algo) 1172 _ExternalData_download_object("${name}" "${hash}" "${algo}" 1173 obj succeeded algoErrorMsg) 1174 string(APPEND errorMsg "\n${algoErrorMsg}") 1175 if(succeeded) 1176 break() 1177 endif() 1178 endforeach() 1179 endif() 1180 if(NOT succeeded) 1181 message(FATAL_ERROR "${errorMsg}") 1182 endif() 1183 # Check if file already corresponds to the object. 1184 set(stamp "-hash-stamp") 1185 set(file_up_to_date 0) 1186 if(EXISTS "${file}" AND EXISTS "${file}${stamp}") 1187 file(READ "${file}${stamp}" f_hash) 1188 string(STRIP "${f_hash}" f_hash) 1189 if("${f_hash}" STREQUAL "${hash}") 1190 set(file_up_to_date 1) 1191 endif() 1192 endif() 1193 1194 if(file_up_to_date) 1195 # Touch the file to convince the build system it is up to date. 1196 file(TOUCH "${file}") 1197 else() 1198 _ExternalData_link_or_copy("${obj}" "${file}") 1199 endif() 1200 1201 # Atomically update the hash/timestamp file to record the object referenced. 1202 _ExternalData_atomic_write("${file}${stamp}" "${hash}\n") 1203elseif("${ExternalData_ACTION}" STREQUAL "local") 1204 foreach(v file name) 1205 if(NOT DEFINED "${v}") 1206 message(FATAL_ERROR "No \"-D${v}=\" value provided!") 1207 endif() 1208 endforeach() 1209 _ExternalData_link_or_copy("${name}" "${file}") 1210else() 1211 message(FATAL_ERROR "Unknown ExternalData_ACTION=[${ExternalData_ACTION}]") 1212endif() 1213
