1PSCI OS-initiated mode 2====================== 3 4:Author: Maulik Shah & Wing Li 5:Organization: Qualcomm Innovation Center, Inc. & Google LLC 6:Contact: Maulik Shah <[email protected]> & Wing Li <[email protected]> 7:Status: Accepted 8 9.. contents:: Table of Contents 10 11Introduction 12------------ 13 14Power state coordination 15^^^^^^^^^^^^^^^^^^^^^^^^ 16 17A power domain topology is a logical hierarchy of power domains in a system that 18arises from the physical dependencies between power domains. 19 20Local power states describe power states for an individual node, and composite 21power states describe the combined power states for an individual node and its 22parent node(s). 23 24Entry into low-power states for a topology node above the core level requires 25coordinating its children nodes. For example, in a system with a power domain 26that encompasses a shared cache, and a separate power domain for each core that 27uses the shared cache, the core power domains must be powered down before the 28shared cache power domain can be powered down. 29 30PSCI supports two modes of power state coordination: platform-coordinated and 31OS-initiated. 32 33Platform-coordinated 34~~~~~~~~~~~~~~~~~~~~ 35 36Platform-coordinated mode is the default mode of power state coordination, and 37is currently the only supported mode in TF-A. 38 39In platform-coordinated mode, the platform is responsible for coordinating power 40states, and chooses the deepest power state for a topology node that can be 41tolerated by its children. 42 43OS-initiated 44~~~~~~~~~~~~ 45 46OS-initiated mode is optional. 47 48In OS-initiated mode, the calling OS is responsible for coordinating power 49states, and may request for a topology node to enter a low-power state when 50its last child enters the low-power state. 51 52Motivation 53---------- 54 55There are two reasons why OS-initiated mode might be a more suitable option than 56platform-coordinated mode for a platform. 57 58Scalability 59^^^^^^^^^^^ 60 61In platform-coordinated mode, each core independently selects their own local 62power states, and doesn't account for composite power states that are shared 63between cores. 64 65In OS-initiated mode, the OS has knowledge of the next wakeup event for each 66core, and can have more precise control over the entry, exit, and wakeup 67latencies when deciding if a composite power state (e.g. for a cluster) is 68appropriate. This is especially important for multi-cluster SMP systems and 69heterogeneous systems like big.LITTLE, where different processor types can have 70different power efficiencies. 71 72Simplicity 73^^^^^^^^^^ 74 75In platform-coordinated mode, the OS doesn't have visibility when the last core 76at a power level enters a low-power state. If the OS wants to perform last man 77activity (e.g. powering off a shared resource when it is no longer needed), it 78would have to communicate with an API side channel to know when it can do so. 79This could result in a design smell where the platform is using 80platform-coordinated mode when it should be using OS-initiated mode instead. 81 82In OS-initiated mode, the OS can perform last man activity if it selects a 83composite power state when the last core enters a low-power state. This 84eliminates the need for a side channel, and uses the well documented API between 85the OS and the platform. 86 87Current vendor implementations and workarounds 88^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 89 90* STMicroelectronics 91 92 * For their ARM32 platforms, they're using OS-initiated mode implemented in 93 OP-TEE. 94 * For their future ARM64 platforms, they are interested in using OS-initiated 95 mode in TF-A. 96 97* Qualcomm 98 99 * For their mobile platforms, they're using OS-initiated mode implemented in 100 their own custom secure monitor firmware. 101 * For their Chrome OS platforms, they're using platform-coordinated mode in 102 TF-A with custom driver logic to perform last man activity. 103 104* Google 105 106 * They're using platform-coordinated mode in TF-A with custom driver logic to 107 perform last man activity. 108 109Both Qualcomm and Google would like to be able to use OS-initiated mode in TF-A 110in order to simplify custom driver logic. 111 112Requirements 113------------ 114 115PSCI_FEATURES 116^^^^^^^^^^^^^ 117 118PSCI_FEATURES is for checking whether or not a PSCI function is implemented and 119what its properties are. 120 121.. c:macro:: PSCI_FEATURES 122 123 :param func_id: 0x8400_000A. 124 :param psci_func_id: the function ID of a PSCI function. 125 :retval NOT_SUPPORTED: if the function is not implemented. 126 :retval feature flags associated with the function: if the function is 127 implemented. 128 129CPU_SUSPEND feature flags 130~~~~~~~~~~~~~~~~~~~~~~~~~ 131 132* Reserved, bits[31:2] 133* Power state parameter format, bit[1] 134 135 * A value of 0 indicates the original format is used. 136 * A value of 1 indicates the extended format is used. 137 138* OS-initiated mode, bit[0] 139 140 * A value of 0 indicates OS-initiated mode is not supported. 141 * A value of 1 indicates OS-initiated mode is supported. 142 143See sections 5.1.14 and 5.15 of the PSCI spec (DEN0022D.b) for more details. 144 145PSCI_SET_SUSPEND_MODE 146^^^^^^^^^^^^^^^^^^^^^ 147 148PSCI_SET_SUSPEND_MODE is for switching between the two different modes of power 149state coordination. 150 151.. c:macro:: PSCI_SET_SUSPEND_MODE 152 153 :param func_id: 0x8400_000F. 154 :param mode: 0 indicates platform-coordinated mode, 1 indicates OS-initiated 155 mode. 156 :retval SUCCESS: if the request is successful. 157 :retval NOT_SUPPORTED: if OS-initiated mode is not supported. 158 :retval INVALID_PARAMETERS: if the requested mode is not a valid value (0 or 159 1). 160 :retval DENIED: if the cores are not in the correct state. 161 162Switching from platform-coordinated to OS-initiated is only allowed if the 163following conditions are met: 164 165* All cores are in one of the following states: 166 167 * Running. 168 * Off, through a call to CPU_OFF or not yet booted. 169 * Suspended, through a call to CPU_DEFAULT_SUSPEND. 170 171* None of the cores has called CPU_SUSPEND since the last change of mode or 172 boot. 173 174Switching from OS-initiated to platform-coordinated is only allowed if all cores 175other than the calling core are off, either through a call to CPU_OFF or not yet 176booted. 177 178If these conditions are not met, the PSCI implementation must return DENIED. 179 180See sections 5.1.19 and 5.20 of the PSCI spec (DEN0022D.b) for more details. 181 182CPU_SUSPEND 183^^^^^^^^^^^ 184 185CPU_SUSPEND is for moving a topology node into a low-power state. 186 187.. c:macro:: CPU_SUSPEND 188 189 :param func_id: 0xC400_0001. 190 :param power_state: the requested low-power state to enter. 191 :param entry_point_address: the address at which the core must resume 192 execution following wakeup from a powerdown state. 193 :param context_id: this field specifies a pointer to the saved context that 194 must be restored on a core following wakeup from a powerdown state. 195 :retval SUCCESS: if the request is successful. 196 :retval INVALID_PARAMETERS: in OS-initiated mode, this error is returned when 197 a low-power state is requested for a topology node above the core level, 198 and at least one of the node's children is in a local low-power state 199 that is incompatible with the request. 200 :retval INVALID_ADDRESS: if the entry_point_address argument is invalid. 201 :retval DENIED: only in OS-initiated mode; this error is returned when a 202 low-power state is requested for a topology node above the core level, 203 and at least one of the node's children is running, i.e. not in a 204 low-power state. 205 206In platform-coordinated mode, the PSCI implementation coordinates requests from 207all cores to determine the deepest power state to enter. 208 209In OS-initiated mode, the calling OS is making an explicit request for a 210specific power state, as opposed to expressing a vote. The PSCI implementation 211must comply with the request, unless the request is not consistent with the 212implementation's view of the system's state, in which case, the implementation 213must return INVALID_PARAMETERS or DENIED. 214 215See sections 5.1.2 and 5.4 of the PSCI spec (DEN0022D.b) for more details. 216 217Power state formats 218~~~~~~~~~~~~~~~~~~~ 219 220Original format 221 222* Power Level, bits[25:24] 223 224 * The requested level in the power domain topology to enter a low-power 225 state. 226 227* State Type, bit[16] 228 229 * A value of 0 indicates a standby or retention state. 230 * A value of 1 indicates a powerdown state. 231 232* State ID, bits[15:0] 233 234 * Field to specify the requested composite power state. 235 * The state ID encodings must uniquely describe every possible composite 236 power state. 237 * In OS-initiated mode, the state ID encoding must allow expressing the 238 power level at which the calling core is the last to enter a powerdown 239 state. 240 241Extended format 242 243* State Type, bit[30] 244* State ID, bits[27:0] 245 246Races in OS-initiated mode 247~~~~~~~~~~~~~~~~~~~~~~~~~~ 248 249In OS-initiated mode, there are race windows where the OS's view and 250implementation's view of the system's state differ. It is possible for the OS to 251make requests that are invalid given the implementation's view of the system's 252state. For example, the OS might request a powerdown state for a node from one 253core, while at the same time, the implementation observes that another core in 254that node is powering up. 255 256To address potential race conditions in power state requests: 257 258* The calling OS must specify in each CPU_SUSPEND request the deepest power 259 level for which it sees the calling core as the last running core (last man). 260 This is required even if the OS doesn't want the node at that power level to 261 enter a low-power state. 262* The implementation must validate that the requested power states in the 263 CPU_SUSPEND request are consistent with the system's state, and that the 264 calling core is the last core running at the requested power level, or deny 265 the request otherwise. 266 267See sections 4.2.3.2, 6.2, and 6.3 of the PSCI spec (DEN0022D.b) for more 268details. 269 270Caveats 271------- 272 273CPU_OFF 274^^^^^^^ 275 276CPU_OFF is always platform-coordinated, regardless of whether the power state 277coordination mode for suspend is platform-coordinated or OS-initiated. If all 278cores in a topology node call CPU_OFF, the last core will power down the node. 279 280In OS-initiated mode, if a subset of the cores in a topology node has called 281CPU_OFF, the last running core may call CPU_SUSPEND to request a powerdown state 282at or above that node's power level. 283 284See section 5.5.2 of the PSCI spec (DEN0022D.b) for more details. 285 286Implementation 287-------------- 288 289Current implementation of platform-coordinated mode 290^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 291 292Platform-coordinated is currently the only supported power state coordination 293mode in TF-A. 294 295The functions of interest in the ``psci_cpu_suspend`` call stack are as follows: 296 297* ``psci_validate_power_state`` 298 299 * This function calls a platform specific ``validate_power_state`` handler, 300 which takes the ``power_state`` parameter, and updates the ``state_info`` 301 object with the requested states for each power level. 302 303* ``psci_find_target_suspend_lvl`` 304 305 * This function takes the ``state_info`` object containing the requested power 306 states for each power level, and returns the deepest power level that was 307 requested to enter a low power state, i.e. the target power level. 308 309* ``psci_do_state_coordination`` 310 311 * This function takes the target power level and the ``state_info`` object 312 containing the requested power states for each power level, and updates the 313 ``state_info`` object with the coordinated target power state for each 314 level. 315 316* ``pwr_domain_suspend`` 317 318 * This is a platform specific handler that takes the ``state_info`` object 319 containing the target power states for each power level, and transitions 320 each power level to the specified power state. 321 322Proposed implementation of OS-initiated mode 323^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 324 325To add support for OS-initiated mode, the following changes are proposed: 326 327* Add a boolean build option ``PSCI_OS_INIT_MODE`` for a platform to enable 328 optional support for PSCI OS-initiated mode. This build option defaults to 0. 329 330.. note:: 331 332 If ``PSCI_OS_INIT_MODE=0``, the following changes will not be compiled into 333 the build. 334 335* Update ``psci_features`` to return 1 in bit[0] to indicate support for 336 OS-initiated mode for CPU_SUSPEND. 337* Define a ``suspend_mode`` enum: ``PLAT_COORD`` and ``OS_INIT``. 338* Define a ``psci_suspend_mode`` global variable with a default value of 339 ``PLAT_COORD``. 340* Implement a new function handler ``psci_set_suspend_mode`` for 341 PSCI_SET_SUSPEND_MODE. 342* Since ``psci_validate_power_state`` calls a platform specific 343 ``validate_power_state`` handler, the platform implementation should populate 344 the ``state_info`` object based on the state ID from the given ``power_state`` 345 parameter. 346* ``psci_find_target_suspend_lvl`` remains unchanged. 347* Implement a new function ``psci_validate_state_coordination`` that ensures the 348 request satisfies the following conditions, and denies any requests 349 that don't: 350 351 * The requested power states for each power level are consistent with the 352 system's state 353 * The calling core is the last core running at the requested power level 354 355 This function differs from ``psci_do_state_coordination`` in that: 356 357 * The ``psci_req_local_pwr_states`` map is not modified if the request were to 358 be denied 359 * The ``state_info`` argument is never modified since it contains the power 360 states requested by the calling OS 361 362* Update ``psci_cpu_suspend_start`` to do the following: 363 364 * If ``PSCI_SUSPEND_MODE`` is ``PLAT_COORD``, call 365 ``psci_do_state_coordination``. 366 * If ``PSCI_SUSPEND_MODE`` is ``OS_INIT``, call 367 ``psci_validate_state_coordination``. If validation fails, propagate the 368 error up the call stack. 369 370* Add a new optional member ``pwr_domain_validate_suspend`` to 371 ``plat_psci_ops_t`` to allow the platform to optionally perform validations 372 based on hardware states. 373 374* The platform specific ``pwr_domain_suspend`` handler remains unchanged. 375 376.. image:: ../resources/diagrams/psci-osi-mode.png 377 378Testing 379------- 380 381The proposed patches can be found at 382https://review.trustedfirmware.org/q/topic:psci-osi. 383 384Testing on FVP and Google platforms 385^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 386 387The proposed patches add a new CPU Suspend in OSI mode test suite to TF-A Tests. 388This has been enabled and verified on the FVP_Base_RevC-2xAEMvA platform and 389Google platforms, and excluded from all other platforms via the build option 390``PLAT_TESTS_SKIP_LIST``. 391 392Testing on STM32MP15 393^^^^^^^^^^^^^^^^^^^^ 394 395The proposed patches have been tested and verified on the STM32MP15 platform, 396which has a single cluster with 2 CPUs, by Gabriel Fernandez 397<[email protected]> from STMicroelectronics with this device tree 398configuration: 399 400.. code-block:: devicetree 401 402 cpus { 403 #address-cells = <1>; 404 #size-cells = <0>; 405 406 cpu0: cpu@0 { 407 device_type = "cpu"; 408 compatible = "arm,cortex-a7"; 409 reg = <0>; 410 enable-method = "psci"; 411 power-domains = <&CPU_PD0>; 412 power-domain-names = "psci"; 413 }; 414 cpu1: cpu@1 { 415 device_type = "cpu"; 416 compatible = "arm,cortex-a7"; 417 reg = <1>; 418 enable-method = "psci"; 419 power-domains = <&CPU_PD1>; 420 power-domain-names = "psci"; 421 }; 422 423 idle-states { 424 cpu_retention: cpu-retention { 425 compatible = "arm,idle-state"; 426 arm,psci-suspend-param = <0x00000001>; 427 entry-latency-us = <130>; 428 exit-latency-us = <620>; 429 min-residency-us = <700>; 430 local-timer-stop; 431 }; 432 }; 433 434 domain-idle-states { 435 CLUSTER_STOP: core-power-domain { 436 compatible = "domain-idle-state"; 437 arm,psci-suspend-param = <0x01000001>; 438 entry-latency-us = <230>; 439 exit-latency-us = <720>; 440 min-residency-us = <2000>; 441 local-timer-stop; 442 }; 443 }; 444 }; 445 446 psci { 447 compatible = "arm,psci-1.0"; 448 method = "smc"; 449 450 CPU_PD0: power-domain-cpu0 { 451 #power-domain-cells = <0>; 452 power-domains = <&pd_core>; 453 domain-idle-states = <&cpu_retention>; 454 }; 455 456 CPU_PD1: power-domain-cpu1 { 457 #power-domain-cells = <0>; 458 power-domains = <&pd_core>; 459 domain-idle-states = <&cpu_retention>; 460 }; 461 462 pd_core: power-domain-cluster { 463 #power-domain-cells = <0>; 464 domain-idle-states = <&CLUSTER_STOP>; 465 }; 466 }; 467 468Testing on Qualcomm SC7280 469^^^^^^^^^^^^^^^^^^^^^^^^^^ 470 471The proposed patches have been tested and verified on the SC7280 platform by 472Maulik Shah <[email protected]> from Qualcomm with this device tree 473configuration: 474 475.. code-block:: devicetree 476 477 cpus { 478 #address-cells = <2>; 479 #size-cells = <0>; 480 481 CPU0: cpu@0 { 482 device_type = "cpu"; 483 compatible = "arm,kryo"; 484 reg = <0x0 0x0>; 485 enable-method = "psci"; 486 power-domains = <&CPU_PD0>; 487 power-domain-names = "psci"; 488 }; 489 490 CPU1: cpu@100 { 491 device_type = "cpu"; 492 compatible = "arm,kryo"; 493 reg = <0x0 0x100>; 494 enable-method = "psci"; 495 power-domains = <&CPU_PD1>; 496 power-domain-names = "psci"; 497 }; 498 499 CPU2: cpu@200 { 500 device_type = "cpu"; 501 compatible = "arm,kryo"; 502 reg = <0x0 0x200>; 503 enable-method = "psci"; 504 power-domains = <&CPU_PD2>; 505 power-domain-names = "psci"; 506 }; 507 508 CPU3: cpu@300 { 509 device_type = "cpu"; 510 compatible = "arm,kryo"; 511 reg = <0x0 0x300>; 512 enable-method = "psci"; 513 power-domains = <&CPU_PD3>; 514 power-domain-names = "psci"; 515 } 516 517 CPU4: cpu@400 { 518 device_type = "cpu"; 519 compatible = "arm,kryo"; 520 reg = <0x0 0x400>; 521 enable-method = "psci"; 522 power-domains = <&CPU_PD4>; 523 power-domain-names = "psci"; 524 }; 525 526 CPU5: cpu@500 { 527 device_type = "cpu"; 528 compatible = "arm,kryo"; 529 reg = <0x0 0x500>; 530 enable-method = "psci"; 531 power-domains = <&CPU_PD5>; 532 power-domain-names = "psci"; 533 }; 534 535 CPU6: cpu@600 { 536 device_type = "cpu"; 537 compatible = "arm,kryo"; 538 reg = <0x0 0x600>; 539 enable-method = "psci"; 540 power-domains = <&CPU_PD6>; 541 power-domain-names = "psci"; 542 }; 543 544 CPU7: cpu@700 { 545 device_type = "cpu"; 546 compatible = "arm,kryo"; 547 reg = <0x0 0x700>; 548 enable-method = "psci"; 549 power-domains = <&CPU_PD7>; 550 power-domain-names = "psci"; 551 }; 552 553 idle-states { 554 entry-method = "psci"; 555 556 LITTLE_CPU_SLEEP_0: cpu-sleep-0-0 { 557 compatible = "arm,idle-state"; 558 idle-state-name = "little-power-down"; 559 arm,psci-suspend-param = <0x40000003>; 560 entry-latency-us = <549>; 561 exit-latency-us = <901>; 562 min-residency-us = <1774>; 563 local-timer-stop; 564 }; 565 566 LITTLE_CPU_SLEEP_1: cpu-sleep-0-1 { 567 compatible = "arm,idle-state"; 568 idle-state-name = "little-rail-power-down"; 569 arm,psci-suspend-param = <0x40000004>; 570 entry-latency-us = <702>; 571 exit-latency-us = <915>; 572 min-residency-us = <4001>; 573 local-timer-stop; 574 }; 575 576 BIG_CPU_SLEEP_0: cpu-sleep-1-0 { 577 compatible = "arm,idle-state"; 578 idle-state-name = "big-power-down"; 579 arm,psci-suspend-param = <0x40000003>; 580 entry-latency-us = <523>; 581 exit-latency-us = <1244>; 582 min-residency-us = <2207>; 583 local-timer-stop; 584 }; 585 586 BIG_CPU_SLEEP_1: cpu-sleep-1-1 { 587 compatible = "arm,idle-state"; 588 idle-state-name = "big-rail-power-down"; 589 arm,psci-suspend-param = <0x40000004>; 590 entry-latency-us = <526>; 591 exit-latency-us = <1854>; 592 min-residency-us = <5555>; 593 local-timer-stop; 594 }; 595 }; 596 597 domain-idle-states { 598 CLUSTER_SLEEP_0: cluster-sleep-0 { 599 compatible = "arm,idle-state"; 600 idle-state-name = "cluster-power-down"; 601 arm,psci-suspend-param = <0x40003444>; 602 entry-latency-us = <3263>; 603 exit-latency-us = <6562>; 604 min-residency-us = <9926>; 605 local-timer-stop; 606 }; 607 }; 608 }; 609 610 psci { 611 compatible = "arm,psci-1.0"; 612 method = "smc"; 613 614 CPU_PD0: cpu0 { 615 #power-domain-cells = <0>; 616 power-domains = <&CLUSTER_PD>; 617 domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>; 618 }; 619 620 CPU_PD1: cpu1 { 621 #power-domain-cells = <0>; 622 power-domains = <&CLUSTER_PD>; 623 domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>; 624 }; 625 626 CPU_PD2: cpu2 { 627 #power-domain-cells = <0>; 628 power-domains = <&CLUSTER_PD>; 629 domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>; 630 }; 631 632 CPU_PD3: cpu3 { 633 #power-domain-cells = <0>; 634 power-domains = <&CLUSTER_PD>; 635 domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>; 636 }; 637 638 CPU_PD4: cpu4 { 639 #power-domain-cells = <0>; 640 power-domains = <&CLUSTER_PD>; 641 domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>; 642 }; 643 644 CPU_PD5: cpu5 { 645 #power-domain-cells = <0>; 646 power-domains = <&CLUSTER_PD>; 647 domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>; 648 }; 649 650 CPU_PD6: cpu6 { 651 #power-domain-cells = <0>; 652 power-domains = <&CLUSTER_PD>; 653 domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>; 654 }; 655 656 CPU_PD7: cpu7 { 657 #power-domain-cells = <0>; 658 power-domains = <&CLUSTER_PD>; 659 domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>; 660 }; 661 662 CLUSTER_PD: cpu-cluster0 { 663 #power-domain-cells = <0>; 664 domain-idle-states = <&CLUSTER_SLEEP_0>; 665 }; 666 }; 667 668Comparisons on Qualcomm SC7280 669^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 670 671CPUIdle states 672~~~~~~~~~~~~~~ 673 674* 8 CPUs, 1 L3 cache 675* Platform-coordinated mode 676 677 * CPUIdle states 678 679 * State0 - WFI 680 * State1 - Core collapse 681 * State2 - Rail collapse 682 * State3 - L3 cache off and system resources voted off 683 684* OS-initiated mode 685 686 * CPUIdle states 687 688 * State0 - WFI 689 * State1 - Core collapse 690 * State2 - Rail collapse 691 692 * Cluster domain idle state 693 694 * State3 - L3 cache off and system resources voted off 695 696.. image:: ../resources/diagrams/psci-flattened-vs-hierarchical-idle-states.png 697 698Results 699~~~~~~~ 700 701* The following stats have been captured with fixed CPU frequencies from the use 702 case of 10 seconds of device idle with the display turned on and Wi-Fi and 703 modem turned off. 704* Count refers to the number of times a CPU or cluster entered power collapse. 705* Residency refers to the time in seconds a CPU or cluster stayed in power 706 collapse. 707* The results are an average of 3 iterations of actual counts and residencies. 708 709.. image:: ../resources/diagrams/psci-pc-mode-vs-osi-mode.png 710 711OS-initiated mode was able to scale better than platform-coordinated mode for 712multiple CPUs. The count and residency results for state3 (i.e. a cluster domain 713idle state) in OS-initiated mode for multiple CPUs were much closer to the 714results for a single CPU than in platform-coordinated mode. 715 716-------------- 717 718*Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.* 719
