1*8a52c783SCole Faust# SDK Stability Regression Tests 2*8a52c783SCole Faust 3*8a52c783SCole Faust## Description 4*8a52c783SCole FaustThis module contains stability regression tests. 5*8a52c783SCole Faust 6*8a52c783SCole FaustWe define "stable" to mean that the SDK does not encounter any client-side errors for identified simple expected load 7*8a52c783SCole Faustscenarios, and the number of transient service or network I/O related errors are relatively small. 8*8a52c783SCole Faust 9*8a52c783SCole Faust## Test Case Acceptance 10*8a52c783SCole Faust 11*8a52c783SCole FaustAs these tests will be running against live, running services, we can expect some subset of requests to fail due to transient 12*8a52c783SCole Faustfailures, network disruptions, throttling, etc. We cannot expect that all requests will always succeed. Instead, we will establish 13*8a52c783SCole Faustthat 5% or less of the total number of requests sent for a test case are allowed to fail. We specify that only errors that extend 14*8a52c783SCole Faustfrom SdkServiceException, or a form of network error such as IOException or ReadTimeoutException may be counted towards the 5%. 15*8a52c783SCole FaustAny other error type, such as SdkClientException will fail the test. 16*8a52c783SCole Faust 17*8a52c783SCole Faust 18*8a52c783SCole Faust## How to run 19*8a52c783SCole Faust 20*8a52c783SCole Faust- Run from your IDE 21*8a52c783SCole Faust 22*8a52c783SCole Faust- Run from maven command line 23*8a52c783SCole Faust 24*8a52c783SCole Faust``` 25*8a52c783SCole Faustmvn clean install -P stability-tests -pl :stability-tests 26*8a52c783SCole Faust``` 27*8a52c783SCole Faust 28*8a52c783SCole Faust- Build JAR and use the executable JAR 29*8a52c783SCole Faust 30*8a52c783SCole FaustFirst add tests to TestRunner Class, then run the following command. 31*8a52c783SCole Faust 32*8a52c783SCole Faust``` 33*8a52c783SCole Faustmvn clean install -pl :stability-tests --am -P quick 34*8a52c783SCole Faustmvn clean install -pl :bom-inernal 35*8a52c783SCole Faustcd test/stability-tests 36*8a52c783SCole Faustmvn package -P test-jar 37*8a52c783SCole Faustjava -jar target/stability-tests-uber.jar 38*8a52c783SCole Faust``` 39*8a52c783SCole Faust 40*8a52c783SCole Faust## Adding New Tests 41*8a52c783SCole Faust 42*8a52c783SCole Faust- The tests are built using [JUnit 5](https://junit.org/junit5/). Make sure you are using the correct APIs and mixing of 43*8a52c783SCole FaustJunit 4 and Junit 5 APIs on the same test can have unexpected results. 44*8a52c783SCole Faust 45*8a52c783SCole Faust- All tests should have the suffix of `StabilityTests`, eg: `S3StabilityTests` 46*8a52c783SCole Faust 47*8a52c783SCole Faust 48*8a52c783SCole Faust 49