Name Date Size #Lines LOC

..--

README.mdH A D25-Apr-202520.4 KiB532371

README.md

1**Design:** New Feature, **Status:** [Released](../../../README.md)
2
3# Request Presigners
4
5"Presigned URLs" are a generic term usually used for an AWS request that has been signed using
6[SigV4's query parameter signing](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) so that it can be
7invoked by a browser, within a certain time period.
8
9The 1.x family of SDKs is able to generate presigned requests of multiple types, with S3's GetObject being the most
10frequently-used flavor of presigned URL. Customers have been [very](https://github.com/aws/aws-sdk-java-v2/issues/203)
11[vocal](https://dzone.com/articles/s3-and-the-aws-java-sdk-20-look-before-you-leap) about this feature not yet being included
12in the 2.x SDK. This document proposes how presigned URLs should be supported by the Java SDK 2.x.
13
14**What is request presigning?**
15
16Request presigning allows a **signature creator** to use their secret **signing credentials** to generate an AWS request. This
17**presigned request** can be executed by a separate **signature user** within a fixed time period, without any additional
18authentication required.
19
20For example, a support engineer for a backend service may temporarily share a service log with a customer by: (1) uploading
21the logs to S3, (2) presigning an S3 GetObject request for the logs, (3) sending the presigned request to the customer. The
22customer can then download the logs using the presigned request. The presigned request would remain valid until it "expires" at
23a time specified by the support engineer when the request was signed.
24
25**What is a presigned URL?**
26
27Colloquially, most people wouldn't consider every presigned request to be a "presigned URL". For example, a presigned DynamoDb
28PutItem can be executed by a signature user, but it would require the signature creator to share the headers and payload that
29were included when the signature was generated, so that the signature user can send them along with the request.
30
31For the purposes of this document:
321. A **presigned request** is any request signed using query parameter signing with the express purpose of another entity
33executing the presigned request at a later time.
342. A **presigned URL** is a presigned request that: (1) does not include a payload, (2) does not include content-type or x-amz-*
35headers, and (3) uses the GET HTTP method.
36
37This distinction is useful, because a presigned URL can be trivially executed by a browser.
38
39*Example*
40
41Under this definition, a presigned S3 GetObjectRequest is a presigned URL if and only if it does not include one of the
42following fields:
43
441. sseCustomerAlgorithm (Header: x-amz-server-side-encryption-customer-algorithm)
452. sseCustomerKey (Header: x-amz-server-side-encryption-customer-key)
463. sseCustomerKeyMD5 (Header: x-amz-server-side-encryption-customer-key-MD5)
474. requesterPays (Header: x-amz-request-payer)
48
49If these fields were included when the presigned request was generated and the URL was opened in a browser, the signature user
50will get a "signature mismatch" error. This is because these headers are included in the signature, but these values are not
51sent by a browser.
52
53
54
55## Proposed APIs
56
57The SDK 2.x will support both presigned requests and presigned URLs. The API will make it easy to distinguish between the two,
58and make it possible to support executing presigned requests using HTTP clients that implement the AWS SDK HTTP client blocking
59or non-blocking SPI.
60
61*FAQ Below: "What about execution of presigned requests?"*
62
63To more quickly address a very vocal desire for presigned URLs, the first iteration will only support generating presigned S3
64GetObject requests. Later milestones will add other operations and services as well as code-generated presigners.
65
66*Section Below: "Milestones"*
67
68### Usage Examples
69
70#### Example 1: Generating presigned requests
71
72```Java
73S3Presigner s3Presigner = S3Presigner.create();
74PresignedGetObjectRequest presignedRequest =
75        s3Presigner.presignGetObject(r -> r.getObject(get -> get.bucket("bucket").key("key"))
76                                           .signatureDuration(Duration.ofMinutes(15)));
77URL URL = presignedRequest.url();
78```
79
80#### Example 2: Determining whether the presigned request will work in a browser
81
82```Java
83S3Presigner s3Presigner = S3Presigner.create();
84PresignedGetObjectRequest presignedRequest = s3Presigner.presignGetObject(...);
85
86Validate.isTrue(presignedRequest.isBrowserCompatible());
87
88System.out.println("Click the following link to download the object: " + presignedRequest.url());
89```
90
91#### Example 3: Executing the presigned request using the URL connection HTTP client.
92
93```Java
94S3Presigner s3Presigner = S3Presigner.create();
95PresignedGetObjectRequest presignedRequest = s3Presigner.presignGetObject(...);
96
97try (SdkHttpClient httpClient = UrlConnectionHttpClient.create()) {
98    ContentStreamProvider payload = presignedRequest.payload()
99                                                    .map(SdkBytes::asInputStream)
100                                                    .map(is -> () -> is)
101                                                    .orElseNull();
102
103    HttpExecuteRequest executeRequest =
104            HttpExecuteRequest.builder()
105                              .request(presignedRequest.httpRequest())
106                              .contentStreamProvider(payload)
107                              .build();
108
109    HttpExecuteResponse httpRequest = client.prepareRequest(executeRequest).call();
110
111    Validate.isTrue(httpRequest.httpResponse().isSuccessful());
112}
113```
114
115### `{Service}Presigner`
116
117A new class will be created for each service: `{Service}Presigner` (e.g. `S3Presigner`). This follows the naming strategy
118established by the current `{Service}Client` and `{Service}Utilities` classes.
119
120#### Example
121
122```Java
123/**
124 * Allows generating presigned URLs for supported S3 operations.
125 */
126public interface S3Presigner {
127    static S3Presigner create();
128    static S3Presigner.Builder builder();
129
130    /**
131     * Presign a `GetObjectRequest` so that it can be invoked directly via an HTTP client.
132     */
133    PresignedGetObjectRequest presignGetObject(GetObjectPresignRequest request);
134    PresignedGetObjectRequest presignGetObject(Consumer<GetObjectPresignRequest.Builder> request);
135
136    interface Builder {
137        Builder region(Region region);
138        Builder credentialsProvider(AwsCredentialsProvider credentials);
139        Builder endpointOverride(URL endpointOverride);
140        // ...
141        S3Presigner build();
142    }
143}
144```
145
146#### Instantiation
147
148This class can be instantiated in a few ways:
149
150**Create method**
151
152Uses the default region / credential chain, similar to `S3Client.create()`.
153
154```Java
155S3Presigner s3Presigner = S3Presigner.create();
156```
157
158**Builder**
159
160Uses configurable region / credentials, similar to `S3Client.builder().build()`.
161
162```Java
163S3Presigner s3Presigner = S3Presigner.builder().region(Region.US_WEST_2).build();
164```
165
166**From an existing S3Client**
167
168Uses the region / credentials from an existing `S3Client` instance, similar to `s3.utilities()`.
169
170```Java
171S3Client s3 = S3Client.create();
172S3Presigner s3Presigner = s3.presigner();
173```
174
175**From the S3 gateway class**
176
177(Implementation date: TBD) A discoverable alias for the `create()` and `builder()` methods.
178
179```Java
180S3Presigner s3Presigner = S3.presigner();
181S3Presigner s3Presigner = S3.presignerBuilder().build();
182```
183
184#### Methods
185
186A method will be generated for each operation: `Presigned{Operation}Request presign{Operation}({Operation}PresignRequest)`
187(e.g. `PresignedGetObjectRequest presignGetObject(GetObjectPresignRequest)`).
188
189*FAQ Below: "Why a different input shape per operation?" and "Why a different output shape per operation?"*.
190
191#### Inner-Class: `{Service}Presigner.Builder`
192
193An inner-class will be created for each service presigner: `{Service}Presigner.Builder` (e.g. `S3Presigner.Builder`).
194This follows the naming strategy established by the current `{Service}Utilities` classes.
195
196##### Methods
197
198The presigner builder will have at least the following configuration:
199
2001. `region(Region)`: The region that should be used when generating the presigned URLs.
2012. `endpointOverride(URI)`: An endpoint that should be used in place of the one derived from the region.
2023. `credentialsProvider(AwsCredentialsProvider)`: The credentials that should be used when signing the request.
203
204Additional configuration will be added later after more investigation (e.g. signer, service-specific configuration).
205
206### `{Operation}PresignRequest`
207
208A new input class will be generated for each operation that supports presigning. These requests
209will extend a common base, allowing for common code to be used to configure the request.
210
211*FAQ Below: "Why a different input shape per operation?"*.
212
213#### Example
214
215```Java
216/**
217 * A request to generate presigned GetObjectRequest, passed to S3Presigner#getObject.
218 */
219public interface GetObjectPresignRequest extends PresignRequest {
220    /**
221     * The GetObjectRequest that should be presigned.
222     */
223    GetObjectRequest getObject();
224    // Plus builder boilerplate
225}
226
227public interface PresignRequest {
228    /**
229     * The duration for which this presigned request should be valid. After this time has expired,
230     * attempting to use the presigned request will fail.
231     */
232    Duration signatureDuration();
233    // Plus builder boilerplate
234}
235```
236
237### `Presigned{Operation}Request`
238
239A new output class will be generated for each operation that supports presigning. These presigned requests
240will extend a common base, allowing for common code to be used to process the response.
241
242*FAQ Below: "Why a different output shape per operation?"*.
243
244#### Example
245
246```Java
247/**
248 * A presigned GetObjectRequest, returned by S3Presigner#getObject.
249 */
250public interface PresignedGetObjectRequest extends PresignedRequest {
251    // Builder boilerplate
252}
253
254/**
255 * A generic presigned request. The isBrowserCompatible method can be used to determine whether this request
256 * can be executed by a web browser.
257 */
258public interface PresignedRequest {
259    /**
260     * The URL that the presigned request will execute against. The isBrowserCompatible method can be used to
261     * determine whether this request will work in a browser.
262     */
263    URL url();
264
265    /**
266     * The exact SERVICE time that the request will expire. After this time, attempting to execute the request
267     * will fail.
268     *
269     * This may differ from the local clock, based on the skew between the local and AWS service clocks.
270     */
271    Instant expiration();
272
273    /**
274     * Returns true if the url returned by the url method can be executed in a browser.
275     *
276     * This is true when the HTTP request method is GET, and hasSignedHeaders and hasSignedPayload are false.
277     *
278     * TODO: This isn't a universally-agreed-upon-good method name. We should iterate on it before GA.
279     */
280    boolean isBrowserCompatible();
281
282    /**
283     * Returns true if there are signed headers in the request. Requests with signed headers must have those
284     * headers sent along with the request to prevent a "signature mismatch" error from the service.
285     */
286    boolean hasSignedHeaders();
287
288    /**
289     * Returns the subset of headers that were signed, and MUST be included in the presigned request to prevent
290     * the request from failing.
291     */
292    Map<String, List<String>> signedHeaders();
293
294    /**
295     * Returns true if there is a signed payload in the request. Requests with signed payloads must have those
296     * payloads sent along with the request to prevent a "signature mismatch" error from the service.
297     */
298    boolean hasSignedPayload();
299
300    /**
301     * Returns the payload that was signed, or Optional.empty() if hasSignedPayload is false.
302     */
303    Optional<SdkBytes> signedPayload();
304
305    /**
306     * The entire SigV4 query-parameter signed request (minus the payload), that can be transmitted as-is to a
307     * service using any HTTP client that implement the SDK's HTTP client SPI.
308     *
309     * This request includes signed AND unsigned headers.
310     */
311    SdkHttpRequest httpRequest();
312
313    // Plus builder boilerplate
314}
315```
316
317
318
319## Milestones
320
321### M1: Hand-Written S3 GetObject Presigner
322
323**Done When:** Customers can use an SDK-provided S3 presigner to generate presigned S3 GetObject requests.
324
325**Expected Tasks:**
326
3271. Hand-write relevant interfaces and class definitions as described in this document.
3282. DO NOT create the `S3Client#presigner` method, yet.
3293. Implement the `presignGetObject` method using minimum refactoring of core classes.
330
331### M2: Hand-Written S3 PutObject Presigner
332
333**Done When:** Customers can use an SDK-provided S3 presigner to generate presigned S3 PutObject requests.
334
335**Expected Tasks:**
336
3371. Hand-write relevant interfaces and class definitions as described in this document.
3382. Implement the `presignPutObject` method using minimum refactoring of core classes.
339
340### M3: Hand-Written Polly SynthesizeSpeech Presigner
341
342**Done When:** Customers can use an SDK-provided Polly presigner to generate presigned Polly SynthesizeSpeech requests.
343
344**Expected Tasks:**
345
3461. Hand-write relevant interfaces and class definitions as described in this document.
3472. Hand-create a `SynthesizeSpeech` marshaller that generates browser-compatible HTTP requests.
3483. Implement the `presignSynthesizeSpeech` method using minimum refactoring of core classes. (Considering whether
349or not the browser-compatible HTTP request marshaller should be the default).
350
351### M4: Generated Presigners
352
353**Done When:** The existing presigners are generated, and customers do not have to make any code changes.
354
355**Expected Tasks:**
356
3571. Refactor core classes to remove unnecessary duplication between `presignGetObject`, `presignPutObject`,
358and `presignSynthesizeSpeech`.
3592. Implement customization for allowing use of the browser-compatible `SynthesizeSpeech` marshaller.
3603. Update hand-written `presign*` methods to use the refactored model.
3614. Update code generation to generate the `presign*` inputs/outputs.
3625. Update code generation to generate the `{Service}Presigner` classes.
363
364### M5: Generate Existing 1.11.x Presigners
365
366**Done When:** Any operation presigners that exist in 1.11.x are available in 2.x.
367
368**Expected Tasks:**
369
3701. Generate EC2 Presigners
3712. Generate RDS Presigners
372
373### M6: Generate All Presigners
374
375**Done When:** All operations (that can support presigning) support presigning.
376
377*FAQ Below: "For which operations will we generate a URL presigner?"*
378
379**Expected Tasks:**
380
3811. Testing generated presigners for a representative sample of operations
382
383### M7: Instantiation and Discoverability Simplifications
384
385**Done When:** All clients contain a `{Service}Client#presigner()` method.
386
387**Expected Tasks:**
388
3891. Determine which pieces of configuration need to be inherited from the service clients, and how their inheritance will work
390   (e.g. how will execution interceptors work?).
3912. Determine whether async clients will have a separate presigner interface, to be forward-compatible if we add blocking
392   credential/region providers.
3933. Update the generated clients to support inheriting this configuration
394   in a sane way.
3954. Generate this method in the client, for all supported services.
396
397
398
399## FAQ
400
401### For which Services will we generate URL presigners?
402
403We will generate a `{Service}Presigner` class if the service has any operations that need presigner support.
404
405### For which operations will we generate a URL presigner?
406
407The support operations vary based on the implementation milestone (see Milestones above).
408
409The set of supported operations assumed by this document is ALL operations, except ones with signed, streaming
410payloads. Signed, streaming payloads require additional modeling (e.g. of chunked encoded payloads or event
411streaming) and can be  implemented at a later time after additional design, if there is sufficient customer
412demand.
413
414### Why a different input shape per operation?
415
416The input shape must be aware of the operation inputs, so the options are: (1) a different, generated input shape per operation,
417or (2) a common "core" input shape that is parameterized with the input shape.
418
419The following compares Option 1 to Option 2, in the interest of illustrating why Option 1 was chosen.
420
421**Option 1:** `presignGetObject(GetObjectPresignRequest)` and `presignGetObject(Consumer<GetObjectPresignRequest.Builder>)`:
422Generated `GetObjectPresignRequest`
423
424```Java
425s3.presignGetObject(GetObjectPresignRequest.builder()
426                                           .getObject(GetObjectRequest.builder().bucket("bucket").key("key").build())
427                                           .signatureDuration(Duration.ofMinutes(15))
428                                           .build());
429```
430
431```Java
432s3.presignGetObject(r -> r.signatureDuration(Duration.ofMinutes(15))
433                          .getObject(go -> go.bucket("bucket").key("key")));
434```
435
436**Option 2:** `presignGetObject(PresignRequest<GetObject>)` and
437`presignGetObject(GetObject, Consumer<PresignRequest.Builder<GetObject>>)`
438
439```Java
440s3.presignGetObject(PresignRequest.builder(GetObjectRequest.builder().bucket("bucket").key("key").build())
441                                  .signatureDuration(Duration.ofMinutes(15))
442                                  .build());
443```
444
445```Java
446s3.presignGetObject(GetObjectRequest.builder().bucket("bucket").key("key").build(),
447                    r -> r.signatureDuration(Duration.ofMinutes(15)));
448```
449
450**Option 1 Pros:**
451
4521. More readable to entry-level developers
4532. Closer to `S3Client` signature
4543. Simpler construction (operation input is not needed to instantiate builder)
4554. Much simpler `Consumer<Builder>` method variant
456
457**Option 2 Pros:**
458
4591. Smaller jar size
460
461### Why a different output shape per operation?
462
463This decision is much less obvious. The output shape does not technically need to understand the input shape,
464so the options are: (1) a different, generated output shape per operation, (2) return the `PresignedRequest`
465directly.
466
467The following compares Option 1 and Option 2, in the interest of illustrating why Option 1 was chosen.
468
469**Option 1:** `PresignedGetObjectRequest presignGetObject(GetObjectPresignRequest)`
470
471```Java
472PresignedGetObjectRequest presignedRequest = s3.presignGetObject(...);
473URL presignedUrl = presignedRequest.getUrl();
474```
475
476**Option 2:** `PresignedRequest presignGetObject(GetObjectPresignRequest)``
477
478```Java
479PresignedRequest presignedRequest = s3.presignGetObject(...);
480URL presignedUrl = presignedRequest.getUrl();
481```
482
483**Option 1 Pros:**
484
4851. Makes type-safe execution of presigned requests possible.
4862. Closest to `S3Client` method signature
487
488**Option 2 Pros:**
489
4901. Smaller jar size
491
492**Decision:** Option 1 will be used, because the cost of an empty interface is very small, and it enables
493support for type-safe execution of presigned requests in the future.
494
495*FAQ Below: "What about execution of presigned requests?"*
496
497### What about execution of presigned requests?
498
499The design proposed above makes it possible to execute the signed requests using any HTTP client that implements
500the AWS SDK's HTTP client SPI.
501
502In the future, it would be beneficial to allow a signature user to **execute** a presigned URL using the entire
503SDK stack, instead of just the HTTP client portion. This would enable:
504
5051. Automatic retries, in the case of network or service outages
5062. Response shape unmarshalling, in the case of modeled responses
5073. SDK metric integration (once implemented)
508
509As an example (this is not a design proposal), if the `DynamoDbClient` supported executing a presigned URL, it would be
510beneficial to make sure that the request was for the correct operation, so that the retries and response processing are
511appropriate for the service/operation.
512
513```Java
514DynamoDbClient dynamo = DynamoDbClient.create();
515PresignedPutItemRequest presignedRequest = dynamo.presigner().presignPutItem(...);
516PutItemResponse response = dynamo.putItem(presignedRequest);
517```
518
519### What about non-blocking request presigning?
520
521The proposal above does not distinguish between blocking or non-blocking request presigning. This is because the
522SDK currently only distinguishes between blocking and non-blocking, when it comes to an HTTP client implementation.
523
524The generated presigned request can be executed by a blocking OR non-blocking HTTP client.
525
526In the future, the SDK could implement non-blocking region providers and non-blocking credential providers, at which
527time it could become relevant to distinguish between blocking or non-blocking URL presigners.
528
529For this reason, we will need to decide whether the `presigner()` method to get to a pre-configured URL presigner will
530only be included on the blocking `{Service}Client` (with a separate non-blocking `{Service}Presigner` async class for the
531`{Service}AsyncClient`s), or whether we should use the same `{Service}Presigner` for sync and async clients.
532