README.md
1## Overview
2
3The S3 Transfer Manager is a high-level transfer utility built on top of the asynchronous S3 client.
4It provides a simple API to allow you to transfer files and directories between your application
5and Amazon S3. The S3 Transfer Manager also enables you to monitor a transfer's progress in real-time,
6as well as pause the transfer for execution at a later time.
7
8## Getting Started
9
10### Add a dependency for the S3 Transfer Manager
11
12First, you need to include `s3-transfer-manager` and `aws-crt` in your project.
13
14```xml
15<dependency>
16 <groupId>software.amazon.awssdk</groupId>
17 <artifactId>s3-transfer-manager</artifactId>
18 <version>${awsjavasdk.version}</version>
19</dependency>
20<dependency>
21 <groupId>software.amazon.awssdk.crt</groupId>
22 <artifactId>aws-crt</artifactId>
23 <version>${awscrt.version}</version>
24</dependency>
25```
26
27Note that you need to replace `${awsjavasdk.version}` and `${awscrt.version}` with the latest
28version.
29
30### Instantiate the S3 Transfer Manager
31
32You can instantiate the transfer manager easily using the default settings:
33
34```java
35S3TransferManager transferManager = S3TransferManager.create();
36```
37
38If you wish to configure settings, or use an underlying CRT-based S3 client you have already constructed,
39we recommend using the builder instead:
40
41
42```java
43S3AsyncClient s3AsyncClient =
44 S3AsyncClient.crtBuilder()
45 .credentialsProvider(DefaultCredentialsProvider.create())
46 .region(Region.US_WEST_2)
47 .targetThroughputInGbps(20.0)
48 .minimumPartSizeInBytes(8 * MB)
49 .build();
50
51S3TransferManager transferManager =
52 S3TransferManager.builder()
53 .s3Client(s3AsyncClient)
54 .build();
55```
56
57### Transfer a single object
58
59#### Upload a file to S3 and log the upload’s progress with a TransferListener
60To upload a file to Amazon S3, you need to provide the source file path and a PutObjectRequest specifying the target bucket and key.
61Optionally, you can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener
62logs a basic progress bar; users can also implement their own listeners.
63
64```java
65S3TransferManager transferManager = S3TransferManager.create();
66
67UploadFileRequest uploadFileRequest =
68 UploadFileRequest.builder()
69 .putObjectRequest(req -> req.bucket("bucket").key("key"))
70 // attaching a LoggingTransferListener that will log the progress
71 .addTransferListener(LoggingTransferListener.create())
72 .source(Paths.get("myFile.txt"))
73 .build();
74
75FileUpload upload = transferManager.uploadFile(uploadFileRequest);
76
77 // Wait for the transfer to complete
78 upload.completionFuture().join();
79```
80
81#### Download an S3 object to a local file and log the download’s progress with a TransferListener
82
83To download an object, you need to provide the destination file path and a GetObjectRequest specifying the source bucket and key.
84Same as upload, you can monitor the progress of the transfer by attaching a TransferListener.
85
86```java
87S3TransferManager transferManager = S3TransferManager.create();
88
89DownloadFileRequest downloadFileRequest =
90DownloadFileRequest.builder()
91 .getObjectRequest(req -> req.bucket("bucket").key("key"))
92 .destination(Paths.get("myFile.txt"))
93 // attaching a LoggingTransferListener that will log the progress
94 .addTransferListener(LoggingTransferListener.create())
95 .build();
96
97FileDownload download = transferManager.downloadFile(downloadFileRequest);
98
99// Wait for the transfer to complete
100download.completionFuture().join();
101```
102
103#### Copy an S3 object from one location to another
104To copy an object, you need to provide a CopyObjectRequest with source and destination location.
105
106```
107S3TransferManager transferManager = S3TransferManager.create();
108CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
109 .sourceBucket("source_bucket")
110 .sourceKey("source_key")
111 .destinationBucket("dest_bucket")
112 .destinationKey("dest_key")
113 .build();
114CopyRequest copyRequest = CopyRequest.builder()
115 .copyObjectRequest(copyObjectRequest)
116 .build();
117
118Copy copy = transferManager.copy(copyRequest);
119
120// Wait for the transfer to complete
121CompletedCopy completedCopy = copy.completionFuture().join();
122```
123
124### Transfer multiple objects in the same directory
125
126#### Upload a local directory to an S3 bucket
127
128To upload a local directory recursively to an S3 bucket, you need to provide the source directory and the target bucket.
129
130```java
131S3TransferManager transferManager = S3TransferManager.create();
132DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder()
133 .sourceDirectory(Paths.get("source/directory"))
134 .bucket("bucket")
135 .build());
136
137// Wait for the transfer to complete
138CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();
139
140// Print out any failed uploads
141completedDirectoryUpload.failedTransfers().forEach(System.out::println);
142```
143
144#### Download S3 objects within the same bucket to a local directory
145
146To download all S3 objects within the same bucket, you need to provide the destination directory and the source bucket.
147
148```java
149S3TransferManager transferManager = S3TransferManager.create();
150 DirectoryDownload directoryDownload =
151 transferManager.downloadDirectory(
152 DownloadDirectoryRequest.builder()
153 .destination(Paths.get("destination/directory"))
154 .bucket("bucket")
155 // only download objects with prefix "photos"
156 .listObjectsV2RequestTransformer(l -> l.prefix("photos"))
157 .build());
158// Wait for the transfer to complete
159CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();
160
161// Print out any failed downloads
162completedDirectoryDownload.failedTransfers().forEach(System.out::println);
163```
164