AWS S3 with Java SDK

This tutorial will explain how to connect AWS S3 programmatically using AWS Java SDK v2. Perform operations like upload file, download file, delete file and delete multiple files. Connect AWS S3 using the access key and secret key or the IAM role.

Table of Contents

Technology Stack

• Java 16
• AWS SDK for Java version 2
• IDE: Eclipse / Intellij Idea

Prerequisites

• AWS Account
• User in AWS account with Programmatic access – create a separate user.
• EC2 instance with a role having AmanzonS3FullAcess permission – only If you are using the IAM role to connect S3

Maven Dependencies

Let’s start with maven dependencies. First, you have to add a dependency manager with the dependency of software.amazon.awssdk group id. Specify the version of AWS Java SDK you are using. The sample code uses 2.17.189. Once this is added, add the dependency for the S3 artifact as shown in the below code snippet.

...    
<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
	...
	...
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
...	 

There are a few more dependencies related to logging and commons.io. but the above ones are the main dependencies we require. To get the complete code, refer to the link given at the end of this article.

Connect to AWS S3

There are two ways to connect to AWS S3 programmatically.

1. Using accesskey and secretkey
2. Using IAM role

To perform any operation on AWS S3, we should get the object of ‘S3Client’ using any one of the above methods.

Using accesskey and secretkey

To generate the accesskey and secretkey to access S3, it’s better to add a new user with programmatic access and attach ‘AmazonS3FullAccess’ policy. Please refer below images for steps:

The code snippet which will build the S3Client object using accesskey, secretkey is as follows. As you can see, its pretty self explanatory:

private static S3Client buildS3Client(String accessKey, String secretKey, Region region) {
StaticCredentialsProvider staticCredentialsProvider = StaticCredentialsProvider                                        .create(AwsBasicCredentials.create(accessKey, secretKey));
S3Client s3client = S3Client.builder()
                            .credentialsProvider(staticCredentialsProvider)
                            .region(region).build();
   return s3client;
}

Using IAM role

The more secure way to connect to S3 is using the IAM role. If you are accessing S3 from AWS EC2 or any other AWS resource, then you should use the IAM role only.

The steps to create a new role, attaching the S3 access policy, and assign it to EC2 are as follows:

Now, let’s see how we can build an S3Client object which uses the IAM role to connect to AWS S3.

We are going to use ‘InstanceProfileCredentialsProvider’ from AWS SDK which loads credentials from the Amazon EC2 Instance Metadata Service. Here, we don’t have to pass any accesskey or secretkey and that’s the reason it is the most secure way of connecting to S3. The below code snippet will show you how to create the S3Client object using InstanceProfileCredentialsProvider.

private static S3Client buildS3Client(Region region) {
S3Client s3client = S3Client
.builder()
.credentialsProvider(InstanceProfileCredentialsProvider.create())
.region(region).build();
return s3client;
}

Operations on S3 Objects

Now, let’s see how we can use S3Client to perform the following operations on AWS S3:

• Listing objects
• Upload object
• Download object
• Delete object
• Delete multiple objects

Firstly, create an Interface with methods to perform these operations.

public interface S3Service {
    void listObjects(String bucketName);
    void putObject(String bucketName, String destPath, Path sourcePath);
    ResponseInputStream getObject(String bucketName, String objectKey);
    void deleteObject(String bucketName, String objectKey);
    void deleteObjects(String bucketName, String[] objectKeys);
}

Secondly, create an implementation class that accepts S3Client in the constructor and implements methods in the S3Service interface.

//constructor
public S3ServiceImpl(S3Client s3client) {
    this.s3client = s3client;
}

Listing Objects

Let’s start with implementing the method to list the objects in S3. We will use listObjectsV2Paginator the method of S3Client which accepts the ListObjectsV2Request and returns iterable ListObjectsV2Iterable. We will check each object using iterable and print the name and size of the object.

public void listObjects(String bucketName) {
 ListObjectsV2Iterable listObjectsV2Iterable =  s3client.listObjectsV2Paginator(builder -> builder.bucket(bucketName));
 logger.info("Objects in {} bucket: ", bucketName);
 listObjectsV2Iterable.contents().stream()
  .forEach(content -> logger.debug("{} {} bytes", content.key(), content.size()));
}

Upload Object

Our upload method accepts bucket name, destination path in S3, and the Path of the file to upload from the local system. We will use the putObject method of S3Client. This method accepts consumer to build PutObjectRequest using bucket name, destination path in S3, and the Path of the file to upload.

public void putObject(String bucketName, String destPath, Path sourcePath) {
 s3client.putObject(builder -> builder.bucket(bucketName).key(destPath), sourcePath);
}

Download Object

The getObject method will accept the S3 bucket name and path of the file in the bucket. It will return the inputStream which can be used to save the file locally.

public ResponseInputStream getObject(String bucketName, String objectKey) {
    return s3client.getObject(builder -> builder.bucket(bucketName).key(objectKey));
}

Delete Object

The deleteObject method accepts the S3 bucket name and path of the file in the bucket to be deleted.

public void deleteObject(String bucketName, String objectKey) {
    s3client.deleteObject(builder -> builder.bucket(bucketName).key(objectKey));
}

Delete Multiple Objects

Delete multiple objects is a three-step operation. First, we have to get a list of ObjectIdentifier for all objects we want to delete. In the second step, build the DeleteObjectsRequest from ObjectIdentifier list. Lastly, call the deleteObjects method of S3Client.

Our deleteObjects method accepts the bucket name and array of paths of objects to delete.

public void deleteObjects(String bucketName, String[] keys) {
    // Create ObjectIdentifier from keys to delete
    List<ObjectIdentifier> objectIdentifiers = Stream.of(keys)
      .map(key -> ObjectIdentifier.builder().key(key.trim()).build())
      .collect(Collectors.toList());

    // Create DeleteObjectsRequest using objectIdentifiers
    DeleteObjectsRequest deleteObjectsRequest = DeleteObjectsRequest.builder()
              .bucket(bucketName)
              .delete(deleteBuilder -> deleteBuilder.objects(objectIdentifiers))
              .build();

    s3client.deleteObjects(deleteObjectsRequest);
}

Running the application

Our sample program has S3Application.java as main java class. It is an interactive program which will ask inputs like how you want to connect to S3 and what operations you want to perform. Check the entire code in the last section of this article.

Snippet of ouput of running our application is as follows:

INFO  S3Application - Enter bucket name to connect:
your-bucket-name-here
INFO  S3Application - Enter bucket Region: 
you-bucket-region-here
INFO  S3Application - Connect S3 via IAM role? y/n
n

INFO  S3Application - Enter Access Key: 
your-access-key
INFO  S3Application - Enter Secret Key: 
your-secret-key
INFO  S3Application - Enter operation to perform on bucket: your-bucket-name-here . LIST/GET/PUT/DELETE/DELETE-MULTIPLE
put

INFO  S3Application - Enter local full path of file to upload:
D:/uploadFile.txt
INFO  S3Application - Enter full path in bucket:
test-folder/uploadFile.txt
INFO  S3Application - Put Operation Completed
INFO  S3Application - Exiting...

...
...

INFO  S3Application - Enter operation to perform on bucket: your-bucket-name-here . LIST/GET/PUT/DELETE/DELETE-MULTIPLE
list

INFO  S3ServiceImpl - Objects in your-bucket-name-here bucket: 
DEBUG S3ServiceImpl - test-folder/ 0 bytes
DEBUG S3ServiceImpl - test-folder/uploadFile.txt 4 bytes
DEBUG S3ServiceImpl - test-folder/uploadFile2.txt 59 bytes
DEBUG S3ServiceImpl - test-folder/uploadFile3.txt 73 bytes
INFO  S3Application - Listing completed
INFO  S3Application - Exiting...
..
..

INFO  S3Application - Enter operation to perform on bucket: your-bucket-name-here . LIST/GET/PUT/DELETE/DELETE-MULTIPLE
get

INFO  S3Application - Enter full path of object in bucket:
test-folder/uploadFile.txt
INFO  S3Application - Download completed
INFO  S3Application - Exiting...
Note: File will be downloaded in downloads directory in current location.
...
...

INFO  S3Application - Enter operation to perform on bucket: your-bucket-name-here . LIST/GET/PUT/DELETE/DELETE-MULTIPLE
delete

INFO  S3Application - Enter full path of object to DELETE in bucket:
test-folder/fileUpload.txt
INFO  S3Application - Delete Operation Completed
INFO  S3Application - Exiting...
..
..

INFO  S3Application - Enter operation to perform on bucket: your-bucket-name-here . LIST/GET/PUT/DELETE/DELETE-MULTIPLE
delete-multiple
INFO  S3Application - Enter comma separated full path of object to DELETE in bucket:
test-folder/uploadFile2.txt, test-folder/uploadFile3.txt
INFO  S3Application - Delete Operation Completed
INFO  S3Application - Exiting...

Source Code

The complete source code for accessing AWS S3 using AWS Java SDK 2 is available on GitHub.