Gradle – Ling's Note

[AWS] Create file checksum file when S3 Object uploaded

C.H. Ling — Mon, 16 Aug 2021 07:15:34 +0000

Checksum ensure the integrity of file. In AWS S3, it also using eTag to do similar things. It create MD5 value and store in eTag. However, it is not a real checksum for s3 object itself. It incudes s3 object metadata. As result, it might not the same if download file and generate checksum by own.

To create a real file checksum, it can be done by Lambda trigger. With lambda function, it can create trigger to execute task when file uploaded. This demo, it it will use Java to create lambda trigger to create MD5 and SHA-256 checksum.

Pre-requests

AWS account should has sufficient privilege to create lambda function and list s3 buckets;
User account for execute Lambda function should be configurated;

Steps

Create Lambda function.
In AWS Management Console, click Create Function, input Function name and select Runtime to Java 11 (Corretto), then click Create Function.
Add Trigger
Open created trigger, click Add trigger.
Configure new trigger
In Add trigger menu, select S3 as trigger, then select target bucket in Bucket dropdown. Optionally, it can add s3 object key prefix and suffix in Prefix and suffix textbox.
After tick Recursive invocation consent, click Add.
Create gradle library project
In command prompt / terminal create project with gradle command as below.
```
gradle init
```

Add dependency.
In build.gradle, add line below to setup dependency and build task.

dependencies {
    implementation 'com.amazonaws:aws-lambda-java-core:1.2.1'
    implementation 'com.amazonaws:aws-lambda-java-events:3.9.0'
    runtimeOnly 'com.amazonaws:aws-lambda-java-log4j2:1.2.0'

    implementation 'software.amazon.awssdk:s3:2.17.9'

    // Use JUnit Jupiter for testing.
    testImplementation 'org.junit.jupiter:junit-jupiter:5.7.1'
}

task("buildZip", type: Zip) {
    from compileJava
    from processResources
    into("lib") {
        from configurations.runtimeClasspath
    }
}

test {
    useJUnitPlatform()
}

java {
    sourceCompatibility = JavaVersion.VERSION_11
    targetCompatibility = JavaVersion.VERSION_11
}

build.dependsOn buildZip

Add handler.
Create new class file named GenerateChecksumEventHandler and add code below.

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;

public class ChecksumEventHandler implements RequestHandler {
    private static final String ENVIRONMENT_VARIABLE_ACCESS_KEY_ID="ACCESS_KEY_ID";
    private static final String ENVIRONMENT_VARIABLE_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY";

    @Override
    public Void handleRequest(S3EventNotification input, Context context) {
        // Initial settings.
        LambdaLogger logger = context.getLogger();
        String s3AccessKeyId = System.getenv(ENVIRONMENT_VARIABLE_L2_REMOTE_ACCESS_KEY_ID);
        String s3SecretAccessKey = System.getenv(ENVIRONMENT_VARIABLE_L2_REMOTE_SECRET_ACCESS_KEY);

        // Connect to s3 with specific access key and secret access key which store in environment variable.
        AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(
                s3AccessKeyId,
                s3SecretAccessKey);
        S3Client s3Client = S3Client.builder().credentialsProvider(StaticCredentialsProvider.create(awsCredentials)).build();

        // Loop with uploaded files.
        input.getRecords().forEach(o-> {
            // Get related s3 object metadata and content.
            String bucketName = o.getS3().getBucket().getName();
            String objectKey = o.getS3().getObject().getKey();
            GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket(bucketName).key(objectKey).build();
            ResponseBytes response = s3Client.getObjectAsBytes(getObjectRequest);
            InputStream objectByte = new ByteArrayInputStream(response.asByteArray());

            // Generate checksum file and store in same bucket.
            generateChecksumFile(logger, bucketName, objectKey, objectByte, "MD5", "md5", s3Client);
            generateChecksumFile(logger, bucketName, objectKey, objectByte, "SHA-256", "sha256", s3Client);
        });
        return null;
    }

    /**
     * Generate checksum with specific algorithm.
     * @param logger injected logger.
     * @param bucketName s3 bucket name.
     * @param objectKey s3 object key name.
     * @param inputStream s3 object input stream.
     * @param algorithm checksum algorithm required.
     * @param fileExtension checksum file extension.
     * @param s3Client injected s3 client.
     */
    private void generateChecksumFile(LambdaLogger logger, String bucketName, String objectKey, InputStream inputStream, String algorithm, String fileExtension, S3Client s3Client) {
        try {
            inputStream.reset();
            String checksum = generateChecksum(inputStream, algorithm);
            logger.log(algorithm+ " checksum = "+ checksum);
            String checksumObjectName = objectKey+ "." +fileExtension;
            RequestBody s3ObjectBody = RequestBody.fromString(checksum);
            PutObjectRequest putObjectRequest = PutObjectRequest.builder().bucket(bucketName).key(checksumObjectName).build();
            PutObjectResponse putObjectResponse = s3Client.putObject(putObjectRequest, s3ObjectBody);
            logger.log("Checksum stored and upload to bucket "+bucketName+"; path = "+ checksumObjectName+ ", e-tag = "+putObjectResponse.eTag()+".");
        } catch (NoSuchAlgorithmException | IOException e) {
            logger.log(e.getMessage());
            logger.log(Arrays.toString(e.getStackTrace()));
        }
    }

    /**
     * Generate checksum with specific algorithm.
     * @param inputStream object stream to be hash.
     * @param algorithm checksum algorithm.
     * @return Checksum in string format.
     * @throws NoSuchAlgorithmException Invalid input algorithm.
     * @throws IOException Exception when read / write byte.
     */
    private String generateChecksum(InputStream inputStream, String algorithm) throws NoSuchAlgorithmException, IOException {
        MessageDigest messageDigest = MessageDigest.getInstance(algorithm);

        DigestInputStream digestInputStream = new DigestInputStream(inputStream, messageDigest);
        byte[] buffer = new byte[4096];
        int count = 0;
        while (digestInputStream.read(buffer) > -1) {
            count++;
        }
        MessageDigest digest = digestInputStream.getMessageDigest();
        digestInputStream.close();

        byte[] checksum = digest.digest();
        StringBuilder sb = new StringBuilder();
        for (byte b : checksum) {
            sb.append(String.format("%02X", b));
        }
        return sb.toString().toUpperCase();
    }
}

As it will generate file and upload to s3 bucket, so it is required to use AWS SDK and put object permission.

Build distributed file.
In command prompt / terminal, execute command below.
```
gradle clean buildZip
```
Upload file.
In Lambda function, select Code tab, click Upload From > zip of jar file, select built file and upload it.
Add credential.
In lambda function, select Configuration > Environment variables, click Edit, then add parameter below and set access key and secret key.
Create Test
In lambda function, select Test tab, click new event and input event name, then input json below. Beware bucket name, arn and object key should align with triggered s3 bucket and its object.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "ap-southeast-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "bucket": {
          "name": "test-bucket",
          "arn": "arn:aws:s3:::test-bucket"
        },
        "object": {
          "key": "test.zip"
        }
      }
    }
  ]
}

Execute test.
Click Test to execute test, expected it will result success.

[Java] 在Gradle project 中加入Swagger

C.H. Ling — Wed, 15 Aug 2018 06:35:01 +0000

Swagger 是一套 Web API 管理工具, 除了一般測試外, 還可以當成API 的user menu. 而Swagger 亦已經有library 使整合變得更方便. 示範中會將swagger 整合到spring RESTful API 中.

加入dependency.
在API project 中內的 build.gradle 加入以下dependencies.

dependencies {
   // Add Swagger.
   compile("io.springfox:springfox-swagger2:2.7.0")
   compile("io.springfox:springfox-swagger-ui:2.7.0")
}

設定swagger.
於project 中建立java 檔, 命名為SwaggerConfig.java, 並輸入以下代碼:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import springfox.documentation.builders.PathSelectors;
import springfox.documentation.builders.RequestHandlerSelectors;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;

@Configuration
@EnableSwagger2
public class SwaggerConfig {                                    
    @Bean
    public Docket api() { 
        return new Docket(DocumentationType.SWAGGER_2)  
          .select()                                  
          .apis(RequestHandlerSelectors.any())              
          .paths(PathSelectors.any())                          
          .build();                                           
    }
}

設定與Spring 的整合.
於project 中建立java 檔, 命名為SwaggerWebMvcConfigurationAdapter.java, 並輸入以下代碼:

import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;

@SuppressWarnings("deprecation")
@EnableWebMvc
public class SwaggerWebMvcConfigurationAdapter extends WebMvcConfigurerAdapter {
	@Override
	public void addResourceHandlers(ResourceHandlerRegistry registry) {
	    registry.addResourceHandler("swagger-ui.html")
	      .addResourceLocations("classpath:/META-INF/resources/");
	 
	    registry.addResourceHandler("/webjars/**")
	      .addResourceLocations("classpath:/META-INF/resources/webjars/");
	}
}

進行測試.
執行Project, 並於瀏覽器輸入以下網址. 若見到swagger 網頁顯示, 代表測試成功.
```
http://localhost:8080/swagger-ui.html
```

[Gradle] 利用profile 打包檔案

C.H. Ling — Thu, 02 Aug 2018 08:15:20 +0000

進行deployment 時, 總需要於properties 檔中進行設定, 除了容易出現人為錯誤, 而且費時失事. 在Maven 中, 可以透過修改pom.xml 進行profile 設定; 而Gradle 中, 亦有同樣的設定.

建立不同的profile 檔.
於%PROJECT_HOME%/src/main/resources 中, 將原本的application.properties 複制數份, 並基於profile 命名如下.
```
appication_<>.properties
```
修改不同profile properties 檔設定並儲存.

設定複制設定.
於build.gradle 中加入以下代碼.

processResources {
    def defaultProfile = 'test'
    def profile = (project.hasProperty('profile') ? project.profile : defaultProfile).toLowerCase()
    println 'Profile: '+profile
    include "**/application_${profile}.properties"
    rename {
        'application.properties'
    }
}

在processResources() 中, 預設了test 為default profile, 若加入了profile 的話, 則會找出相對應的profile 取代原本的application.properties.

測試打包檔.
於command prompt 中輸入以下指令:
```
cd %PROJECT_DIR%
gradlew build -Pprofile <>
```
參數-P 代表parameter, profile 則是profile 設定
檢查打包檔.
利用解壓軟件打開打包檔, 打開WEB-INF/classes/application.properties, 若見到設定正確, 代表檔案複製成功.

[Java] 利用Spring Data JPA 接駁 SQL server

C.H. Ling — Fri, 06 Jul 2018 06:50:45 +0000

Java Persistence API (JPA) 是 Java 內建的library 用作ORM 互動. 著名的衍生品之一就是Hibernate, 而Spring 亦基於Hibernate, 在Spring Boot 中建立了Spring Data JPA. 在示範中, 會以Gradle project 接駁SQL server 及存取資料.示範步驟如下.

加入Spring 及SQL server library.
修改Build.gradle 如下:

/*
 * This build file was generated by the Gradle 'init' task.
 *
 * This generated file contains a sample Java Library project to get you started.
 * For more details take a look at the Java Libraries chapter in the Gradle
 * user guide available at https://docs.gradle.org/4.3/userguide/java_library_plugin.html
 */
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:2.0.3.RELEASE")
    }
}

// Apply the java-library plugin to add support for Java Library
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'

// In this section you declare where to find the dependencies of your project
repositories {
    mavenCentral()
    maven { url "https://repository.jboss.org/nexus/content/repositories/releases" }
}https://www.chunho-ling.com/wp-admin/post-new.php#
sourceCompatibility = 1.8
targetCompatibility = 1.8

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.springframework.boot:spring-boot-starter-data-jpa")
    compile("com.fasterxml.jackson.core:jackson-databind")
    compile("com.microsoft.sqlserver:mssql-jdbc")
    testCompile("junit:junit")
}

取得library.
於Project 中按右鍵並選取 Gradle > Refresh Gradle Project.
設定資料庫.
於/src/main/resources/ 建立檔案 application.property, 並修改內容如下.
```
spring.jpa.hibernate.ddl-auto=none
spring.datasource.url=jdbc:sqlserver://[[SQL server name]]:[[Port no.]];databaseName=[[Database name]]
spring.datasource.username=[[User name here]]
spring.datasource.password=[[Password]]
```
spring.jpa.hibernate.ddl-auto 是spring JPA 對database 的結構設定, 類似於Entity Framework 的code first. 而spring.datasource.url 指的是connection string. 會因應不同database 而有所分別.

建立Model.
於/src/main/java/models 中建立檔案Staff.java, 並輸入以下內容.

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.JoinColumn;
import javax.persistence.OneToOne;

import com.fasterxml.jackson.annotation.JsonIdentityInfo;
import com.fasterxml.jackson.annotation.ObjectIdGenerators;

@Entity
public class Staff {
	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	private int StaffID;
	private String Name;
	
	public int getStaffID() {
		return StaffID;
	}

	public void setStaffID(int staffID) {
		StaffID = staffID;
	}

	public String getName() {
		return Name;
	}

	public void setName(String name) {
		Name = name;
	}
}

@Entity 是JPA annotation, 說明這是database entity, 若database table 與class name 不同, 則須要用@Table(name=”[[Table Name]]”) 說明. @Id 說明這是table 內的primary key, 而@GenerateValue 則說明如果建立key. 在這裡利用AUTO, 即是使用database 預設.

建立Repository.
於/src/main/java/repositories 中建立檔案StaffRepository.java, 並輸入以下內容.

import java.util.List;

import org.matilda.backend.roi.models.Staff;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.CrudRepository;
import org.springframework.data.repository.query.Param;

public interface StaffRepository extends CrudRepository {
	@Query(value="SELECT s FROM Staff s WHERE StaffID=:staffID")
	public Staff findByStaffID(@Param("staffID") int staffID);
	@Query(value="SELECT s FROM Staff s WHERE Name=:name", nativeQuery=true)
	public List findByName(@Param("name") String name);
}

@Query 是建立Hibernate Query Language (HQL), 一種類似SQL 的query. 而@Param 則設定@Query 內的參數.

建立Controller.
於/src/main/java/controllers 中建立檔案Staff.java, 並輸入以下內容.

import java.util.List;

import org.matilda.backend.roi.Repositories.StaffRepository;
import org.matilda.backend.roi.models.Staff;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
@RequestMapping(path="/main")
public class MainController {
	private static final Logger log = LoggerFactory.getLogger(MainController.class);
	@Autowired
	private StaffRepository staffRepository;
	
	@GetMapping(path="/findByStaffID")
	public @ResponseBody Staff findByStaffID(@RequestParam int staffID) {
		log.info("findByStaffID() started. staffID="+staffID);
		Staff result=staffRepository.findByStaffID(staffID);
		return result;
	}
	
	@GetMapping(path="/findByStaffName")
	public @ResponseBody List findByStaffName(@RequestParam String name) {
		log.info("findByStaffName() started. name="+name);
		List result=staffRepository.findByName(name);
		return result;
	}
}

@Controller 會在spring 中認定為controller class; @RequestMapping 設定了Controller 的root path; @Autowired 則是Spring boot 中一個DI , 將之前要設定好的bean 或compoent 叫用. @GetMapping 是@RequestMapping(method=GET) 的縮寫.

建立Spring Application.
於/src/main/java/中建立檔案Application.java, 並輸入以下內容.

@SpringBootApplication
public class Application {
	private static final Logger log = LoggerFactory.getLogger(Application.class);
	public static void main(String[] args) {
		log.info("Backend service ROI Starting...");
		SpringApplication.run(Application.class);
		log.info("Backend service ROI Startup completed.");
	}
}

@SpringBootApplication 會執行三個annotation 的動作:

@EnableAutoConfiguration: 執行 Spring Boot自動設定;
@ComponentScan: 搜尋所有 @Component 及inherit 的class;
@Configuration: 設定額外加入的 bean;

執行程式.
於Project 中按右鍵並選取 Debug As > Spring Boot App.
測試程式.
於瀏覽器中輸入 http://localhost:8080/main/findByStaffID?staffID=47 , 若有JSON 回傳, 代表結果成功.

[Java] 利用Gradle 取得SQL server driver

C.H. Ling — Fri, 06 Jul 2018 05:44:26 +0000

用Java 常用的repository 通常是Maven 或 Gradle. Microsoft 除了將部份project 開源外, 還有利用Maven 做它其中一個software repository. 然而, 利用Gradle 亦有方法在Mavel 中取得有關的library.

Build.gradle

repositories {
    mavenCentral()
}
sourceCompatibility = 1.8
targetCompatibility = 1.8

dependencies {
    compile("com.microsoft.sqlserver:mssql-jdbc")
}

利用mavenCentral() 可以將repository 指向Maven. 而sourceCompatibility及targetCompatibility則設定取得版本的對應Java 的版本. dependencies 則設定要叫用的library. 語法如下:

[[group name]]:[[package name]]: [[optional: version no]]

[Java] 利用Eclipse將 gradle project 打包做war檔

C.H. Ling — Wed, 20 Jun 2018 08:53:09 +0000

預設下Gradle 會將project 以jar 的形式打包, 然而, 若要打包的是web application的話, jar 並不能於Tomcat 或者JBoss 中發佈, 故war 檔格式雖然舊但仍有其價值. 所以須要透過plugin 去進行. (所以個人討厭寫Java 其中一個原因就是太多相類似的plugin)

加入library 及dependency.
因為spring-boot-starter-web 本身已經有tomcat container, 所以可以將之排除.
在build.gradle 中, 加入以下粗字表示的句子:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:2.0.3.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
apply plugin: 'war'

jar {
    baseName = 'gs-handling-form-submission'
    version =  '0.1.0'
}

repositories {
    mavenCentral()
}

sourceCompatibility = 1.8
targetCompatibility = 1.8

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    providedRuntime("org.springframework.boot:spring-boot-starter-tomcat")
    compile("org.springframework.boot:spring-boot-starter-thymeleaf")
    testCompile("junit:junit")
}令

在application.java 中, 加入以下method:

protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
	      return application.sources(Application.class);
	}

更新Gradle project 設定, 以獲得library 及Gradle command.
於project 中按右鍵 > Gradle > Refresh Gradle Project.
建立war 檔.
於Gradle Task 視窗內, 右鍵選取build > Run Gradle Task.
檢查結果.
執行成功的話, war 檔會放在%ProjectDir%\build\libs\ 內.