Benchmarking password hashing algorithms using CircleCI build matrix

When you create an account online, your password isn’t just saved somewhere — it goes through password hashing. This step is crucial for protecting data, but there’s more to it than just scrambling characters. Hashing speed plays a big role: if it’s too fast, attackers can guess passwords quickly; if it’s too slow, it can make login processes frustrating for users.

Password hashing algorithms like BCrypt, SCrypt, and Argon2 are designed to require significant computational effort to resist attacks, even on modern fast servers. Benchmarking different configurations helps us balance security needs with practical performance without spending unnecessary time. For the benchmarking process, we selected Java due to its mature libraries, strong performance profiling tools, and wide adoption in backend development.

In this article, you’ll learn why hashing performance matters and how to automate your testing process using CircleCI’s build matrix feature.

Prerequisites

To get the most from this tutorial, you will need:

• A CircleCI account
• A GitHub account
• Intellij IDEA installed
• Familiarity with using Gradle
• Git installed

Benchmarking methodology

Best practices for benchmarking methodology focus on strong and trusted algorithms. Argon2, BCrypt, and SCrypt are among the top choices for password hashing. They are specifically designed to:

1. Slow down brute-force attacks (where attackers try millions of passwords),
2. Make password cracking expensive by using a lot of computing power — and for some, even a lot of memory.

The benchmarking in this tutorial focuses on these three widely-adopted password hashing algorithms:

• Argon2
• BCrypt
• SCrypt

The next sections describe each of these algorithms and why they are trusted.

Argon2

Argon2 is a modern password hashing algorithm designed to be secure, flexible, and resistant to brute-force attacks that defined in RFC 9106 and is entitled “Memory-Hard Function for Password Hashing and Proof-of-Work Applications”. and is used for password hashing, key derivation and proof-of-work.

The main variant is Argon2id which is optimized for x86 architectures. It has two sub-variants: Argon2d and Argon2i. Overall, Argon2d provides data-dependent memory access method and is useful for cryptocurrencies and proof-of-work applications. It also does not have any threats for side-channel timing attacks. Alternatively, Argon2i uses data-independent memory access and can be used for password hashing and password-based key derivation (KDF).

Argon2 format example:

$argon2id$v=19$m=65536,t=3,p=4$c29tZXNhbHQ$eS5Dj3JD7U3AGAvc5UHE4f+lgjxuVqHkRCk70Yg3GoA
\________/\___/\________________/\___________/\________________________________________/
      |      |          |             |                       |
      |      |          |             |                       +--- Base64-encoded hash output
      |      |          |             +--------------------------- Base64-encoded salt
      |      |          +----------------------------------------- Parameters (memory, iterations, parallelism)
      |      +---------------------------------------------------- Version number
      +----------------------------------------------------------- Algorithm type (argon2i, argon2d, argon2id)

This example is made up of the following parts:

• $argon2id$: Algorithm type (argon2i, argon2d, or argon2id)
• v=19: Version (19 = Argon2 v1.3, defined in RFC 9106)
• m=65536,t=3,p=4: Parameters
  • m: Memory cost in KiB (65536 KiB = 64 MB)
  • t: Number of iterations (time cost)
  • p: Number of parallel threads (parallelism)
• c29tZXNhbHQ: Base64-encoded salt
• eS5Dj3JD7U3AGAvc5UHE4f+lgjxuVqHkRCk70Yg3GoA: Base64-encoded derived key (hash)

Argon2 is widely recommended for general use for these reasons:

• Winner of the 2015 Password Hashing Competition (PHC) — basically the Olympics for password hashing.
• It’s highly customizable, letting you tune memory usage, computing time, and parallelism.
• It’s built to stay strong even against the most recent and future hardware.

In short: Argon2 is state-of-the-art, flexible, and forward-looking — perfect for modern applications.

BCrypt

BCrypt is a password hashing algorithm designed to securely protect passwords. Developed in 1999 by Niels Provos and David Mazières, BCrypt is based on the Blowfish cipher and features an important characteristic called an adaptive cost factor.

A example of the BCrypt format:

$2a$12$R9h/cIPz0gi.URNNX3kh2OPST9/PgBkqquzi.Ss7KIUgO2t0jWMUW
\__/\/ \____________________/\_____________________________/
 |   |          |                         |
 |   |          |                         +-- Hash output
 |   |          +---------------------------- Salt
 |   +--------------------------------------- Cost (log2 number of rounds)
 +------------------------------------------- Algorithm version

This example is made up of the following parts:

• $2a$: Algorithm identifier (2a, 2b, etc.)
• 12: Cost factor (log2 of number of hashing rounds)
• R9h/cIPz0gi.URNNX3kh2O: 22-character base64-encoded salt
• PST9/PgBkqquzi.Ss7KIUgO2t0jWMUW: 31-character base64-encoded hash

Here are some important notes about BCrypt:

• Used since 1999, it’s very well-tested and trusted in the security world.
• It intentionally slows down the password hashing process,taking milliseconds per hash.
• It allows you to adjust a “cost factor”, making the hashing process slower as computers get faster.
• It’s strong against basic CPU-based attacks but is less resistant to modern attacks using GPUs or specialized hardware.

In short: BCrypt is proven and reliable, but slightly outdated against modern high-speed cracking hardware.

SCrypt

SCrypt is a password hashing algorithm specifically designed to resist brute-force attacks by requiring significant computational resources. Its design makes it expensive to attack using specialized hardware like GPUs, FPGAs, or ASICs. Here are some key characteristics of SCrypt that contribute to its effectiveness in securing passwords:

• Designed to be both CPU and memory intensive.
• Attackers can’t just use a fast GPU — they also need a lot of memory for every password they try, making attacks much more expensive.
• It lets you adjust settings like CPU effort, memory use, and parallelism.
• It’s a strong defense against large-scale, automated password attacks.

An example SCrypt format:

$SCrypt$16384$8$1$MzY2bmZtZWtwb2F3ZzNsaQ==$Z8sYdkNRwhujgKe37KcfVswD8T/3xv6Qyktp9STXlj0=
\______/\____/\_/\_/\_____________________/\________________________________________/
    |      |   |  |            |                         |
    |      |   |  |            |                         +--- Base64-encoded hash output
    |      |   |  |            +----------------------------- Base64-encoded salt
    |      |   |  +------------------------------------------ Parallelization factor (p)
    |      |   +--------------------------------------------- Block size factor (r)
    |      +------------------------------------------------- CPU/Memory cost factor (N)
    +-------------------------------------------------------- Algorithm identifier

This example is made up of the following parts:

• $SCrypt$: Algorithm identifier
• 16384: Cost parameter (N) — CPU/memory cost
• 8: Block size factor (r)
• 1: Parallelization factor (p)
• MzY2bmZtZWtwb2F3ZzNsaQ==: Base64-encoded salt
• Z8sYdkNRwhujgKe37KcfVswD8T/3xv6Qyktp9STXlj0=: Base64-encoded derived key (hash)

In short: SCrypt is great for making attacks costly, especially when attackers use powerful hardware.

Operating System (OS) variants

In the benchmarking setup for this tutorial, Operating System (OS) variants are used to evaluate password hashing performance. The hashing performance can vary across different operating systems due to differences in how the system manages CPU, memory, threading, and low-level system calls.

The operating systems tested are:

• Linux
  The most common OS in server environments. Linux tends to provide highly optimized performance for multi-threaded workloads, making it a critical platform for testing.
• Windows
  Often used in enterprise environments and development setups. Windows has different scheduling and memory handling behavior compared to Linux, which might impact performance, especially for memory-hard algorithms like SCrypt and Argon2.
• macOS
  Popular among developers. macOS uses a UNIX-like core (Darwin) but with different optimizations compared to Linux, making it important to observe any performance differences.

Why OS matters:
Password hashing heavily depends on CPU and memory operations, and even subtle differences in how an OS manages these resources can cause significant variations in hashing speed.

Why this methodology makes sense

By testing across different OSes and Java versions, the benchmarking provides a comprehensive view of real-world performance.
It ensures that the results are relevant whether the password hashing library is used in a cloud server (Linux), an enterprise workstation (Windows), or a developer laptop (macOS).

Implementing the benchmarking

Implementation consists of these six steps:

1. Create a new project
2. Set up JMH
3. Set up hashing libraries in Spring
4. Set up jBCrypt for BCrypt
5. Set up SCrypt library
6. Set up Bouncy Castle for Argon2

Create new project

Now you can set up a new Java project to benchmark password hashing algorithms. Here’s what you will configure:

• Project name: PasswordHashBenchmark (you’ll test performance of hashing methods like BCrypt or Argon2).
• Location: Stored in your /Java projects folder (adjust yours as needed).
• Git: You’ll version-control it—check “Create Git repository” to track changes.
• Build tool: Gradle (Groovy DSL) for dependency management.
• JDK: Amazon Corretto 17 (Java 17 LTS for stability).

Click Create, and you’re ready to add dependencies (like Spring Security for hashing) and write benchmarks. The project structure will be like the tree shown here:

PasswordHashBenchmark/
├── gradle/
│   └── wrapper/
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── src/
│   └── main/
│       └── java/
├── src/
│   └── test/
│       └── java/
├── build.gradle
├── gradle.properties
├── gradlew
├── gradlew.bat
└── settings.gradle

Set up JMH

JMH (Java Microbenchmark Harness) is a Java library developed by the creators of the JDK for building, running, and analyzing benchmarks of Java code with high precision. It’s specifically designed for microbenchmarking—measuring the performance of small units of code while avoiding common pitfalls like JVM optimizations and warm-up issues.

To begin set up a Gradle project with the libraries you need to benchmark password hashing algorithms. Open the build.gradle file and add the configurations as shown. Add these plugins for JMH:

plugins {
  // other plugin
  id 'me.champeau.jmh' version '0.7.2'
}

Add these dependencies:

dependencies {
  // ...
  implementation 'org.openjdk.jmh:jmh-core:1.37'
  compileOnly 'org.openjdk.jmh:jmh-generator-annprocess:1.37'
  annotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.37'
}

Set up hashing libraries in Spring

For Spring users, this is quite simple; Spring Security provides built-in support for the following password hashing algorithms:

• BCrypt: BCryptPasswordEncoder
• SCrypt: SCryptPasswordEncoder
• Argon2: Argon2PasswordEncoder

To use these algorithms in your Spring project, add the spring-security-crypto dependency in your build.gradle:

dependencies {
  // ...
  implementation 'org.springframework.security:spring-security-crypto:6.4.4'
}

Example:

BCryptPasswordEncoder encoder = new BCryptPasswordEncoder();
String hashedPassword = encoder.encode("myPassword");

SCryptPasswordEncoder encoder = new SCryptPasswordEncoder();
String hashedPassword = encoder.encode("myPassword");

Argon2PasswordEncoder encoder = new Argon2PasswordEncoder();
String hashedPassword = encoder.encode("myPassword");

All three algorithms (BCrypt, SCrypt, Argon2) are included in spring-security-crypto and can be easily configured via the above classes.

Note: Not everyone uses Spring, so we’ll explore at another way to work with the three hashing algorithms (BCrypt, SCrypt, and Argon2) in the following discussion.</>

jBCrypt for BCrypt

jBCrypt is a Java library that provides an implementation of the BCrypt password hashing function. It is designed specifically for Java applications to securely hash and verify passwords.

To install jBCrypt, add this dependency:

  dependencies {
    // ...
    implementation 'org.mindrot:jbcrypt:0.4'
  }

Example:

  String password = "mySuperSecretPassword";
  String hashedPassword = BCrypt.hashpw(password, BCrypt.gensalt(12));

jBCrypt makes it easy to use BCrypt hashing in Java by providing simple methods for generating salted hashes (hashpw) and verifying passwords (checkpw).

Set up SCrypt library

One of the easiest SCrypt libraries to find is wg/scrypt, which is also used as the upstream source for the Fedora package: fedoraproject.org/java-scrypt.

wg/scrypt is a lightweight Java library that provides an implementation of scrypt, a password hashing algorithm designed to be highly resistant to brute-force attacks, especially those using specialized hardware like ASICs.

dependencies {
  // ...
  implementation 'com.lambdaworks:scrypt:1.4.0' 
}

Example:

byte[] salt = new byte[16];  // random salt
String password = "mySecurePassword";

// Parameter scrypt
int N = 16384, r = 8, p = 1, dkLen = 32;

// Hash password using scrypt
byte[] hashedPassword = SCrypt.scrypt(password.getBytes(), salt, N, r, p, dkLen);

The library focuses on being simple, minimalistic, and efficient, with no external framework dependencies. It is widely used in the Java community and even serves as the upstream source for several Linux distributions, such as Fedora. It offers straightforward functions for generating and verifying password hashes with secure scrypt parameters.

Set up Bouncy Castle for Argon2

To implement Argon2 in Java, one great tool you can use is the Bouncy Castle library. You can find it here: Bouncy Castle Java.

Bouncy Castle is a well-known, open-source library that provides a huge range of cryptographic algorithms — from classic ones like RSA and AES to modern ones like Argon2 for password hashing. It’s trusted, actively maintained, and widely used in security-critical applications.

By adding Bouncy Castle to your Gradle dependencies, you can directly start coding password hashing and encryption features without worrying about low-level cryptographic implementation details.

dependencies {
  // ...
  implementation 'org.bouncycastle:bcpkix-jdk18on:1.80'
}

For this project, you’ll use the bcpkix-jdk18on package from Bouncy Castle to access advanced cryptographic functions, including Argon2.

Note: Make sure you’re using the latest version. Right now, it’s 1.80 — because earlier versions, like 1.76, had a serious security vulnerability.

If you’re using IntelliJ IDEA, you’ll probably get a warning about it.

However, it won’t prevent you from pushing to the repository.

It’s safer to also run a vulnerability scan in CircleCI this way.

Creating a unit test

In this section, you will create a unit test class called PasswordValidationTest in the src/test/java directory.

Here is the PasswordValidationTest class complete with its test methods:

import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Arrays;

import org.bouncycastle.crypto.generators.Argon2BytesGenerator;
import org.bouncycastle.crypto.params.Argon2Parameters;
import org.junit.jupiter.api.Test;
import org.mindrot.jbcrypt.BCrypt;
import org.springframework.security.crypto.argon2.Argon2PasswordEncoder;

import com.lambdaworks.crypto.SCrypt;

public class PasswordValidationTest {

    String password = "Bismillah!@#123!";

    /**
     * This test demonstrates the use of BCrypt for password hashing and verification.
     * It includes:
     * 1. Hashing a password with a cost factor.
     * 2. Verifying the hashed password.
     * 3. Ensuring that hashing produces different results each time due to salting.
     * 4. Checking that an incorrect password fails verification.
     */
    @Test
    public void testBCryptHashAndVerify() {
        int cost = 12; // log rounds (e.g., 10, 12, 14)

        // 1. Hash the password
        String hashedPassword = BCrypt.hashpw(password, BCrypt.gensalt(cost));

        // 2. Hash should verify successfully
        assertTrue(BCrypt.checkpw(password, hashedPassword));

        // 3. Different hash on each call (due to salt)
        String secondHash = BCrypt.hashpw(password, BCrypt.gensalt(cost));
        assertNotEquals(hashedPassword, secondHash);

        // 4. Wrong password fails to verify
        assertFalse(BCrypt.checkpw("WrongPassword", hashedPassword));
    }

    /**
     * This test demonstrates the use of SCrypt for password hashing and verification.
     * It includes:
     * 1. Hashing a password with SCrypt parameters.
     * 2. Verifying the hashed password.
     * 3. Ensuring that hashing produces different results each time due to salting.
     * 4. Checking that an incorrect password fails verification.
     */
    @Test
    public void testScryptHashAndVerify() throws GeneralSecurityException {
        // Scrypt parameters
        int N = 16384;
        int r = 8;
        int p = 1;
        int dkLen = 64; // output length in bytes

        // Generate random salt
        byte[] salt = new byte[16];
        new SecureRandom().nextBytes(salt);

        // Hash the password
        byte[] hashedPassword = SCrypt.scrypt(password.getBytes(), salt, N, r, p, dkLen);

        // -- VALIDATION --

        // 1. Correct password should match
        byte[] attempt = SCrypt.scrypt(password.getBytes(), salt, N, r, p, dkLen);
        assertArrayEquals(hashedPassword, attempt);

        // 2. Different password should fail
        byte[] wrongAttempt = SCrypt.scrypt("WrongPassword".getBytes(), salt, N, r, p, dkLen);
        assertFalse(Arrays.equals(hashedPassword, wrongAttempt));

        // 3. Salt ensures different hash
        byte[] newSalt = new byte[16];
        new SecureRandom().nextBytes(newSalt);
        byte[] secondHash = SCrypt.scrypt(password.getBytes(), newSalt, N, r, p, dkLen);
        assertFalse(Arrays.equals(hashedPassword, secondHash));
    }

    /**
     * This test demonstrates the use of Argon2 for password hashing and verification.
     * It includes:
     * 1. Hashing a password with Argon2 parameters.
     * 2. Verifying the hashed password.
     * 3. Ensuring that hashing produces different results each time due to salting.
     * 4. Checking that an incorrect password fails verification.
     */
    @Test
    public void givenRawPassword_whenEncodedWithArgon2_thenMatchesEncodedPassword() {
        Argon2PasswordEncoder arg2SpringSecurity = new Argon2PasswordEncoder(16, 32, 1, 60000, 10);
        String springBouncyHash = arg2SpringSecurity.encode(password);

        assertTrue(arg2SpringSecurity.matches(password, springBouncyHash));
    }

    /**
     * Generates a random 16-byte salt using SecureRandom.
     *
     * @return A byte array representing the salt.
     */
    private byte[] generateSalt16Byte() {
        SecureRandom secureRandom = new SecureRandom();
        byte[] salt = new byte[16];
        secureRandom.nextBytes(salt);

        return salt;
    }

    /**
     * This test demonstrates the use of Argon2 for password hashing and verification.
     * It includes:
     * 1. Hashing a password with Argon2 parameters.
     * 2. Verifying the hashed password.
     * 3. Ensuring that hashing produces different results each time due to salting.
     * 4. Checking that an incorrect password fails verification.
     */
    @Test
    public void givenRawPasswordAndSalt_whenArgon2AlgorithmIsUsed_thenHashIsCorrect() {
        byte[] salt = generateSalt16Byte();

        int iterations = 2;
        int memLimit = 66536;
        int hashLength = 32;
        int parallelism = 1;

        Argon2Parameters.Builder builder = new Argon2Parameters.Builder(Argon2Parameters.ARGON2_id)
            .withVersion(Argon2Parameters.ARGON2_VERSION_13)
            .withIterations(iterations)
            .withMemoryAsKB(memLimit)
            .withParallelism(parallelism)
            .withSalt(salt);

        Argon2BytesGenerator generate = new Argon2BytesGenerator();
        generate.init(builder.build());
        byte[] result = new byte[hashLength];
        generate.generateBytes(password.getBytes(StandardCharsets.UTF_8), result, 0, result.length);

        Argon2BytesGenerator verifier = new Argon2BytesGenerator();
        verifier.init(builder.build());
        byte[] testHash = new byte[hashLength];
        verifier.generateBytes(password.getBytes(StandardCharsets.UTF_8), testHash, 0, testHash.length);

        assertArrayEquals(result, testHash);
    }
}

Unit test details

PasswordValidationTest checks that password hashing and verification work securely using BCrypt, SCrypt, and Argon2.

Each test follows the same flow:

1. Hash - the password.
2. Verify - the correct password matches the hash.
3. Ensure - different hashes are created (because of random salt).
4. Check - that wrong passwords do not match.

Here’s a quick breakdown of each test:

• testBCryptHashAndVerify(): Uses BCrypt to hash, verify correct/wrong passwords, and confirm salting.
• testSCryptHashAndVerify(): Uses SCrypt, verifies correct hash with same salt, and checks different salt gives different results.
• givenRawPassword_whenEncodedWithArgon2_thenMatchesEncodedPassword(): Uses Spring Security Argon2 encoder to hash and verify passwords.
• givenRawPasswordAndSalt_whenArgon2AlgorithmIsUsed_thenHashIsCorrect(): Manually uses Argon2 (via BouncyCastle) with full control over hashing parameters.

Note: generateSalt16Byte() is a helper to generate a random 16-byte salt. This test must pass — it’s an essential safety check before merging anything to main.

Writing benchmarking code

Next, create a class for benchmarking, called PasswordHashingBenchmark. Make sure to put it inside a package, such as id.web.hangga. JMH can run only benchmarks that are located within a package.

package id.web.hangga;

import org.bouncycastle.crypto.generators.Argon2BytesGenerator;
import org.bouncycastle.crypto.params.Argon2Parameters;
import org.mindrot.jbcrypt.BCrypt;
import org.openjdk.jmh.annotations.*;
import org.springframework.security.crypto.argon2.Argon2PasswordEncoder;
import com.lambdaworks.crypto.SCrypt;

import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.concurrent.TimeUnit;

public class PasswordHashingBenchmark {

    private static final String password = "Bismillah!@#123!";

    private byte[] generateSalt16Byte() {
        SecureRandom secureRandom = new SecureRandom();
        byte[] salt = new byte[16];
        secureRandom.nextBytes(salt);
        return salt;
    }

    // BCrypt Benchmark
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    @Fork(1)
    public void benchmarkBCrypt() {
        int cost = 12;
        String hashedPassword = BCrypt.hashpw(password, BCrypt.gensalt(cost));
        BCrypt.checkpw(password, hashedPassword);
    }

    // SCrypt Benchmark
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    @Fork(1)
    public void benchmarkScrypt() throws Exception {
        int N = 16384;
        int r = 8;
        int p = 1;
        int dkLen = 64;
        byte[] salt = generateSalt16Byte();
        byte[] hashedPassword = SCrypt.scrypt(password.getBytes(), salt, N, r, p, dkLen);
        byte[] attempt = SCrypt.scrypt(password.getBytes(), salt, N, r, p, dkLen);
    }

    // Argon2 Benchmark using Spring Security
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    @Fork(1)
    public void benchmarkArgon2SpringSecurity() {
        Argon2PasswordEncoder argon2 = new Argon2PasswordEncoder(16, 32, 1, 60000, 10);
        String hash = argon2.encode(password);
        argon2.matches(password, hash);
    }

    // Argon2 Benchmark using BouncyCastle
    @Benchmark
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    @Fork(1)
    public void benchmarkArgon2BouncyCastle() {
        byte[] salt = generateSalt16Byte();

        int iterations = 2;
        int memLimit = 66536;
        int hashLength = 32;
        int parallelism = 1;

        Argon2Parameters.Builder builder = new Argon2Parameters.Builder(Argon2Parameters.ARGON2_id)
            .withVersion(Argon2Parameters.ARGON2_VERSION_13)
            .withIterations(iterations)
            .withMemoryAsKB(memLimit)
            .withParallelism(parallelism)
            .withSalt(salt);

        Argon2BytesGenerator generator = new Argon2BytesGenerator();
        generator.init(builder.build());
        byte[] result = new byte[hashLength];
        generator.generateBytes(password.getBytes(StandardCharsets.UTF_8), result, 0, result.length);
    }
}

Understanding the benchmark strategy

Take a quick look at the flow inside the PasswordHashingBenchmark class.

Start by setting up a fixed password string, and a utility function to generate random 16-byte salts — something needed for some of the algorithms later.

The class itself contains four benchmark methods, each focusing on a different password hashing strategy:

• BCrypt hashes the password using a cost factor of 12, and immediately verify it. BCrypt handles its own salt internally, so you don’t have to manage it manually.
• SCrypt: This generates a fresh salt for each operation and applies the SCrypt function with parameters like N, r, p, and a 64-byte output. It simulates a verification by rehashing the same password with the same salt.
• Argon2 via Spring Security: This uses Spring’s Argon2PasswordEncoder, setting parameters like memory, iterations, and parallelism. Hashing and matching are straightforward because Spring takes care of salt generation and parameter encoding behind the scenes.
• Argon2 via BouncyCastle: This uses BouncyCastle’s Argon2BytesGenerator lower-level control. It manually configures all the hashing parameters, feeds in the salt, and generates the hash manually. This is closer to how Argon2 works under the hood but requires a few extra lines compared to the Spring version.

Each method is annotated properly for JMH benchmarking. This project measures average execution time (Mode.AverageTime) in milliseconds, and only one fork to keep the test simple.

In short: this class benchmarks the full cycle of hashing and verifying passwords across different algorithms and implementations — giving you a practical comparison of performance under real conditions.

Now you’re ready to run the benchmark. But before you push everything to GitHub and integrate it with CircleCI, test it locally first.

Run on your local machine

To test the benchmarks locally, it’s super simple. If you’re using Gradle with the JMH plugin set up, just open your terminal and run:

./gradlew jmh

Gradle will build your project, compile the benchmarks, and then execute them.
Your outputs will be something like this:

Here’s a quick breakdown:

• Mode avgt means we’re measuring average execution time.
• Cnt is how many times the benchmark was run (e.g., 25 iterations).
• Score shows the average time it took (in milliseconds) for one operation.
• Error gives the confidence margin (the ± value).
• Units are milliseconds per operation.

Lower scores mean faster hashing+verification. But faster doesn’t always mean better — especially when it comes to password security.

Once you’re happy with the local results, it’s a good time to push everything to GitHub and start setting up your CircleCI workflow.

Integration into CircleCI with matrix builds

To run benchmarking on CircleCI, start by pushing your local benchmarking code to a new Github repository:

git init
git commit -m "first commit"
git remote add origin https://github.com/YOUR-USER-NAME/PasswordHashBenchmark.git
git push -u origin main

Then, integrate it with CircleCI. During the setup, CircleCI will automatically create a new branch called circleci-project-setup, which includes an initial configuration file located at .circleci/config.yml.

The circleci-project-setup branch is generated by CircleCI to help initialize the pipeline configuration without directly modifying your main branch. This approach gives you the opportunity to review and customize the configuration before applying it to your primary development workflow.

When you open .circleci/config.yml, the default contents are:

# This config was automatically generated from your source code
# Stacks detected: deps:java:.,tool:gradle:
version: 2.1
jobs:
  test-java:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - run:
          name: Calculate cache key
          command: |-
            find . -name 'pom.xml' -o -name 'gradlew*' -o -name '*.gradle*' | \
                    sort | xargs cat > /tmp/CIRCLECI_CACHE_KEY
      - restore_cache:
          key: cache-{{ checksum "/tmp/CIRCLECI_CACHE_KEY" }}
      - run:
          command: ./gradlew check
      - store_test_results:
          path: build/test-results
      - save_cache:
          key: cache-{{ checksum "/tmp/CIRCLECI_CACHE_KEY" }}
          paths:
            - ~/.gradle/caches
      - store_artifacts:
          path: build/reports

  deploy:
    # This is an example deploy job, not actually used by the workflow
    docker:
      - image: cimg/base:stable
    steps:
      # Replace this with steps to deploy to users
      - run:
          name: deploy
          command: "#e.g. ./deploy.sh"

workflows:
  build-and-test:
    jobs:
      - test-java
    # - deploy:
    #     requires:
    #       - test-java

This workflow has two jobs by default, but only one is active.

• test-java: Runs Java tests with Gradle in a Docker container (OpenJDK 17), uses caching, and stores test results and reports.
• deploy: Placeholder for deployment; currently not used (commented out).

You’ll customize this CircleCI setup to run unit tests (test-java) and performance benchmarks (benchmark-*) in parallel across three operating systems: Linux, macOS, and Windows, using different Java versions (JDK 17 and 21).

The test-java job checks that everything passes as expected, while the benchmark jobs use JMH to measure how fast or efficient the algorithms are,then store the results for review.

The benchmark-linux Job

Let’s define a reusable CircleCI job to benchmark Java password hashing speeds on Linux using JMH. We’ll make the Java version configurable so you can easily compare performance across versions. Here’s the benchmark-linux job:

benchmark-linux:
  parameters:
    java-version:
      type: string
  docker:
    - image: cimg/openjdk:<< parameters.java-version >>
  steps:
    - checkout
    - run:
        name: Run JMH Benchmark (Linux, Java << parameters.java-version >>)
        command: ./gradlew jmh
    - store_artifacts:
        path: build/reports

Here’s a breakdown:

• Parameters Section: defines a parameter java-version that allows you to specify the Java version for the benchmark.
• Docker Image: It uses the cimg/openjdk Docker image with the specified Java version (<< parameters.java-version >>), ensuring the correct Java environment is available.
• Steps:
  • checkout: Pulls the project code to the build environment.
  • run: Executes the JMH benchmark using Gradle (./gradlew jmh), running performance tests on the specified Java version.
  • store_artifacts: Saves the benchmark results from build/reports so you can access them later.

This config allows you to benchmark different Java versions in parallel by passing different java-version parameters in CircleCI’s matrix builds.

The benchmark-macos job

Unlike Linux, macOS jobs in CircleCI don’t use Docker but instead use the native macOS environment provided by:

macos:
  xcode: "14.2.0"

Since there’s no pre-installed OpenJDK by default, you’ll use Homebrew to install the desired version dynamically:

brew install openjdk@<< parameters.java-version >>

And then, manually set the JAVA_HOME environment variable.

Finally, just like before, run:

./gradlew jmh

So the complete version is like this:

benchmark-macos:
  parameters:
    java-version:
      type: string
  macos:
    xcode: "14.2.0"
  steps:
    - checkout
    - run:
        name: Install Java << parameters.java-version >>
        command: |
          brew install openjdk@<< parameters.java-version >>
          echo 'export JAVA_HOME=/opt/homebrew/opt/openjdk@<< parameters.java-version >>' >> $BASH_ENV
    - run:
        name: Run JMH Benchmark (macOS, Java << parameters.java-version >>)
        command: ./gradlew jmh
    - store_artifacts:
        path: build/reports

The benchmark-windows job

Now add Windows support to your benchmarking setup. Here’s the benchmark-windows job configuration, allowing us to measure Java performance on a Windows runner:

benchmark-windows:
  parameters:
    java-version:
      type: string
  machine:
    resource_class: windows.medium
  steps:
    - checkout
    - run:
        name: Install Java << parameters.java-version >>
        command: |
          choco install openjdk<< parameters.java-version >>
          setx JAVA_HOME "C:\Program Files\OpenJDK\openjdk-<< parameters.java-version >>"
    - run:
        name: Run JMH Benchmark (Windows, Java << parameters.java-version >>)
        command: ./gradlew.bat jmh
    - store_artifacts:
        path: build/reports

This job uses CircleCI’s Windows environment:

machine:
  resource_class: windows.medium

Install Java using Chocolatey:

choco install openjdk<< parameters.java-version >>

Then set JAVA_HOME so Gradle knows which Java to use.

Since you’re on Windows, invoke the benchmark using the .bat variant:

./gradlew.bat jmh

Matrix expansion in workflows

Lastly, tie everything together using a workflow that runs benchmarks across platforms and Java versions in parallel. This helps you compare performance consistently. Here’s the setup:

workflows:
  build-and-benchmark:
    jobs:
      - test-java
      - benchmark-linux:
          name: Benchmark - Linux (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17.0", "21.0"]
      - benchmark-macos:
          name: Benchmark - macOS (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17", "21"]
      - benchmark-windows:
          name: Benchmark - Windows (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17"]

Breakdown:

• test-java is a general testing job to ensure code correctness before benchmarking (you define it elsewhere).
• benchmark-linux (matrix) runs benchmarks on Linux for Java 17.0 and 21.0 using the cimg/openjdk Docker image.
• benchmark-macos (matrix) runs on macOS with Homebrew-installed Java 17 and 21.
• benchmark-windows (matrix) benchmarks on Windows, currently only with Java 17 (you can expand this later).

Each matrix job runs in parallel, so you get fast, cross-platform performance insights with minimal CI time.

Here is the full version:

version: 2.1

jobs:
  test-java:
    docker:
      - image: cimg/openjdk:17.0
    steps:
      - checkout
      - run:
          name: Calculate cache key
          command: |-
            find . -name 'pom.xml' -o -name 'gradlew*' -o -name '*.gradle*' | \
                    sort | xargs cat > /tmp/CIRCLECI_CACHE_KEY
      - restore_cache:
          key: cache-{{ checksum "/tmp/CIRCLECI_CACHE_KEY" }}
      - run:
          command: ./gradlew check
      - store_test_results:
          path: build/test-results
      - save_cache:
          key: cache-{{ checksum "/tmp/CIRCLECI_CACHE_KEY" }}
          paths:
            - ~/.gradle/caches
      - store_artifacts:
          path: build/reports

  benchmark-linux:
    parameters:
      java-version:
        type: string
    docker:
      - image: cimg/openjdk:<< parameters.java-version >>
    steps:
      - checkout
      - run:
          name: Run JMH Benchmark (Linux, Java << parameters.java-version >>)
          command: ./gradlew jmh
      - store_artifacts:
          path: build/reports

  benchmark-macos:
    parameters:
      java-version:
        type: string
    macos:
      xcode: "14.2.0"
    steps:
      - checkout
      - run:
          name: Install Java << parameters.java-version >>
          command: |
            brew install openjdk@<< parameters.java-version >>
            echo 'export JAVA_HOME=/opt/homebrew/opt/openjdk@<< parameters.java-version >>' >> $BASH_ENV
      - run:
          name: Run JMH Benchmark (macOS, Java << parameters.java-version >>)
          command: ./gradlew jmh
      - store_artifacts:
          path: build/reports

  benchmark-windows:
    parameters:
      java-version:
        type: string
    machine:
      resource_class: windows.medium
    steps:
      - checkout
      - run:
          name: Install Java << parameters.java-version >>
          command: |
            choco install openjdk<< parameters.java-version >>
            setx JAVA_HOME "C:\Program Files\OpenJDK\openjdk-<< parameters.java-version >>"
      - run:
          name: Run JMH Benchmark (Windows, Java << parameters.java-version >>)
          command: ./gradlew.bat jmh
      - store_artifacts:
          path: build/reports

workflows:
  build-and-benchmark:
    jobs:
      - test-java
      - benchmark-linux:
          name: Benchmark - Linux (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17.0", "21.0"]
      - benchmark-macos:
          name: Benchmark - macOS (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17", "21"]
      - benchmark-windows:
          name: Benchmark - Windows (JDK << matrix.java-version >>)
          matrix:
            parameters:
              java-version: ["17"]

You can merge the circleci-project-setup branch into your main branch and safely delete it afterward.

Now you can review the CircleCI dashboard.

A green test result signifies no issues; therefore, you can disregard it for now as your focus is on benchmarking. Check out the macOS - JDK 21 the benchmarks. Here’s what’s going on in the log section:

After reviewing all the benchmarks, you can now proceed with the analysis and conclusions.

Analyzing the benchmarking results

To summarize, the sections show the benchmark results obtained from testing across three operating systems — Windows, Linux, and macOS — using Java versions 17 and 21.

macOS - Java 21

Algorithm	Mode	Iterations	Avg Time (ms/op)	Error Margin
Argon2BouncyCastle	avgt	5	104.577	±1.780
Argon2SpringSecurity	avgt	5	848.533	±9.731
BCrypt	avgt	5	587.585	±0.317
SCrypt	avgt	5	118.461	±0.156

macOS - Java 17

Algorithm	Mode	Iterations	Avg Time (ms/op)	Error Margin
Argon2BouncyCastle	avgt	5	98.220	±0.506
Argon2SpringSecurity	avgt	5	835.586	±2.843
BCrypt	avgt	5	589.828	±1.050
SCrypt	avgt	5	118.837	±0.039

Windows - Java 17

Algorithm	Mode	Iterations	Avg Time (ms/op)	Error Margin
Argon2BouncyCastle	avgt	5	144.544	±2.971
Argon2SpringSecurity	avgt	5	1442.758	±18.959
BCrypt	avgt	5	589.552	±3.514
SCrypt	avgt	5	217.067	±1.205

Linux - Java 21

Algorithm	Mode	Iterations	Avg Time (ms/op)	Error Margin
Argon2BouncyCastle	avgt	5	190.983	±7.391
Argon2SpringSecurity	avgt	5	1282.785	±83.405
BCrypt	avgt	5	571.318	±20.378
SCrypt	avgt	5	61.288	±2.608

Linux - Java 17

Algorithm	Mode	Iterations	Avg Time (ms/op)	Error Margin
Argon2BouncyCastle	avgt	5	149.488	±11.036
Argon2SpringSecurity	avgt	5	1391.815	±744.002
BCrypt	avgt	5	576.833	±13.447
SCrypt	avgt	5	78.659	±5.396

Based on the benchmark results across three operating systems (macOS, Windows, and Linux) and two Java versions (17 and 21), here is a consolidated analysis and conclusion.

Key observations for each algorithm

The benchmarking uses four algorithms:

1. Argon2BouncyCastle
2. Argon2SpringSecurity
3. BCrypt
4. SCrypt

Argon2BouncyCastle

• Fastest on macOS (Java 17 and 21): ~98–104 ms/op
• Slower on Windows (144 ms) and Linux (149–190 ms)
• Most consistent performer across all platforms with moderate error margins
• Suggests strong cross-platform performance, especially on macOS

Argon2SpringSecurity

• Significantly slower than other algorithms across all environments
• Peaks at 1442 ms on Windows and 1391 ms on Linux Java 17
• Shows very high variance (especially ±744 ms on Linux Java 17)
• Indicates potential inefficiencies or implementation overhead in this version of Argon2

BCrypt

• Performance is relatively stable across all platforms (~571–590 ms/op)
• Consistent regardless of OS or Java version, but not fast
• Suitable for systems that prioritize battle-tested stability over speed

SCrypt

• Fastest on Linux Java 21: only 61 ms/op
• Slightly slower on macOS (~118 ms) and considerably slower on Windows (217 ms)
• Performance highly depends on OS — best on Linux, worst on Windows
• May be optimized for Linux JVM runtime or syscall handling

Overall analysis

• Best overall performer: Argon2BouncyCastle — consistently fast and stable across all platforms, making it an ideal cross-platform choice.
• Best speed on Linux: SCrypt on Java 21 Linux shows the best raw performance, useful for high-throughput services.
• Most stable algorithm: BCrypt — while not the fastest, it provides consistent behavior across environments, which is valuable for predictable system load.

Note: Avoid using Argon2SpringSecurity for performance-sensitive applications unless further tuning or optimizations are applied.

Here is the information formatted in a table:

Algorithm	Fastest Platform	Slowest Platform	Notes
Argon2BouncyCastle	macOS (Java 17)	Linux (Java 21)	Balanced, stable across OSes
Argon2SpringSecurity	macOS (Java 17/21)	Windows (Java 17)	Very slow and high variance
BCrypt	Linux (Java 21)	macOS (Java 17/21)	Stable, but always moderate
SCrypt	Linux (Java 21)	Windows (Java 17)	Impressive Linux performance

Conclusion

Password hashing requires a careful balance between security and usability. Through automated benchmarking of Argon2, BCrypt, and SCrypt across operating systems (Linux, Windows, macOS) and Java versions (17/21), key findings emerged, on which we’ve based the following recommendations:

• Prioritize Argon2 for applications requiring future-proof security and cross-platform compatibility.
• Leverage SCrypt in Linux-centric environments with Java 21 for speed.
• Opt for BCrypt if stability and simplicity are critical.

Methodology Impact

CircleCI’s matrix builds proved indispensable for automating cross-platform benchmarking, ensuring consistency, and accelerating testing cycles. By integrating OS and Java version variables, the methodology mirrors real-world conditions, enabling actionable insights for diverse deployment environments.

Ultimately, the choice of algorithm depends on operational context:

• Security-first applications benefit from Argon2’s memory-hard properties.
• High-throughput systems on Linux may favor SCrypt’s speed.
• Legacy or stability-focused systems can rely on BCrypt’s time-tested design.

Automation with CircleCI not only validates performance but also future-proofs security practices, ensuring adaptability as computational power evolves. The complete code for this project is available on GitHub.