Ish Bhana
Implementing MongoDB Client-Side Field-Level Encryption Using KMS and Spring Boot

Implementing MongoDB Client-Side Field-Level Encryption Using KMS and Spring Boot

Ish Bhana's photo
Ish Bhana
·

9 min read

In this article we'll look at how to implement Client Side Field-Level Encryption (CSFLE) in MongoDB with AWS KMS and Spring Boot. CSFLE allow us to encrypt and decrypt individual MongoDB document fields in our applications rather than in the database. Support for CSFLE was added in MongoDB 4.2.

CSFLE in MongoDB provides two levels of compatibility - "automatic encryption" and "explicit encryption". Automatic encryption is s more complete implementation, but it is only available in MongoDB Atlas and Enterprise editions.

We'll use the free Community Edition of MongoDB instead and implement our own explicit encryption with Spring Boot.

Why & When to Use CSFLE

CSFLE could be useful for storing highly sensitive data in the database. I say "could" because it imposes some limitations on MongoDB features (e.g. indexes) that should also be considered carefully. So make sure you fully understand the implications and the limitations in the documentation before using CSFLE.

I think it is still good practice to encrypt the database at rest on the filesystem (including it's backups) and the data in transit between the application and the database (i.e. using TLS). CSFLE adds another level of protection for sensitive user data.

I also wouldn't use CSFLE for passwords. Salted hashes are still much more approriate for that use case because passwords are never stored at all.

Also, a warning; if you use this feature and delete the encryption key (and you don't have a backup), the data encrypted with that key will be lost with no possibility of recovery.

Approach

This article presents the following approach:

  • AWS KMS will be configured with Cloudformation. KMS is used to manage a key that the encryption library (mongodb-crypt) will use as a master key. The master key is used to encrypt and decrypt one or more data encryption keys that will be stored in the database.

  • We'll use and configure the mongodb-crypt library to make it work with Spring Data Mongo.

  • We'll implement a listener to intercept entity save and load events, allowing us to encrypt and decrypt specific fields transparently to our application.

  • Finally, we'll use a Sprint integration test to test our encryption integration, including using a MongoDB Testcontainer to run mongo.

AWS Setup

We'll use AWS Cloudformation to create key management infrastructure using Key Management Service (KMS).

The first resource to define is our master key (i.e. the master key that encrypts and decrypts one or more data encryption keys used on data):

  MongoClientEncryptionMasterKey:
    Type: AWS::KMS::Key
    Properties:
      Description: Mongo Client Encryption Master Key
      KeyUsage: ENCRYPT_DECRYPT
      KeyPolicy:
        Version: '2012-10-17'
        Id: MongoClientEncryptionMasterKeyPolicy
        Statement:
          - Sid: Allow access for Key Administrators
            Effect: Allow
            Principal:
              AWS:
                - !Join  [ ':', [ 'arn:aws:iam:', Ref: 'AWS::AccountId', 'root' ] ]
                - !Join  [ ':', [ 'arn:aws:iam:', Ref: 'AWS::AccountId', 'role/super-admin-role' ] ]
            Action:
              - kms:*
            Resource: "*"

Next we need a key alias:

  MongoClientEncryptionMasterKeyAlias:
    Type: 'AWS::KMS::Alias'
    Properties:
      AliasName: 'alias/mongo-client-encryption-master-key'
      TargetKeyId: !Ref MongoClientEncryptionMasterKey

We also need to create a user that will access the KMS API to use this key:

  MongoClientEncryptionUser:
    Type: AWS::IAM::User
    Properties:
      UserName: 'mongo-client-encryption-user'

Finally, we need a policy to allow the user to encrypt and decrypt data using the key:

  MongoClientEncryptionMasterKeyAccessPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: MongoClientEncryptionMasterKeyAccessPolicy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowEncryptDecrypt
            Effect: Allow
            Action:
              - kms:Encrypt
              - kms:Decrypt
            Resource: !GetAtt MongoClientEncryptionMasterKey.Arn
      Users:
        - Ref: MongoClientEncryptionUser

Note that because we created an IAM user, the application has to be deployed with a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and they should be stored and accessed securely (i.e. using a secrets management system that stores the values in encrypted form). Don't commit these value to the codebase. The keys can be created in the IAM interface in the AWS Web Console.

Using key credentials like this isn't AWS best practice, but it is required by the mongodb-crypt library that we will use. Mongodb-crypt uses these credentials to call the KMS API and generate a data encryption key.

It's therefore good practice that the policy associated with the MongoClientEncryptionUser has the minimal set of permissions required.

Spring Boot Configuration

The first thing we'll need is the mongodb-crypt library (here in Gradle):

implementation 'org.mongodb:mongodb-crypt:1.8.0'

Let's define some configuration:

app:
  mongodb:
    encryption:
      provider: aws
      aws-region: eu-west-2
      aws-kms-key-id: key-id
      key-vault-namespace: example.key_vault
      key-vault-alias: example
      # aws-access-key-id: replace-with-secret
      # aws-secret-access-key: replace-with-secret

We'll use a property class for convenience (Lombok annotations are used to keep it concise):

@Data
@NoArgsConstructor
@ConfigurationProperties(prefix = "app.mongodb.encryption")
public class MongoEncryptionProperties {
    private String provider;
    private String keyVaultAlias;
    private String keyVaultNamespace;
    private String awsRegion;
    private String awsKmsKeyId;
    private String awsAccessKeyId;
    private String awsSecretAccessKey;
}

And include it in our mongo configuration:

@AllArgsConstructor
@Configuration
@AutoConfigureAfter(MongoAutoConfiguration.class)
@EnableConfigurationProperties(MongoEncryptionProperties.class)
public class MongoEncryptionConfig {

    private final MongoEncryptionProperties properties;

The mongodb-crypt library requires us to configure some options and then we'll need to create a ClientEncryption object and mongoClientEncryptionDataKey to use later.

Configuring MongoDB Crypt

We'll need the AWS configuration by default when running the application. We will also use the alternative "local" provider but only during integration testing with a local key.

@Bean
@ConditionalOnProperty(
        name = "app.mongodb.encryption.provider", havingValue = "aws")
public Map<String, Map<String, Object>> kmsProviderConfigMap() {
    return Map.of("aws", Map.of(
            "accessKeyId", properties.getAwsAccessKeyId(),
            "secretAccessKey", properties.getAwsSecretAccessKey()
    ));
}

Here's the configuration options for KMS:

@Bean
@ConditionalOnProperty(
        name = "app.mongodb.encryption.provider", havingValue = "aws")
public DataKeyOptions dataKeyOptions() {
    DataKeyOptions options = new DataKeyOptions();
    BsonDocument masterKey = new BsonDocument();
    masterKey.put("provider", new BsonString("aws"));
    masterKey.put("region", new BsonString(properties.getAwsRegion()));
    masterKey.put("key", new BsonString(properties.getAwsKmsKeyId()));
    options.masterKey(masterKey);
    options.keyAltNames(Collections.singletonList(properties.getKeyVaultAlias()));
    return options;
}

We'll need to create the ClientEncryption instance by providing the customised MongoClientSettings:

@Bean
public ClientEncryption mongoClientEncryption(
        ObjectProvider<MongoClientSettingsBuilderCustomizer> customizers,
        MongoClientSettings standardSettings,
        Map<String, Map<String, Object>> kmsProviderConfigMap) {

    MongoClientSettings.Builder builder =
            MongoClientSettings.builder(standardSettings);

    customizers.orderedStream()
            .toList()
            .forEach(c -> c.customize(builder));

    MongoClientSettings customSettings = builder.build();

    ClientEncryptionSettings clientEncryptionSettings =
            ClientEncryptionSettings.builder()
                    .keyVaultMongoClientSettings(customSettings)
                    .keyVaultNamespace(properties.getKeyVaultNamespace())
                    .kmsProviders(kmsProviderConfigMap)
                    .build();

    return ClientEncryptions.create(clientEncryptionSettings);
}

Finally, we'll create a data encryption key or load it from the database if it already exists:

@Bean
public BsonBinary mongoClientEncryptionDataKey(
        ClientEncryption clientEncryption, DataKeyOptions dataKeyOptions) {
    BsonDocument key = clientEncryption.getKeyByAltName(
            properties.getKeyVaultAlias());
    if (key == null) {
        return clientEncryption.createDataKey(
                properties.getProvider(), dataKeyOptions);
    } else {
        return key.getBinary("_id");
    }
}

Encrypting and Decrypting Fields

We can contain this functionality in a single class called MongoEncrypter. We're using the mongoClientEncryption and mongoClientEncryptionDataKey Spring beans that were created earlier in the configuration:

@AllArgsConstructor
@Component
public class MongoEncrypter {

    private static final String ENCRYPTION_ALGORITHM =
            EncryptionAlgorithms.AEAD_AES_256_CBC_HMAC_SHA_512_Random;

    private final ClientEncryption mongoClientEncryption;
    private final BsonBinary mongoClientEncryptionDataKey;

Take particular note of the _Random at the end of the algorithm name. This randomises the encrypted output. If you need a deterministic result (e.g. for indexing), there is another algorithm called AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic.

The implementation of encrypt looks like this:

private BsonBinary encrypt(BsonValue bsonValue) {
    return mongoClientEncryption.encrypt(bsonValue,
            new EncryptOptions(ENCRYPTION_ALGORITHM)
                    .keyId(mongoClientEncryptionDataKey));
}

And for decrypt we don't even need to specify the algorithm as it is encoded in the data:

private BsonValue decrypt(Binary input) {
    return mongoClientEncryption.decrypt(
            new BsonBinary(input.getType(), input.getData()));
}

In this example we'll make it simple and only support String fields. You can extend this to support other field types you want:

public void encrypt(String key, Document document) {
    Optional.ofNullable(document.get(key))
            .map(String.class::cast)
            .map(BsonString::new)
            .ifPresent(bson -> document.replace(key, encrypt(bson)));
}

Finally, to decrypt:

public void decrypt(String key, Document document) {
    Optional.ofNullable(document.get(key))
            .map(Binary.class::cast)
            .map(this::decrypt)
            .ifPresent(bson -> 
                document.replace(key, bson.asString().getValue()));
}

Note that it is at the decryption stage that knowing the type of the destination field is important. In this simple example, we can safely assume the encrypted binary fields are always converted back to strings.

Handling Entity Events

Let's define an entity containing a sensitive field socialSecurityNumber:

@Builder
@Value
@Immutable
@Document(collection = "person")
public class Person {
    @Id
    String id;
    String firstName;
    String lastName;
    String socialSecurityNumber;
}

We can then use a AbstractMongoEventListener to encrypt and decrypt the fields we want before the Person entity is saved and after it is loaded:

@AllArgsConstructor
@Component
public class PersonMongoEventListener 
        extends AbstractMongoEventListener<Person> {

    private static final String
            SOCIAL_SECURITY_NUMBER = "socialSecurityNumber";

    private final MongoEncrypter encrypter;

    @Override
    public void onBeforeSave(BeforeSaveEvent<Person> event) {
        Optional.ofNullable(event.getDocument())
                .ifPresent(doc ->
                        encrypter.encrypt(SOCIAL_SECURITY_NUMBER, doc));
    }

    @Override
    public void onAfterLoad(AfterLoadEvent<Person> event) {
        Optional.ofNullable(event.getDocument())
                .ifPresent(doc ->
                        encrypter.decrypt(SOCIAL_SECURITY_NUMBER, doc));
    }
}

The getDocument() method is annotated with @Nullable, so the code guards this.

The key thing advantage of this implementation is that the encryption is transparent to the application when Person is accessed via the repository.

Integration Testing

For testing, instead of using KMS we'll just use a local test key that we can generate ourselves. We need to substitute some of the configuration to use the "local" provider instead of the "aws" provider inside a @TestConifguration class.

First, we'll trigger the local configuration in application-it.yml like this:

app:
  mongodb:
    encryption:
      provider: local

Next, we substitute the kmsProviderConfigMap using @ConditionalOnProperty:

@Bean
@ConditionalOnProperty(
        name = "app.mongodb.encryption.provider", havingValue = "local")
public Map<String, Map<String, Object>> kmsProviderConfigMap() {
    Map<String, Object> keyMap = new HashMap<>();
    keyMap.put("key", loadTestMasterKey());
    Map<String, Map<String, Object>> providersMap = new HashMap<>();
    providersMap.put("local", keyMap);
    return providersMap;
}

The test key is loaded from the filesystem like this:

private byte[] loadTestMasterKey() {
    try {
        return readKey("src/test/resources/test-master.key");
    } catch (IOException e) {
        throw new IllegalStateException("Could not load local master key", e);
    }
}

public byte[] readKey(String masterKeyPath) throws IOException {
    byte[] masterKey = new byte[96];
    try (FileInputStream stream = new FileInputStream(masterKeyPath)) {
        if (stream.read(masterKey, 0, 96) == 0)
            throw new IOException("invalid key");
    }
    return masterKey;
}

The local key needs to be a base64-encoded 96-byte string with no line breaks and can be created like this:

openssl rand 96 | openssl base64 -A > test-master.key

We also need to substitute dataKeyOptions:

@Bean
@ConditionalOnProperty(
    name = "app.mongodb.encryption.provider", havingValue = "local")
public DataKeyOptions dataKeyOptions() {
    DataKeyOptions options = new DataKeyOptions();
    options.keyAltNames(Collections.singletonList("test"));
    return options;
}

Finally, let's create a couple of test cases. We'll need some supporting objects, including PersonRepository for saving a loading the Person entity, our MongoEncrypter and an ArgumentCaptor to capture and inspect the raw Mongo document.

@ContextConfiguration(classes = 
    {TestcontainersConfig.class, IntegrationTestConfig.class})
@Tag("integration")
@ActiveProfiles({"it"})
@SpringBootTest
public class MongoEncryptionIT {

    static final String 
        SOCIAL_SECURITY_NUMBER_FIELD = "socialSecurityNumber";

    @Autowired
    PersonRepository personRepository;
    @SpyBean
    MongoEncrypter mongoEncrypter;
    @Captor
    ArgumentCaptor<Document> documentCaptor;

Notice the use of @SpyBean as so we can validate the MongoEncrypter is called the correct number of times. We'll also capture the raw Mongo Document using ArgumentCaptor to check if the socialSecurityNumber field was converted.

Here's our encryption test:

@Test
void shouldEncryptSocialSecurityNumberOnSave() {
    personRepository.save(Person.builder()
            .firstName("Joe")
            .lastName("Blogs")
            .socialSecurityNumber("123456")
            .build());

    verify(mongoEncrypter, times(1))
            .encrypt(
                eq(SOCIAL_SECURITY_NUMBER_FIELD), 
                documentCaptor.capture());

    Document document = documentCaptor.getValue();
    assertThat(document.get(SOCIAL_SECURITY_NUMBER_FIELD))
        .isInstanceOf(BsonBinary.class);
}

We see that the socialSecurityNumber has been coverted from a plaintext String to an encrypted BsonBinary.

Next up is the decryption test:

@Test
void shouldDecryptSocialSecurityNumberOnLoad() {
    Person persisted = personRepository.save(Person.builder()
            .firstName("Joe")
            .lastName("Blogs")
            .socialSecurityNumber("123456")
            .build());

    personRepository.findById(persisted.getId());

    verify(mongoEncrypter, times(1))
            .decrypt(
                eq(SOCIAL_SECURITY_NUMBER_FIELD), 
                documentCaptor.capture());

    Document document = documentCaptor.getValue();
    assertThat(document.get(SOCIAL_SECURITY_NUMBER_FIELD))
        .isEqualTo("123456");
}

Notice how we have first saved the entity, then reloaded it using the findById method.

Finally, we'll need a running instance of Mongo which is easy using a Testcontainer:

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfig {

    @Bean
    @ServiceConnection
    MongoDBContainer mongoDbContainer() {
        return new MongoDBContainer(DockerImageName.parse("mongo:5.0"));
    }
}

Conclusion

In this article we covered how to implement CSFLE in Mongo with Spring Boot and KMS for free using the Community Edition of MongoDB. The implementation is simple and can be extended. There are limitations, however, and I encourage you to read the documentation fully.

The Enterprise and Atlas licensed version of CSFLE come with a JSON-based configuration for the fields to encrypt as opposed to hardcoding, as I've shown here. That allows some flexbility to turn the feature on or off in different environments.

The complete accompanying code for this article is available on github.

JavaSpringbootMongoDBAWSencryption