Spring Cloud Config Hot Reloading

Applying configuration changes in Spring Cloud without restarting the application using Kafka

Introduction

In the ever-evolving landscape of microservices and cloud-native applications, managing configurations effectively is a critical aspect of maintaining application reliability and flexibility. This article explores a step-by-step journey in optimizing configuration management for microservices, highlighting the evolution from manual updates to achieving hot reloading with the help of tools like Git, Vault, Spring Cloud Config Server, and Kafka. We’ll delve into each step, uncovering the pros and cons of each approach, and finally demonstrate how to achieve seamless configuration updates without the need for application restarts. Join us as we embark on this journey toward more efficient and dynamic microservices configuration management.

Spring Cloud Config Server Introduction

Step 1 (Traditional way)

updading the application configuration manually using SSH or kubectl edit configmap commands

In the above picture, you can see that you need to change the configuration of your microservices one by one. The Catalog Service and Order Service are the names of two microservices that I used as examples.

Pros:

• Simplicity: It’s a lot simpler and easier to understand.

Cons:

• Time-consuming: It takes a lot of time, and you need to SSH into your VM or use kubectl edit configmap … to edit your configuration.
• Restart required for config changes: You need to restart your application when you change the configuration file.

Step 2 (Store configs in Git)

Storing microservices configurations inside a Git repository

In the above architecture, you can see that the applications are retrieving their configuration from a Git repository instead of obtaining it from a local volume.

Pros:

• Version Control over Configuration Files: It helps your team track configuration changes and revert misconfigurations.
• Easy Access to All Configurations: You don’t need to SSH or use kubectl to update your configuration; you just need to commit changes to the Git repository.

Cons:

• Security: Storing passwords in a Git repository can lead to security vulnerabilities. You need to store your passwords encrypted and decrypt them when necessary for changes.

Step 3 (Adding Vault for Sensitive Configs)

In the above diagram, I added Vault to store passwords inside it instead of the Git repository. The git will be used for only non-sensitive configurations.

Pros:

• Security: Storing sensitive configurations like database passwords in Vault improves security.

Cons:

• Scalability: You need to insert the Git repository password and Vault token into all microservices to connect them to Git and Vault for retrieving their configurations. When you change the Vault token or Git password, you’ll need to restart all applications.

Step 4 (Adding Central Config Server)

Pros:

• Central Config Server: Reduces the Cost of Change by incorporating Spring Cloud Config Server to provide configurations to the applications via REST calls.
• Improved Security: The credentials of Vault and Git are stored exclusively in the config server.

Cons:

• Configuration changes require microservices restart: If you change the configuration inside the Git or Vault servers, you’ll need to restart the microservices to retrieve the new configurations.

Step 5 (Adding Kafka for hot reloading)

The Spring Cloud Config has an endpoint called /actuator/busrefresh. When you send a request to this endpoint, you can check configuration changes and put them in the Kafka queue. You don’t need to write any code for interacting with Kafka (it’s handled by the Spring Cloud library); you just need to configure the Kafka address inside the application.properties file.

Pros:

• Hot Reloading: Configuration changes do not require microservices to be restarted.

Cons:

• High complexity: Adding Kafka and maintaining it increases the complexity of the system.
• Inconsistency: After changing the configuration and before calling /actuator/busrefresh if microservices restart, inconsistency may happen.

In Action — Practice

In the following section, I’ll describe the ways to run a config server and connect it to the Kafka queue for reloading the application configurations without the need to restart applications.

$ git clone https://github.com/mlkmhd/devops-diagrams
$ cd devops-diagrams/spring-cloud-hot-reloading
$ docker-compose up -d vault
 ✔ Network spring-cloud-hot-reloading_default              Created
 ✔ Container vault-server                                  Started

By the above command, we brought up the Vault server. First, go to the Vault dashboard at the following address:

http://localhost:8200/ui/vault/secrets/secret/list

token: root

hit the Create secret + button and then:

Create two secrets for two microservices as shown in the above image. One of our microservices is catalog-service in the dev profile, and another one is order-service in the dev profile. You can separate the service name and profile name by a comma (,) separator. You will need to define the following information:

Path for this secret: catalog-service,dev
Secret data: (key: db.password, value: 123)

Path for this secret: order-service,dev
Secret data: (key: db.password, value: moresecurePassw0rd)

Now, you can bring up all other services using the following command:

$ cd microservices/config-server
$ ./gradlew clean build
$
$ cd ../catalog-service
$ ./gradlew clean build
$
$ cd ../order-service
$ ./gradlew clean build
$
$ cd ../..
$ docker-compose up -d 
[+] Running 5/5
 ✔ Container spring-cloud-hot-reloading-config-server-1    Started
 ✔ Container kafka                                         Started
 ✔ Container spring-cloud-hot-reloading-order-service-1    Started
 ✔ Container spring-cloud-hot-reloading-catalog-service-1  Started
 ✔ Container vault-server                                  Running

Okay, now all applications are ready, and you can test the configuration hot reloading. First, visit the http://localhost:8082/config address. This endpoint exposes a db.password which is loaded from the config server:

You can go to the vault dashboard and change the password:

And hit the save button to store the new password. Then, you need to call the /actuator/busrefresh endpoint of the config-server to reload the application configuration:

$ curl -vvX POST http://localhost:8888/actuator/busrefresh 
* processing: http://localhost:8888/actuator/busrefresh
*   Trying [::1]:8888...
* Connected to localhost (::1) port 8888
> POST /actuator/busrefresh HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 204 
< Date: Fri, 02 Feb 2024 07:06:18 GMT
<
* Connection #0 to host localhost left intact

Okay, it returned 204, which means the refresh was successful. Now you can see the new configuration by refreshing the http://localhost:8082/config address:

Let’s deep dive into the code details to review the code behind the scenes.

Config Server

Let’s review the detailed codes of modules used in this article. This is the simple main class of the Config Server. You need to use the @EnableConfigServer annotation for enabling the configuration server:

@SpringBootApplication
@EnableConfigServer
public class ConfigServerApplication {

 public static void main(String[] args) {
  SpringApplication.run(ConfigServerApplication.class, args);
 }

}

the configuration of the Config Server should be something like this:

server.port=8888

spring.profiles.active=git,vault

spring.cloud.config.server.git.uri=https://github.com/mlkmhd/spring-cloud-sample-config
spring.cloud.config.server.git.defaultLabel=main
spring.cloud.config.server.git.skipSslValidation=true

# Git Refresh Rate
# You can control how often the config server will fetch updated configuration data from your Git backend by using spring.cloud.config.server.git.refreshRate.
# The value of this property is specified in seconds.
# By default the value is 0, meaning the config server will fetch updated configuration from the Git repo every time it is requested.
spring.cloud.config.server.git.refreshRate=60

spring.cloud.config.server.vault.host=vault
spring.cloud.config.server.vault.port=8200
spring.cloud.config.server.vault.scheme=http
spring.cloud.config.server.vault.backend=secret
spring.cloud.config.server.vault.default-key=application
spring.cloud.config.server.vault.profile-separator=,
spring.cloud.config.server.vault.authentication=TOKEN
spring.cloud.config.server.vault.token=root
spring.cloud.config.server.vault.kvVersion=2
spring.cloud.vault.config.lifecycle.enabled=true
spring.cloud.vault.config.lifecycle.min-renewal=10s
spring.cloud.vault.config.lifecycle.expiry-threshold=1m

spring.kafka.bootstrap-servers=kafka1:9092

management.endpoints.web.exposure.include=*
management.endpoint.health.show-details=always

spring.cloud.bus.enabled=true

Order Service

This is one of the microservices that loads its configuration from the config server. In the detailed code, you need to use @RefreshScope for reloading your configuration:

package com.example.orderservice;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RefreshScope
@RestController
@RequestMapping("/config")
public class MyController {

    @Value("${db.password}")
    private String dbpassword;

    @Value("${db.name}")
    private String dbname;

    @GetMapping
    public String getConfigValue() {
        return "database password is: "+ dbpassword + ", and dbname is: "+ dbname;
    }
}

and the configuration of order service should be something like this:

server.port=8082
# config reads from http://CONFIG_SERVER:PORT/{spring.application.name}/{spring.profiles.active}/{master or main}
spring.config.import=optional:configserver:http://localhost:8888
spring.cloud.config.name=order-service
spring.cloud.config.fail-fast=true
spring.cloud.config.max-attempts=10
spring.cloud.config.max-interval=1500
spring.cloud.config.multiplier=1.2
spring.cloud.config.initial-interval=1100
spring.cloud.config.allowOverride=true
spring.cloud.config.overrideNone=true
spring.kafka.bootstrap-servers=localhost:9092
spring.cloud.bus.enabled=true

GitHub

You can find the code used in this documentation in the following GitHub repository:

devops-diagrams/spring-cloud-hot-reloading at main · mlkmhd/devops-diagrams

Conclusion

In conclusion, effective configuration management in microservices architectures is paramount to maintain agility, security, and reliability. This article has outlined a step-by-step evolution in configuration management, from traditional manual updates to achieving hot reloading. Each step has its advantages and challenges, and the choice of approach depends on the specific requirements and constraints of your environment.

By leveraging tools like Git, Vault, Spring Cloud Config Server, and Kafka, organizations can empower their development and operations teams to manage configurations more efficiently. The ability to make changes without requiring application restarts not only enhances productivity but also reduces potential disruptions in production environments.

Support My Blog ☕️

If you enjoy my technical blog posts and find them valuable, please consider buying me a coffee at here. Your support goes a long way in helping me produce more quality articles and content. If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.