/ Spring Cloud  

Spring Cloud 10: Unified Configuration Center

In monolithic applications, we often use a configuration file (application (*). Properties (yml)) to manage all the configuration of the application. These profiles are very good at their role in monolithic applications, and they don’t make us feel a headache. However, with the introduction of the microservice framework, the number of microservices will continue to increase in our products. Previously, we focused on the scalability and extensibility of the system, but then the problems of configuration management were exposed. At first, the microservers managed their own configurations. There was no problem in the development stage, but in the production environment management, it will be a headache. If you want to update a configuration on a large scale, you can imagine the difficulties.

To this end, in a distributed system, Spring Cloud provides a Config sub-project. The core of this project is the configuration center, which implements configuration services through one server and multiple clients. We can use a configuration server to centrally manage various environment profiles for all services. The configuration service center uses Git for storage by default, so we can easily deploy and modify it, and we can version control the environment configuration.

Spring Cloud Config has features such as centralization, version control, support for dynamic updates, and language independence. Its characteristics includes:

  • Provide server and client support (Spring Cloud Config Server and Spring Cloud Config Client);
  • Centralized management of application configuration in a distributed environment;
  • Based on the Spring environment, it achieves seamless integration with Spring applications;
  • Programs that can be used in any language development;
  • The default implementation is based on Git repositories (also supports SVN), so that version management can be performed;

The structure diagram of Spring Cloud Config is as follows:



We can see that Spring Cloud Config has two roles (similar to Eureka): Server and Client. As the server of the configuration center, Spring Cloud Config Server assumes the following functions:

  • Update the Git repository copy when pulling the configuration to ensure that the configuration is up to date;
  • Support loading configuration from yml, json, properties and other files;
  • Service discovery can be implemented with Eureke, and configuration push updates can be implemented with Cloud Bus (which I will explain in detail later);
  • The default configuration store is based on a Git repository (can be switched to SVN), which supports configuration version management.

For theSpring Cloud Config Client, it is very convenient. We only need to add which configuration file on the Config Server to the startup configuration file.

Example project

Config-Server

1. Pom.xml

Config-Server is a standard Spring Boot application

1
2
3
4
5
6
<dependencies>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-config-server</artifactId>
    </dependency>
</dependencies>

2. Main application

1
2
3
4
5
6
7
@SpringBootApplication
@EnableConfigServer
public class ConfigServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConfigServerApplication.class, args);
    }
}

Add @EnableConfigServer, others are consistent with previous applications.

3. application.properties

1
2
3
4
5
6
7
server.port=8280

spring.application.name=config-server

spring.cloud.config.server.git.uri=https://github.com/nicklee1006/SpringCloudDemoSampleConfig
spring.cloud.config.server.git.username=your git username
spring.cloud.config.server.git.password=your git password

The most important thing here is to configure the Git repository address and login username and password.

4. Sample config file

We add a sample configuration file to the SpringCloudDemoSampleConfig repository.

sample-service.properties

1
foo = bar

sample-service-dev.properties

1
bar = 12345

5. Test

Start service-discovery and config-server. In the terminal we enter the following command:

1
curl localhost:8280/sample-service/dev

The following will be output in the terminal:

1
2
3
4
5
6
{"name":"sample-service","profiles":["dev"],"label":null,"version":"17ca16db404875d8b7ab57c9bfce33b45557114f","state":null,"propertySources":
    [
        {"name":"https://github.com/nicklee1006/SpringCloudDemoSampleConfig/sample-service-dev.properties","source":{"bar":"12345"}},
        {"name":"https://github.com/nicklee1006/SpringCloudDemoSampleConfig/sample-service.properties","source":{"foo":"bar"}}
    ]
}

Here we can see that the configuration file we submitted to Git can be correctly read by config-server.

config-server default configuration

When we look at the source code, we find that there is a default configuration file configserver.yml in the spring-cloud-config-server.jar package, which means that when we set spring.application.name = configserver, the configuration will be loaded by default. The configuration file is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
info:
  component: Config Server
spring:
  application:
    name: configserver
  jmx:
    default_domain: cloud.config.server
  cloud:
    config:
      server:
        git:
          uri: https://github.com/spring-cloud-samples/config-repo
          repos:
            - patterns: multi-repo-demo-*
              uri: https://github.com/spring-cloud-samples/config-repo

server:
  port: 8888
management:
  context_path: /admin

Port 8888 is used by default, and the configuration file is found from the Git repository at https://github.com/spring-cloud-samples/config-repo.

config-client

config-client can be any application based on Spring boot. Here we build a very simple web project.

1. pom.xml

1
2
3
4
5
6
7
8
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

2. Main application

1
2
3
4
5
6
7
@EnableDiscoveryClient
@SpringBootApplication
public class ConfigClientApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConfigClientApplication.class, args);
    }
}

3. Controller

The main purpose of this controller is to verify that we can get the configuration content from the Git repository.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@RestController
@RequestMapping("/cfg")
public class ConfigController {
    @Value("${foo}")
    String foo;

    @Value("${bar}")
    String bar;

    @RequestMapping(value = "/foo")
    public String foo(){
        return foo + "——" + bar;
    }
}

4. bootstrap.properties

1
2
3
4
5
server.port=8080

spring.application.name=sample-service
spring.cloud.config.profile=dev
spring.cloud.config.uri= http://localhost:8280/

Defines the name and profile of the microservice and the address of the configuration server.

Note: These configurations cannot be configured in the application.properties file, because there is a boot context and application context when Spring Boot is started. Only when the configuration server information is defined in the boot context can the configuration information be obtained from the configuration server. Otherwise, when the service starts, it will report an error that the foo variable definition cannot be found.

5. Test

Start config-client and visit: http://localhost:8080/cfg/foo:



It indicates that our config-client has successfully obtained the configuration data from the config-server.

Spring project configuration load order

  • Command line arguments
  • SPRING_APPLICATION_JSON parameter
  • Load JNDI properties from java:comp/env
  • Java system properties (System.getProperties())
  • Operating system environment variables
  • If configured using random.* Properties, use RandomValuePropertySource to generate
  • External application-specific configuration files, such as application-{profile}.properties or YAML variants
  • Internal application-specific profile files such as: application-{profile}.properties or YAML variants
  • External application configuration files, such as application.properties or YAML variants
  • Internal application configuration files such as: application.properties or YAML variants
  • Load the configuration file pointed to by @PropertySource or @ConfigurationProperties of the @Configuration class
  • Default configuration, set by SpringApplication.setDefaultProperties

Detailed configuration rules

Take a look at how config-client obtaining configuration data from config-server:

  1. When the config-client starts, it requests the config-server for configuration data according to the application, profile, and label configured in bootstrap.properties;
  2. The config-server searches for a matching configuration file from a Git repository (here Git is taken as an example) according to the request and configuration of the config-client;
  3. The config-server pulls the matched Git repository to local and establishes a local cache;
  4. The config-server creates a Spring ApplicationContext instance, populates the configuration information according to the pulled configuration file, and then returns the configuration information to the config-client;
  5. After the config-client obtains the configuration data returned by the config-server, it loads these configuration data into its own context. At the same time, because the priority of these configuration data is higher than the configuration in the local Jar package, the local configuration will no longer be loaded.

So, how does the config-server match the configuration files in the Git repository? Usually, we will create a configuration file similar to the following for a project:

  • service.properties: basic configuration file;
  • service-dev.properties: configuration file for development;
  • service-test.properties: configuration file used for testing;
  • service-prod.properties: configuration file used by the production environment;

When we visit the endpoint of config-server, we will match the corresponding configuration file according to the following mapping relationship:

  1. /{application}/{profile}[/{label}]
  2. /{application}-{profile}.yml
  3. /{label}/{application}-{profile}.yml
  4. /{application}-{profile}.properties
  5. /{label}/{application}-{profile}.properties

The above Url will be mapped to a profile with the format: {application}-{profile}.properties(yml). In addition, label corresponds to the branch name on Git and is an optional parameter. If not, it is the default master branch.

The bootstrap.properties of config-client corresponds to the following:

  • spring.application.name <==> application;
  • spring.cloud.config.profile <==> profile;
  • spring.cloud.config.label <==> label.

Git repository config

The config-server uses Git by default, so the configuration is very simple, such as the configuration(application.properties) above:

1
2
3
spring.cloud.config.server.git.uri=http://
spring.cloud.config.server.git.username=username
spring.cloud.config.server.git.password=password

1. Using placeholders

We can also use {application}, {profile} and {label} placeholders in the server configuration, as follows:

1
2
3
spring.cloud.config.server.git.uri = http://github.com/cd826/{application}
spring.cloud.config.server.git.username = username
spring.cloud.config.server.git.password = password

In this way, we can create a separate repository for each application client.

It should be noted here that if a branch or label in Git contains “/“, you need to use “(_)” instead in the {label} parameter. This is mainly to avoid conflicts with Http URL escape character processing.

2. Use pattern matching

We can also use {application}/{profile} for pattern matching in order to get the corresponding profile. An example configuration is as follows:

1
2
3
4
5
6
7
8
9
spring.cloud.config.server.git.uri = https://github.com/spring-cloud-samples/config-repo

spring.cloud.config.server.git.repos.simple = https://github.com/simple/config-repo

spring.cloud.config.server.git.repos.special.pattern = special */dev*, *special*/dev*
spring.cloud.config.server.git.repos.special.uri = https://github.com/special/config-repo

spring.cloud.config.server.git.repos.local.pattern = local*
spring.cloud.config.server.git.repos.local.uri = file:/home/configsvc/config-repo

If multiple values ​​need to be configured in the pattern, they can be separated by commas.

If {application}/{profile} does not match any resources, the default URI configured by spring.cloud.config.server.git.uri is used.

When we use a yml type file for configuration, if the schema attribute is a YAML array, it can also be defined using a YAML array format. This can be set to multiple configuration files, such as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
spring:
  cloud:
    config:
      server:
        git:
          uri: https://github.com/spring-cloud-samples/config-repo
          repos:
            development:
              pattern:
                - */development
                - */staging
              uri: https://github.com/development/config-repo
            staging:
              pattern:
                - */qa
                - */production
              uri: https://github.com/staging/config-repo

3. Search directory

When we store the configuration file in a subdirectory in the Git repository, we can specify the directory by setting search-path. Similarly, search-path also supports the above placeholders. Examples are as follows:

1
2
spring.cloud.config.server.git.uri = https://github.com/spring-cloud-samples/config-repo
spring.cloud.config.server.git.searchPaths = foo, bar*

In this way, the system will automatically search the subdirectories of foo and the subdirectories in the folders starting with bar.

4. SSH configuration

If you do not want to use HTTPS and user authentication, you can also use SSH directly. At this time, we only need to store the keys required by ssh in the ~/.ssh directory, and point the configured URI to the SSH address, such as: git@github.com:nicklee1006/SpringCloudDemoSampleConfig.

If you know your ~/.git directory clearly, then you can use git config --global to configure it. Otherwise, you can use global configuration, such as: git config --global http.sslVerify false.

5. Proxy

config-server uses JGit to access the configuration repository, so we can configure the proxy used by HTTPS under ~/.git/config, or use the JVM system properties -Dhttps.proxyHost and -Dhttps.proxyPort to configure.

6. Local cache

When config-server obtains the configuration information from Git (or SVN), it will store a copy in the local file system. By default, it will be stored in the system temporary directory, and it starts with config-repo-. On Linux systems, the default directory is /tmp/config-repo-<randomid>. config-server storing configuration information locally can effectively prevent the problem that the Git repository fails and cannot be accessed. When config-server cannot access the Git repository, it will read the configuration stored in the local file before, and then The configuration information is returned to config-server.

The official Spring Cloud documentation suggests that we specify a local file path in config-server to avoid unpredictable errors. You can use the following property configuration to specify the local file path:

1
2
3
4
5
## Git
spring.cloud.config.server.git.basedir=tmp/

## SVN
spring.cloud.config.server.svn.basedir=tmp/

SVN configuration

If using SVN instead of Git, only need to make the following changes in config-server to support SVN repositories.

pom.xml

Add the following dependencies to the pom file:

1
2
3
4
<dependency>
     <groupId>org.tmatesoft.svnkit</groupId>
     <artifactId>svnkit</artifactId>
</dependency>

appliaction.properties

1
2
3
spring.cloud.config.server.svn.uri = {your svn server}
spring.cloud.config.server.svn.username = username
spring.cloud.config.server.svn.password = password

File system

If don’t want to use Git or SVN in your config-server, then we can also load the corresponding configuration file directly from the current classpath or file system, just set in the configuration file as follows:

1
spring.profiles.active = native

Note that config-server is loaded from the classpath by default. We can use the spring.cloud.config.server.native.searchLocations property to set the directory of the configuration file. For the file path, our configuration must start with file:. If it is a Windows system, we must escape / for the absolute path. For example, under Windows we need to configure the following: file:///${user.home}/config-repo.

In addition, when we use the file system as a configuration file repository, the configuration of spring.cloud.config.server.native.searchLocations also supports {application}, {profile}, and {label} placeholders.

Spring Cloud officially recommends that you can use the file system during testing and development, but in the formal environment, try to use Git or SVN.

In addition, Spring Cloud Config also supports another configuration file method: Vault Server.

Security

config-server access security

For some of the configuration content that we store in the configuration center, there will always be some sensitive information, such as the user name and password for the database connection. You ca n’t run naked, so we still need to do some security control on config-server. Of course, there are many types of security controls for config-server, such as physical network restrictions, OAuth2 authorization, and so on. However, because we are using SpringBoot here, using Spring Security will be easier and simpler. At this time, we only need to add the following dependencies to config-server

1
2
3
4
<dependency>
     <groupId>org.springframework.boot</ groupId>
     <artifactId>spring-boot-starter-security</artifactId>
</dependency>

When we start config-server, Spring Security will generate an access password for us by default. This method is often not what we need, so generally we need to configure the username and password in the configuration file, such as:

1
2
spring.security.user.name=username
spring.security.user.password=pwd

When we need to access config-server, a user authentication dialog will pop up. At this time, for config-client we need to add user and access password configuration in the configuration file, as follows:

1
2
spring.cloud.config.username=username
spring.cloud.config.password=pwd

Encryption and decryption

Access security is the overall control. In most cases, we also need to encrypt the sensitive content and store it, such as the user name and login password for database access. Fortunately, Spring Cloud Config provides us with corresponding support.

Spring Cloud Config provides two encryption and decryption methods:

  • symmetric encryption;
  • asymmetric encryption.

Before describing how to use it, let’s look at some prerequisites.

Install JCE (Java Cryptography Extension)

The encryption and decryption provided by Spring Cloud Config depends on JCE.

For JDK 9 and later, JCE is already included

For earlier releases, we need to install JCE first. The installation method is also relatively simple, that is, download the corresponding Jar packages, and then replace these packages with the files corresponding to $JDK_HOME/jar/lib/security. For JDK8, the download address is: JCE.

Encryption and decryption endpoint

In addition, Spring Cloud Config also provides two endpoints for encryption and decryption, as follows:

  • /encrypt: Encrypt endpoint, use the format: curl $CONFIG_SERVER/encrypt -d The content to be encrypted
  • /decrypt: The decryption endpoint, using the format: curl $CONFIG_SERVER/decrypt -d The content to be decrypted

Note: When your test contains special characters in encryption and decryption, URL encoding is required. At this time, you need to use --data-urlencode instead of -d.

Symmetric encryption

The configuration of symmetric encryption and decryption is very simple. We just need to add the key used for encryption and decryption in the configuration file, such as:

1
encrypt.key = key

After configuration, you can start config-server and use the endpoint mentioned above for encryption and decryption testing.

For the configuration file, we need to add a {cipher} leader to the encrypted content. Such as:

1
2
spring.datasource.username = dbuser
spring.datasource.password = {cipher}FKSAJDFGYOS8F7GLHAKERGFHLSAJ

However, if the configuration file you use is in yml format, you need to use single quotes to enclose the encrypted content, as follows:

1
2
3
4
spring:
     datasource:
         username: dbuser
         password: '{cipher}FKSAJDFGYOS8F7GLHAKERGFHLSAJ'

Asymmetric encryption

Asymmetric encryption is a bit more complicated than symmetric encryption. First, we need to use Java’s keytool to generate a key pair, and then create a Key Store and copy it to the server directory.

  1. Use keytool to generate Key Store, the command is as follows:

    1
    2
    3
    $keytool -genkeypair -alias tsfjckey -keyalg RSA \
       -dname "CN = Mall Web, OU = TwoStepsFromJava, O = Organization, L = city, S = province, C = china" \
       -keypass javatwostepsfrom -keystore server.jks -storepass twostepsfromjava

  2. Copy the generated server.jks to the resources directory of the project config-server.

  3. Modify the configuration file:

1
2
3
4
encrypt.key-store.location = server.jks
encrypt.key-store.alias = tsfjckey
encrypt.key-store.password = pwd
encrypt.key-store.secret = pwd

Asymmetric encryption is also more complicated to configure than symmetric encryption, but its security is also much higher.

Use multiple encryption keys

Maybe we need to use different encryption keys for different sensitive information. For this, our configuration file only needs to be written as follows:

1
foo.bar = (cipher) {key: testkey} ...

When config-server decrypts, it will try to obtain testkey from the configuration file as the key.

High availability configuration

Integrate Eureka

We used the specific address when configuring config.uri in config-server, so can I use Eureka? The answer is yes, we can use config-server as a basic unit of services just like other microservices. We just need to make the following modifications.

Config-Server

Add following dependency to pom.xml

1
2
3
4
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

Configure the name of our service in the configuration file and the address of the Eureka server:

1
2
3
spring.application.name=config-server

eureka.client.service-url.defaultZone=http://localhost:8761/eureka

Added @EnableDiscoveryClient annotation to main class.

1
2
3
4
5
6
7
8
@SpringBootApplication
@EnableConfigServer
@EnableDiscoveryClient
public class ConfigServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConfigServerApplication.class, args);
    }
}

If we start config-server, we will see the corresponding service registration on the Eureka server.

Config-Client

Add following dependency to pom.xml

1
2
3
4
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

Modify bootstrap.properties:

1
2
3
4
5
6
7
8
9
server.port=8080

spring.application.name=sample-service
spring.cloud.config.profile=dev

eureka.client.service-url.defaultZone=http://localhost:8761/eureka

spring.cloud.config.discovery.enabled=true
spring.cloud.config.discovery.service-id=config-server

The most important thing here is to add: spring.cloud.config.discovery.enabled = true in the configuration, and change the spring.cloud.config.uri configured to spring.cloud.config.discovery.service-id.

Modify main class.

1
2
3
4
5
6
7
@EnableDiscoveryClient
@SpringBootApplication
public class ConfigClientApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConfigClientApplication.class, args);
    }
}

Fast failure and response

Start Config-Server to start loading

By default, the server will load from the configured Git repository only when the client requests it. We can set clone-on-start to make the server load at startup.

1
2
3
4
5
6
7
8
9
10
11
12
spring.cloud.config.server.git.uri = https://git/common/config-repo.git

spring.cloud.config.server.git.repos.team-a.pattern = team-a- *
spring.cloud.config.server.git.repos.team-a.clone-on-start = true
spring.cloud.config.server.git.repos.team-a.uri = http://git/team-a/config-repo.git

spring.cloud.config.server.git.repos.team-b.pattern = team-b- *
spring.cloud.config.server.git.repos.team-b.clone-on-start = false
spring.cloud.config.server.git.repos.team-b.uri = http://git/team-b/config-repo.git

spring.cloud.config.server.git.repos.team-c.pattern = team-c- *
spring.cloud.config.server.git.repos.team-c.uri = http://git/team-a/config-repo.git

The above configuration, for team-a, will load the corresponding configuration when config-server starts, but not for others. Of course, we can do global configuration by setting the value of spring.cloud.config.server.git.clone-on-start.

Enable Config-Client Fast failure

In some cases, we want to fast failure a service when it is unable to connect to the server. We can achieve it by set:

1
spring.cloud.config.fail-fast = true

Set Config-Client to retry

If config-server happens to be unavailable at startup and you want to retry later, then we start to enable config-clients retry mechanism. First, we need to configure:

1
spring.cloud.config.fail-fast = true

Then we need to add to our dependencies:

1
2
3
4
5
6
7
8
<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

In this way, we can enable the retry mechanism for the config-client. When the connection to the config-server fails, the config-client will continue to try to connect to the config-server. By default, it will try to connect 6 times. The time interval is initially 1000 milliseconds. Each connection attempt will increase the interval between connection attempts by a factor of 1.1. An error will only be returned if the connection cannot be connected to config-server at the end. We can do this by overriding spring.cloud.config.retry.* In the configuration file.

If you want full control over the retry mechanism, you can implement the class: RetryOperationsInterceptor and set the bean id to: configServerRetryInterceptor.

Dynamic refresh configuration

THe config-client provides a refresh endpoint to refresh the configuration file. To use this feature, we need to add the following dependencies to the pom.xml file of config-client:

1
2
3
4
<dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId> spring-boot-starter-actuator</artifactId>
</dependency>

In this way, when the configuration file is modified and submitted to the Git repository, you can use: http://localhost: 8080/refresh to refresh the local configuration data.

However, the best way is to integrate with Spring Cloud Bus so that the automatic distribution of the configuration can be achieved instead of manually refreshing the configuration.

Check out the source code here: Config demo