Configuration Management - Clustered Deployments with Kafka

Apache Kafka is an open-source message broker project developed by the Apache Software Foundation. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It is, in its essence, a “massively scalable pub/sub message queue designed as a distributed transaction log”, making it highly valuable for enterprise infrastructures to process streaming data.

Support is enabled by including the following dependency in the final overlay:

1
2
3
4
5
<dependency>
    <groupId>org.apereo.cas</groupId>
    <artifactId>cas-server-support-configuration-cloud-kafka</artifactId>
    <version>${cas.version}</version>
</dependency>
1
implementation "org.apereo.cas:cas-server-support-configuration-cloud-kafka:${project.'cas.version'}"
1
2
3
4
5
6
7
8
9
dependencyManagement {
    imports {
        mavenBom "org.apereo.cas:cas-server-support-bom:${project.'cas.version'}"
    }
}

dependencies {
    implementation "org.apereo.cas:cas-server-support-configuration-cloud-kafka"
}
1
2
3
4
5
6
7
8
9
10
dependencies {
    /*
        The following platform references should be included automatically and are listed here for reference only.

        implementation enforcedPlatform("org.apereo.cas:cas-server-support-bom:${project.'cas.version'}")
        implementation platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES)
        
    */
    implementation "org.apereo.cas:cas-server-support-configuration-cloud-kafka"
}

Broadcast CAS configuration updates to other nodes in the cluster via Kafka.

The following settings and properties are available from the CAS configuration catalog:

The configuration settings listed below are tagged as Required in the CAS configuration metadata. This flag indicates that the presence of the setting may be needed to activate or affect the behavior of the CAS feature and generally should be reviewed, possibly owned and adjusted. If the setting is assigned a default value, you do not need to strictly put the setting in your copy of the configuration, but should review it nonetheless to make sure it matches your deployment expectations.

The configuration settings listed below are tagged as Optional in the CAS configuration metadata. This flag indicates that the presence of the setting is not immediately necessary in the end-user CAS configuration, because a default value is assigned or the activation of the feature is not conditionally controlled by the setting value. In other words, you should only include this field in your configuration if you need to modify the default value or if you need to turn on the feature controlled by the setting.

The configuration settings listed below are tagged as Third Party in the CAS configuration metadata. This flag indicates that the configuration setting is not controlled, owned or managed by the CAS ecosystem, and affects functionality that is offered by a third-party library, such as Spring Boot or Spring Cloud to CAS. For additional info, you might have to visit the third-party source to find more details.

  • spring.cloud.stream.kafka.binder.authorization-exception-retry-interval=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.auto-add-partitions=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.auto-alter-topics=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.auto-create-topics=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.brokers=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.certificate-store-directory=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.configuration=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.consider-down-when-any-partition-has-no-leader=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.consumer-properties=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.enable-observation=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.header-mapper-bean-name=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.headers=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.health-indicator-consumer-group=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.health-timeout=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.jaas=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.metrics.default-offset-lag-metrics-enabled=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$Metrics.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.metrics.offset-lag-metrics-interval=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$Metrics.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.min-partition-count=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.producer-properties=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.replication-factor=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.required-acks=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.batch-timeout=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.buffer-size=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.compression-type=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.configuration=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.error-channel-enabled=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.header-mode=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.header-patterns=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.message-key-expression=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.partition-count=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.partition-key-expression=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.partition-key-extractor-name=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.partition-selector-expression=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.partition-selector-name=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.required-groups=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.sync=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.topic=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.producer.use-native-encoding=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$CombinedProducerProperties.

    How can I configure this property?

  • spring.cloud.stream.kafka.binder.transaction.transaction-id-prefix=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaBinderConfigurationProperties$Transaction.

    How can I configure this property?

  • spring.cloud.stream.kafka.bindings=
  • org.springframework.cloud.stream.binder.kafka.properties.KafkaExtendedBindingProperties.

    How can I configure this property?

    Configuration Metadata

    The collection of configuration properties listed in this section are automatically generated from the CAS source and components that contain the actual field definitions, types, descriptions, modules, etc. This metadata may not always be 100% accurate, or could be lacking details and sufficient explanations.

    Be Selective

    This section is meant as a guide only. Do NOT copy/paste the entire collection of settings into your CAS configuration; rather pick only the properties that you need. Do NOT enable settings unless you are certain of their purpose and do NOT copy settings into your configuration only to keep them as reference. All these ideas lead to upgrade headaches, maintenance nightmares and premature aging.

    YAGNI

    Note that for nearly ALL use cases, declaring and configuring properties listed here is sufficient. You should NOT have to explicitly massage a CAS XML/Java/etc configuration file to design an authentication handler, create attribute release policies, etc. CAS at runtime will auto-configure all required changes for you. If you are unsure about the meaning of a given CAS setting, do NOT turn it on without hesitation. Review the codebase or better yet, ask questions to clarify the intended behavior.

    Naming Convention

    Property names can be specified in very relaxed terms. For instance cas.someProperty, cas.some-property, cas.some_property are all valid names. While all forms are accepted by CAS, there are certain components (in CAS and other frameworks used) whose activation at runtime is conditional on a property value, where this property is required to have been specified in CAS configuration using kebab case. This is both true for properties that are owned by CAS as well as those that might be presented to the system via an external library or framework such as Spring Boot, etc.

    :information_source: Note

    When possible, properties should be stored in lower-case kebab format, such as cas.property-name=value. The only possible exception to this rule is when naming actuator endpoints; The name of the actuator endpoints (i.e. ssoSessions) MUST remain in camelCase mode.

    Settings and properties that are controlled by the CAS platform directly always begin with the prefix cas. All other settings are controlled and provided to CAS via other underlying frameworks and may have their own schemas and syntax. BE CAREFUL with the distinction. Unrecognized properties are rejected by CAS and/or frameworks upon which CAS depends. This means if you somehow misspell a property definition or fail to adhere to the dot-notation syntax and such, your setting is entirely refused by CAS and likely the feature it controls will never be activated in the way you intend.

    Validation

    Configuration properties are automatically validated on CAS startup to report issues with configuration binding, specially if defined CAS settings cannot be recognized or validated by the configuration schema. Additional validation processes are also handled via Configuration Metadata and property migrations applied automatically on startup by Spring Boot and family.

    Indexed Settings

    CAS settings able to accept multiple values are typically documented with an index, such as cas.some.setting[0]=value. The index [0] is meant to be incremented by the adopter to allow for distinct multiple configuration blocks.

    Troubleshooting

    To enable additional logging, modify the logging configuration file to add the following:

    1
    2
    3
    4
    
    <Logger name="org.springframework.kafka" level="debug" additivity="false">
        <AppenderRef ref="casConsole"/>
        <AppenderRef ref="casFile"/>
    </Logger>