Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022
0 likes883 views
This document discusses building interactive queries in Kafka Streams. It provides background on Kafka Streams and state management. It then covers the requirements and steps to build a query service, including implementing it with Spring Boot. It describes executing queries by finding the correct host and querying locally or remotely. Finally, it discusses options for query types and results and displaying results in an index page.
![@bbejeck
Set “this” HostInfo
@Value("${application.server}")
private String applicationServer;
. . . .
@PostConstruct
public void init() {
String[] parts = applicationServer.split(":");
thisHostInfo = new HostInfo(parts[0], Integer.parseInt(parts[1]));
}
42](https://image.slidesharecdn.com/billbejeckbr0121005final-221021184411-665ce9ee/85/Building-an-Interactive-Query-Service-in-Kafka-Streams-With-Bill-Bejeck-Current-2022-42-320.jpg)
Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022
- 1. Building Interactive Queries In Kafka Streams Bill Bejeck @bbejeck
- 2. @bbejeck Nice to meet you! • Member of the DevX team • Prior to DevX ~3 years as engineer on Kafka Streams team • Apache Kafka® Committer and PMC member • Author of “Kafka Streams in Action” - 2nd edition underway! 2
- 3. @bbejeck Agenda • Kafka Streams Background • Interactive Queries • Interactive Query Requirements • Building the Query Service • Executing Queries 3
- 6. @bbejeck Kafka Streams – DSL API StreamsBuilder builder = new StreamsBuilder(); KStream<String, String > stockStr = builder.stream(”stocks”); stockStr.groupByKey() .windowedBy(TimeWindows.of(Duration.ofMinutes(5)) .aggregate(()-> new TradeStats(), (k, v, tradeStats) -> tradeStats.add(v), Materialized.as(“trade-stats-store”)) .toStream().mapValues(..) .to(“output”); 6
- 11. State Management
- 13. @bbejeck State Management & Task Assignment 13
- 14. @bbejeck State Management & Task Assignment 14
- 15. @bbejeck Fault Tolerance with State 15
- 17. @bbejeck Failover with Standby Tasks 17
- 27. @bbejeck Interactive Query Requirements Naming the store makes it available for Interactive Queries Materialized.as(“trade-stats-store”)) 27
- 28. @bbejeck Interactive Query Requirements Set the application.server configuration 28
- 30. Implementing The Query Service
- 31. @bbejeck Implementing The Query Service 1. Inversion of Control container 2. Uses Dependency Injection to implement IoC 3. Inject dependencies without implementation specifics 4. Autowiring achieved with annotations 31
- 32. @bbejeck Implementing The Query Service Spring Boot 1. Provides the Spring container 2. Embeds a web-server 3. Quickly set up a stand-alone application 32
- 33. @bbejeck Implementing The Query Service 1. Build the Topology 2. Handle Kafka Streams Lifecycle 3. Expose REST endpoint / Handle Queries 4. A view for rendering query results 33
- 34. @bbejeck Build the Topology @Component public class AggregationStream { private final KafkaStreamsAppConfiguration streamsConfiguration; @Autowired public AggregationStream(KafkaStreamsAppConfiguration streamsConfiguration) { this.streamsConfiguration = streamsConfiguration; } 34
- 35. @bbejeck Build the Topology public Topology topology() { StreamsBuilder builder = new StreamsBuilder(); KStream<String, String> input = builder.stream(streamsConfiguration.inputTopic(), Consumed.with(stringSerde, stringSerde) //Details omitted return builder.build() } 35
- 36. @bbejeck KafkaStreams Instance @Component public class KafkaStreamsContainer { private final KafkaStreamsAppConfiguration appConfiguration; private final AggregationStream aggregationStream; private KafkaStreams kafkaStreams @Autowired public KafkaStreamsContainer(final AggregationStream aggregationStream, final KafkaStreamsAppConfiguration appConfiguration) { } //Details omitted 36
- 37. @bbejeck KafkaStreams Instance @Component public class KafkaStreamsContainer { @Bean public KafkaStreams kafkaStreams() { return kafkaStreams; } } 37
- 38. @bbejeck Start Kafka Streams @PostConstruct public void init() { Properties properties = appConfiguration.streamsConfigs(); Topology topology = aggregationStream.topology(); kafkaStreams = new KafkaStreams(topology, properties); kafkaStreams.cleanUp(); //Optional – not a production setting kafkaStreams.start(); } 38
- 39. @bbejeck Stop Kafka Streams @PreDestroy public void tearDown(){ kafkaStreams.close(Duration.ofSeconds(10)); } 39
- 40. @bbejeck Expose REST endpoint @RestController @RequestMapping("/streams-iq”) public class StockController { private final KafkaStreams kafkaStreams; private final RestTemplate restTemplate } 40
- 41. @bbejeck Inject KafkaStreams Object @Autowired public StockController(final KafkaStreams kafkaStreams, final RestTemplateBuilder restTemplateBuilder) { this.kafkaStreams = kafkaStreams; this.restTemplate = restTemplateBuilder.build() } 41
- 42. @bbejeck Set “this” HostInfo @Value("${application.server}") private String applicationServer; . . . . @PostConstruct public void init() { String[] parts = applicationServer.split(":"); thisHostInfo = new HostInfo(parts[0], Integer.parseInt(parts[1])); } 42
- 44. @bbejeck Handle Incoming Requests private static final String IQ_URL = "http://{host}:{port}/streams-iq{path}"; @GetMapping(value = "/keyquery/{symbol}") public QueryResponse<V> getAggregation(@PathVariable String symbol) { //Execute the query } 44
- 45. @bbejeck Handle Incoming Requests private static final String IQ_URL = "http://{host}:{port}/streams-iq{path}"; @GetMapping(value = "/range") public QueryResponse<List<V>> getAggregationRange( @RequestParam(required = false) String lower, @RequestParam(required = false) String upper) { //Execute the query 45
- 47. @bbejeck Executing Queries Two step process 1. Find the correct host * 2. Query the current instance or forward to the correct one 47
- 48. @bbejeck Executing Queries KeyQueryMetadata = KafkaStreams.queryMetadataForKey(StoreName, <Key>, <Key Serializer>) 48
- 49. @bbejeck Executing Queries HostInfo = KeyQueryMetadata.activeHost() Set<HostInfo> = KeyQueryMetadata.standByHosts() int partition = KeyQueryMetadata.partition(); 49
- 50. @bbejeck Executing Queries if (keyMetadata.activeHost().equals(thisHostInfo)) { //query current Kafka Streams instance } 50
- 51. @bbejeck Executing Queries } else { String host = keyMetadata.activeHost().host(); int port = keyMetadata.activeHost().port(); String path = "/keyquery/” + symbol; 51
- 52. @bbejeck Executing Queries } else { String host = keyMetadata.activeHost().host(); int port = keyMetadata.activeHost().port(); String path = String.format("/keyquery/" + symbol; try { resp = restTemplate.getForObject(IQ_URL, QueryResponse.class, host, port, path); } catch (RestClientException exception) { //Handle error } 52
- 53. @bbejeck Executing Queries – Query All Hosts* 53
- 54. @bbejeck Executing Queries – Query All Hosts* Collection<StreamsMetadata> storeMetadata = KafkaStreams.allMetadataForStore(“store-name”); Collection<StreamsMetadata> allStreamsMetadata = KafkaStreams.metadataForAllStreamsClients();* 54
- 55. @bbejeck Executing Queries – Query All Hosts* @GetMapping(value = "/range") public QueryResponse<R> getAggregationRange( @RequestParam(required = false) String lower, @RequestParam(required = false) String upper) { Collection<StreamsMetadata> streamsMetadata = kafkaStreams.streamsMetadataForStore(storeName); streamsMetadata.forEach(streamsClient -> { // query each client } 55
- 56. @bbejeck Executing Queries – Query All Hosts* @GetMapping(value = "/internal/range") public QueryResponse<R> getAggregationRangeInternal( @RequestParam(required = false) String lower, @RequestParam(required = false) String upper, @RequestParam(required = false) List<> partitions return doRangeQuery(thisHostInfo, partitions, lower, upper); } 56
- 58. @bbejeck Executing Queries StateQueryResult<R> result = KafkaStreams.query( StateQueryRequest.inStore(StoreName) .withQuery(Query<R> instance) ); 58
- 59. @bbejeck Executing Queries – Query Types KeyQuery RangeQuery * WindowKeyQuery WindowRangeQuery * 59
- 60. @bbejeck Executing Queries – Options // Default is active or standby StateQueryRequest.requireActive(); //Default is all partitions StateQueryRequest.withPartitions(Set<Integer>partitions); 60
- 61. @bbejeck Executing Queries – Options StateQueryRequest.withPositionBound(PositionBound pb); StateQueryRequest.enableExecutionInfo(); 61
- 62. @bbejeck Executing Queries – Results Map<Integer QueryResult<R>> result = StateQueryResult.getPartitionResults(); //other methods StateQueryResult.getGlobalResult(); StateQueryResult.getOnlyPartitionResult(); 62
- 64. @bbejeck Index Page Location src main resources public index.html js and css files 64
- 65. @bbejeck Single Page Application <script type="text/javascript"> function runIqTables() { getRangeTableData("http://localhost:7076/streams-iq/range") updateKeyTableWithList("http://localhost:7076/streams-iq/keyquery/"); } function getRangeTableData(path) { $.getJSON(path, function (response) { . . . }); } .... setInterval(runIqTables, 5000); </script> 65
- 66. @bbejeck Summary • Interactive Queries allow to you view the state of a running Kafka Streams application • You can leverage standbys for higher availability with IQ • Use Interactive Queries v2 • By leveraging Spring Boot and creating an @RestController quickly get a Kafka Streams application with IQ enabled 66
- 68. @bbejeck Resources • Source Code - https://github.com/bbejeck/KafkaStreamsInteractiveQueries • Kafka Tutorials - https://kafka-tutorials.confluent.io/ • Confluent Developer - developer.confluent.io • Spring Boot - https://spring.io/projects/spring-boot/ • Spring Kafka - https://spring.io/projects/spring-kafka • Documentation - https://docs.confluent.io/current/streams 68
- 69. Your Apache Kafka® journey begins here developer.confluent.io 69