Cloud Based Integration & Messaging. Data Processing & Streaming (aka Data Pipeline). Open Data Hub¶
- Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming
- Message Brokers
- Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions
- Debezium open source distributed platform for Change Data Capture (CDC) software design pattern
- Red Hat Integration service registry and Apicurio
- Data Mesh
- Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)
- Apache Kafka
- AWS Kinesis
- Banzai Cloud Supertubes (Cloud Native Kafka implementation)
- Confluent Cloud (Apache Kafka Re-engineered for the Cloud)
- Redpanda (kafka alternative). A modern streaming platform for mission critical workloads
- Apache Pulsar
- Apache Flink
- Hazelcast JET
- Workflow Engines
- Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)
- Open Data Hub AI-as-a-Service (AIaaS) platform
- KEDA. Kubernetes Event Driven Autoscaling
- Platforms for collecting, storing and routing customer event data
- Questions and Answers
Message Queue in Kubernetes. Event-driven Messaging. Real-Time Data Streaming¶
- Wikipedia: Message Broker
- Wikipedia: Event-driven messaging
- Wikipedia: Streaming Data
- nginx.com: Event-Driven Data Management for Microservices 🌟
- dzone: Event-Driven Architecture as a Strategy Event-driven architecture provides five key benefits to modern application architecture: scalability, resilience, agility, data sharing, and cloud enabling.
- infoq.com: From Monolith to Event-Driven: Finding Seams in Your Future Architecture
- wikipedia: Enterprise service bus
- thenewstack.io: The Rise of the Event Streaming Database 🌟
- cncf.io: The need for Kubernetes Native Messaging Platform in Hybrid Cloud Environment
- wiprodigital.com: A Guide to Enterprise Event-Driven Architecture
- medium: Introduction to Event-Driven Architecture The essential concepts that every developer should know
- ibm.com: Event-driven cloud-native applications (microservices) The event backbone is being part of the microservices mesh, providing the publish-and-subscribe communication between microservices and enabling the support of loosely coupled event-driven microservices.
- stackoverflow.blog: How event-driven architecture solves modern web app problems 🌟 In this article, we’ll discuss some of the problems driving innovation in modern web development. Then we’ll dive into the basics of event-driven architecture (EDA), which tries to address these problems by thinking about back-end architecture in a novel way.
- sebalopezz.medium.com: Monolith to Microservices + Event-Driven Architecture 🌟
- confluent.io: Event-Driven Microservices Architecture (white paper) 🌟 Microservices are an architectural pattern that structures an application as a collection of small, loosely coupled services that operate together to achieve a common goal. Because they work independently, they can be added, removed, or upgraded without interfering with other applications. While there are numerous benefits to microservices architecture, like easier deployment and testing, improved productivity, flexibility, and scalability, they also pose a few disadvantages, as independently run microservices require a seamless method of communication to operate as one larger application. Event-driven microservices allow for real-time microservices communication, enabling data to be consumed in the form of events before they’re even requested. In this white paper, we’ll cover how event-driven microservices work, presenting a sample currency exchange platform to illustrate the design and architecture of an application composed of event-driven microservices using Apache Kafka® and Confluent Platform. We also discuss other aspects of microservices architectures, such as team structure, continuous delivery, deployment, and testing. Lastly, we discuss how Apache Kafka and Confluent Platform enable and extend core principles of microservices, including decoupling, separation of concerns, agility, and real-time streaming of event data.
- Apache ActiveMQ
- Dzone: Introduction to Message Brokers. Part 1: Apache Kafka vs. RabbitMQ
- Dzone: Introduction to Message Brokers. Part 2: ActiveMQ vs. Redis Pub/Sub
- developers.redhat.com: Choosing the right asynchronous-messaging infrastructure for the job
ActiveMQ message broker¶
- ActiveMQ 5.x “classic”
- ActiveMQ Artemis Apache ActiveMQ is a subproject of Apache ActiveMQ. It has been donated to the Apache Software Foundation in 2015. There were lots of changes in project names in the past. The Artemis project first started as JBoss Messaging and got renamed to HornetQ in August 2009.
- Apache Artemis JMeter Running the ActiveMQ Artemis JMeter Performance Testing Examples.
RabbitMQ message broker¶
- K8s prevent queue worker Pod from being killed during deployment How to prevent a Kubernetes (like RabbitMQ) queue worker Pod from being killed during deployment while handling a message?
- medium.com: RabbitMQ vs. Kafka An architect’s dilemma
Redis message broker¶
Apache Camel message broker¶
- Apache Camel
- Quora.com: What’s the difference between Apache Camel and Kafka?
- dzone: Hybrid multi-cloud event mesh architectural design Building the event mesh with Camel
Apache Camel K¶
- Apache Camel K is a lightweight cloud-integration platform that runs natively on Kubernetes. Based on the famous Apache Camel, Camel K is designed and optimized for serverless and microservices architectures.
- developers.redhat.com: Six reasons to love Camel K
- developers.redhat.com: Extending Kafka connectivity with Apache Camel Kafka connectors
- developers.redhat.com: Design event-driven integrations with Kamelets and Camel K
KubeMQ message broker¶
- KubeMQ.io: Kubernetes Native Message Queue Broker
- devops.com: Best of 2019: Implementing Message Queue in Kubernetes
- kubemq.io: Kafka VS KubeMQ 🌟
Google Cloud Platform Pub/Sub¶
- Google Cloud Platform Pub/Sub
- A generic framework of concurrent consumers for Google Cloud Platform Pub/Sub 🌟 An example shows how to publish messages to Pub/Sub and build a service to consume the messages concurrently using the Python multiprocessing module
Cloud Based Integration. Integration Platform-as-a-Service (iPaaS) solutions¶
- Wikipedia: Cloud Based Integration (iPaaS)
- Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
- Integration platform as a service (iPaaS) is a set of automated tools for connecting software applications that are deployed in different environments. iPaaS is often used by large business-to-business (B2B) enterprises that need to integrate on-premises applications and data with cloud applications and data.
- blog.axway.com: What is iPaaS?
- ibm.com: iPaaS (Integration-Platform-as-a-Service): iPaaS is a cloud-based solution that simplifies application integration across on-premises and cloud environments, to help you accelerate innovation and lower your integration and operations costs.
Red Hat Fuse and Red Hat Fuse Online¶
Syndesis open source integration platform¶
- Syndesis open source integration platform (OpenSource Project for Red Hat Fuse Online)
- developers.redhat.com: Low-code microservices orchestration with Syndesis
Debezium open source distributed platform for Change Data Capture (CDC) software design pattern¶
- Change Data Capture, or CDC, is a well-established software design pattern for a system that monitors and captures the changes in data so that other software can respond to those changes. CDC captures row-level changes to database tables and passes corresponding change events to a data streaming bus. Applications can read these change event streams and access these change events in the order in which they occurred.
- Debezium: Stream changes from your database
- developers.redhat.com: Decoupling microservices with Apache Camel and Debezium
- A good explanation of how to avoid distributed transactions using outbox pattern: Transaction Log Tailing With Debezium
- developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
- developers.redhat.com: Change data capture for microservices without writing any code
- debezium.io: Lessons Learned from Running Debezium with PostgreSQL on Amazon RDS
- info.crunchydata.com: PostgreSQL Change Data Capture With Debezium
- medium.com: Stream Your Database into Kafka with Debezium An Introduction and Experience Report. Insightful post by David Hettler of comsysto about their usage of Debezium, touching on many details like outbox pattern, Avro schemas, Postgres on RDS etc.
- noti.st: Change Data Capture with Flink SQL and Debezium 🌟
- vladmihalcea.com: A beginner’s guide to CDC (Change Data Capture)
- shopify.engineering: Capturing Every Change From Shopify’s Sharded Monolith
- developers.redhat.com: Db2 and Oracle connectors coming to Debezium 1.4 GA
Red Hat Integration service registry and Apicurio¶
- Red Hat Integration service registry
- Apicurio Registry An API/Schema registry - stores APIs and Schemas.
- Event streaming and data federation: A citizen integrator’s story
- redhat.com: Using a schema registry to ensure data consistency between microservices Make interservice communication easier by using a schema registry.
- martinfowler.com: Data Mesh Principles and Logical Architecture
- infoq.com: Data Mesh Principles and Logical Architecture Defined
- martinfowler.com: How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
Data Processing (aka Streaming Data, Data Pipeline or Big Data Pipeline)¶
- Awesome Streaming A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources.
- cloudblog.withgoogle.com: Turn any Dataflow pipeline into a reusable template
- thenewstack.io: Part 1: The Evolution of Data Pipeline Architecture
- eng.uber.com: Uber’s Journey Toward Better Data Culture From First Principles
- satishchandragupta.com: Scalable Efficient Big Data Pipeline Architecture
- openshift.com: How to Orchestrate Data Pipelines with Applications Deployed on OpenShift
- Apache Kafka
- developers.redhat.com: Using secrets in Kafka Connect configuration
- developers.redhat.com: Capture database changes with Debezium Apache Kafka connectors
- Awesome Kafka
- Single Message Transformations - The Swiss Army Knife of Kafka Connect
- medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake Replacing Apache Airflow with Debezium.
- medium: Apache Kafka Startup Guide: System Design Architectures: Notification System, Web Activity Tracker, ELT Pipeline, Storage System 🌟
- medium: Getting Started With Kafka on OpenShift
- containerjournal.com: Red Hat Platform Brings Kafka Closer to Kubernetes
- lightbend.com: Monitor Kafka Consumer Group Latency with Kafka Lag Exporter
- AKHQ (previously known as KafkaHQ) 🌟 Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more…
- banzaicloud.com: Kafka Schema Registry on Kubernetes the declarative way
- Build a simple cloud-native change data capture pipeline
- banzaicloud.com: Bulletproof Kafka, and the tale of an Amazon outage=
- confluent.fr: Infrastructure Modernization with Google Anthos and Apache Kafka
- confluent.io: Apache Kafka DevOps with Kubernetes and GitOps
- Build a data streaming pipeline using Kafka Streams and Quarkus
- levelup.gitconnected.com: Kafka for Engineers 🌟 Here are things about Kafka that you need to understand as a software engineer.
- confluent.io: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
- banzaicloud.com: Kafka on Kubernetes - using etcd 🌟
- softwareengineeringdaily.com: Kafka Applications with Tim Berglund (podcast) 🌟
- medium: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
- infoq.com: Building a SQL Database Audit System using Kafka, MongoDB and Maxwell’s Daemon
- tecmint: How to Install Apache Kafka in CentOS/RHEL 7
- medium: Processing guarantees in Kafka “Duplicates and lost messages are due not only to features of the messaging systems, but in the design of producer and consumer applications as well.” One of the best posts on processing guarantees in kafka.
- davidxiang.com: Kafka As A Database? Yes Or No
- medium: How Pinterest runs Kafka at scale
- medium: Google Pub/Sub Lite for Kafka Users
- medium: 4 Microservices Caching Patterns at Wix
- Confluent.io: Intro to Apache Kafka: How Kafka Works 🌟
- levelup.gitconnected.com: Kafka for Engineers
- medium: Microservices in Rust with Kafka
- medium: Apache Kafka in a Nutshell 🌟 Architecture, Use Cases, and a Getting Started guide — rolled into one
- confluent.io: Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry
- kai-waehner.de: App Modernization and Hybrid Cloud Architectures with Apache Kafka
- kai-waehner.de: Apache Kafka and MQTT (Part 1 of 5) – Overview and Comparison
- medium: Solutions to Communication Problems in Microservices using Apache Kafka and Kafka Lens
- kafka-tutorials.confluent.io 🌟
- kafka-tutorials.confluent.io: How to join a stream and a lookup table 🌟 If I have events in a Kafka topic and a table of reference data (aka a lookup table), how can I join each event in the stream to a piece of data in the table based on a common key?
- confluent.io: DevOps for Apache Kafka with Kubernetes and GitOps 🌟
- dzone.com: Microservices, Event-Driven Architecture and Kafka 🌟
- medium: Understanding Kafka Topic Partitions Everything in Kafka is modeled around partitions. They rule Kafka’s storage, scalability, replication, and message movement.
- kafka-tutorials.confluent.io: How to count messages in a Kafka topic
- confluent.io: Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper 🌟
- piotrminkowski.com: Knative Eventing with Kafka and Quarkus
- blog.cloudera.com: Scalability of Kafka Messaging using Consumer Groups
- thenewstack.io: Beyond the Quickstart: Running Apache Kafka as a Service on Kubernetes
- towardsdatascience.com: You Can Replace Kafka with a Database
- Handling Retries in Kafka: If You’re Using Kafka With Your Microservices, You’re Probably Handling Retries Wrong
- Kafdrop – Kafka Web UI 🌟
- confluent.io: What’s New in Apache Kafka 2.8
- devclass.com: Apache Kafka 2.8.0 previews life without ZooKeeper
- KLoadGen - Kafka + (Avro/Json Schema) Load Generator 🌟 KLoadGen is kafka load generator plugin for jmeter designed to work with AVRO and JSON schema. It allows sending kafka messages with a structure defined as an AVRO Schema or a Json Schema. It connects to the Scheme Registry Server, retrieve the subject to send and generate a random message every time.
- instaclustr.com: Apache Kafka Architecture: A Complete Guide 🌟
- youtube playlist: Kafka Connect Tutorials | Kafka Connect 101: REST API 🌟 KafkaConnect uses a REST API to expose its management capabilities. tlberglund demonstrates many of the key functions available using the REST API, including creating connectors, viewing their status, and accessing troubleshooting information.
- developers.redhat.com: Event-driven APIs and schema governance for Apache Kafka: Get ready for Kafka Summit Europe 2021
- developers.redhat.com: Building resilient event-driven architectures with Apache Kafka
- tech.ebayinc.com: Resiliency and Disaster Recovery with Kafka
Strimzi kubernetes operator for apache kafka¶
- developers.redhat.com: how easy to deploy and configure a Kafka Connect on Kubernetes through strimziio operator and use secrets
- developers.redhat.com: Introduction to Strimzi: Apache Kafka on Kubernetes (KubeCon Europe 2020) 🌟
- strimzi.io: Optimizing Kafka producers
- strimzi.io: Optimizing Kafka consumers 🌟
- strimzi.io: Optimizing Kafka producers 🌟
- pepy.tech/project/strimzi-kafka-cli 🌟 - pypi.org/project/strimzi-kafka-cli
- mqtt.org MQTT: The Standard for IoT Messaging
- developers.redhat.com: Deploying the Mosquitto MQTT message broker on Red Hat OpenShift, Part 1
Banzai Cloud Supertubes (Cloud Native Kafka implementation)¶
Confluent Cloud (Apache Kafka Re-engineered for the Cloud)¶
- confluent.io The Complete Event Streaming Platform for Apache Kafka.
- Focus on building apps and not managing clusters with a scalable, resilient and secure event streaming platform. Event streaming with Kafka made simple on AWS, Azure and GCP clouds.
- mongodb.com: DaaS with MongoDB and Confluent
- confluent.io: Confluent and Microsoft Announce Strategic Alliance
- confluent.io: Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana
Redpanda (kafka alternative). A modern streaming platform for mission critical workloads¶
- Redpanda 🌟 is a Kafka® compatible event streaming platform. No Zookeeper, no JVM, and no code changes required. Use all your favorite open source tooling - 10x faster.
- Redpanda is now Free & Source Available
- softwareengineeringdaily.com: Redpanda: Kafka Alternative with Alexander Gallego 🌟
- ksqlDB The event streaming database purpose-built for stream processing applications.
- Kafka Streams and ksqlDB Compared – How to Choose
- Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation
- Pulsar vs Kafka – Comparison and Myths Explored
- Apache Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
- How to set up Apache Flink on Kubernetes for real time data processing
- flink.apache.org: How to natively deploy Flink on Kubernetes with High-Availability (HA)
- Hazelcast JET Open-Source Distributed Stream Processing
- devops.com: Hazelcast Simplifies Streaming for Extremely Fast Event Processing in IoT, Edge and Cloud Environments
- infoq.com: Event Streams and Workflow Engines – Kafka and Zeebe 🌟
- Zeebe workflow engine
- Orchestration Made Easy with Zeebe and Kafka
- towardsdatascience.com: A journey to Airflow on Kubernetes
- dzone: Apache Airflow Architecture on OpenShift
- redhat.com: Monitoring Apache Airflow using Prometheus
- towardsdatascience.com: Apache Airflow for containerized data-pipelines Are you having problems running tasks with a different version of Python on Airflow? In this article, I explain how to solve this issue.
- Couler Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
Red Hat AMQ (ActiveMQ Artemis broker and Apache Kafka)¶
- Red Hat AMQ overview
- Red Hat AMQ = AMQ Broker (Apache ActiveMQ Artemis) + AMQ Streams (Apache Kafka)
Red Hat AMQ Broker (ActiveMQ Artemis)¶
- Apache ActiveMQ Artemis broker
- developers.redhat.com: JDBC Master-Slave Persistence setup with Activemq using Postgresql database
- developers.redhat.com: Connecting external clients to Red Hat AMQ Broker on Red Hat OpenShift
Red Hat AMQ Streams¶
- Understanding Red Hat AMQ Streams components for OpenShift and Kubernetes 🌟
- Red Hat AMQ streams (kafka): Simplify Apache Kafka on Red Hat OpenShift
- Set up Red Hat AMQ Streams custom certificates on OpenShift
- speakerdeck.com: Apache Kafka with Red Hat AMQ Streams 🌟
- HTTP-based Kafka messaging with Red Hat AMQ Streams
- blog.jromanmartin.io: How to upgrade Strimzi Operator using the CLI
|Product||Also Known As||Components||URL|
|Red Hat AMQ 6||JBoss AMQ 6||Apache ActiveMQ||Ref|
|Red Hat AMQ 7||JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite||AMQ Broker + AMQ Streams||Ref|
|Red Hat AMQ 7||JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite||JBoss AMQ 7 (Broker) + Apache Kafka||Ref|
|Red Hat AMQ 7||JBoss AMQ 7 (Broker) or Red Hat AMQ 7 Suite||Apache ActiveMQ Artemis + Apache Kafka||Ref|
Open Data Hub AI-as-a-Service (AIaaS) platform¶
- Open Data Hub
- Open Data Hub 0.6 brings component updates and Kubeflow architecture
- A development roadmap for Open Data Hub
KEDA. Kubernetes Event Driven Autoscaling¶
- KEDA Kubernetes Event-driven Autoscaling. Application autoscaling made simple.
- Dzone: Autoscaling Your Kubernetes Microservice with KEDA Introduction to KEDA—event-driven autoscaler for Kubernetes, Apache Camel, and ActiveMQ Artemis—and how to use it to scale a Java microservice on Kubernetes.
- tomd.xyz: Event-driven integration on Kubernetes with Camel & KEDA 🌟 Can we develop apps in Kubernetes that autoscale based on events? Perhaps, with this example using KEDA, ActiveMQ and Apache Camel.
Platforms for collecting, storing and routing customer event data¶
- Service meshes to the rescue: Load balancing and scaling long-lived connections in Kubernetes 🌟 Kubernetes doesn’t load balance long-lived connections, some Pods might receive more requests than others, In case you are using HTTP/2, gRPC, RSockets, AMQP. Any work around?
Questions and Answers¶
- adambien.blog - 75th airhacks.tv Questions and Answers: Kafka, JAX-RS, MicroProfile, JSON-B, GSON, JWT, VSC, NetBeans, Java Fullstack “Kafka vs. JAX-RS / RPC, thoughts about APIs, JSON-B vs. GSON, Path.of over Paths.get, Java Records, MicroProfile JWT, beginners vs. expert content, best Java fullstack, code coverage, NetBeans in 2020, Visual Studio Setup for Java, screencast configuration, ReactJS / Angular over JSF?, JSON-P vs. JSON-B, security code scanning”
Pub-Sub ≠ Partitioning ≠ Multiplexing pic.twitter.com/0ZVaH9Mxvr— Clemens Vasters 🇪🇺☁📨 (@clemensv) July 28, 2020