AIP-82: Add common messaging system providers

[jason810496]

Body

related: #49938

Why

Several users have pointed out that the Event-Driven Scheduling feature introduced in Airflow 3 currently supports only Amazon SQS.

So I add Kafka common message queue provider in #49938, but there are still quite a few common providers that should be useful for airflow users to implement Event-Driven workflows.

What

Here are some common messaging systems:

• Amazon Kinesis Add Kinesis Data Streams trigger #71135
• Amazon MSK Add MskHook #69000
• Amazon SQS
• Apache Kafka
• Azure Message Bus
  • Trigger: Add Azure Service Bus Queue and Subscription triggers for async message processing #53356
  • CommonMessageQueue Interface
• Google PubSub AIP-82: implement Google Pub/Sub message queue provider #54494
• Redis Add AwaitMessageTrigger for Redis PubSub #52917

reference for implementation: https://github.com/apache/airflow/pull/49938/files

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[400Ping]

Hi, may I try the pubsub one?

[Jasperora]

Hi, I would like to try the Redis one. Thanks!

[prdai]

hi, could I implement the "Azure Message Bus" one? thanks :)