Beam
Unified programming model for defining batch and stream data processing pipelines that can run portably across multiple execution engines.
- Actively maintained
- Foundation-backed
- Well documented
- Permissive license
Overview
Apache Beam lets a team write a data pipeline once using its unified batch/stream API, then run that same pipeline on different underlying execution engines - including Flink and Spark (also in this catalog) - without rewriting it for each. This avoids locking a pipeline's logic to one specific processing engine.
Beam is a fit for teams that want pipeline portability across execution engines, or who are building on Google Cloud Dataflow (which uses Beam as its programming model), a different layer than Flink or Spark themselves, which are the engines Beam pipelines actually run on.
Why it's in this catalog
Catalog rating: ★★★☆☆ (3/5)
Unified batch/stream processing model, a distinct approach within analytics.
- Mature
- Foundation-governed
- Categories
- Analytics
- Languages
- Java, Python, Go
- License
- Apache-2.0
Related Projects
Spark
Apache-2.0Unified analytics engine for large-scale batch and stream data processing, one of the most widely adopted big data frameworks.
Actively maintained
- Analytics
Flink
Apache-2.0Distributed stream processing engine for computing over continuous, unbounded data streams with exactly-once correctness guarantees.
Actively maintained
- Analytics
Trino
Apache-2.0Distributed SQL query engine that lets you run federated queries across multiple data sources - databases, data lakes, and more - from one place.
Actively maintained
- Analytics
ClickHouse
Apache-2.0Column-oriented database built for real-time analytical queries over massive datasets, generating aggregated reports in milliseconds rather than seconds or minutes.
Actively maintained
- Analytics
Spotted an error? Suggest an edit on GitHub.