Spark
Unified analytics engine for large-scale batch and stream data processing, one of the most widely adopted big data frameworks.
- Actively maintained
- Foundation-backed
- Well documented
- Permissive license
Overview
Apache Spark processes very large datasets across a distributed cluster of machines, supporting batch processing, streaming, SQL queries, and machine learning workloads through one unified engine and API, in multiple languages including Python, Scala, and Java. It's been the standard big-data processing framework for over a decade.
Spark is a fit for teams running large-scale data transformation, ETL, or machine learning pipelines across distributed clusters, a broader and more general-purpose engine than the more specialized Flink (real-time streaming) or Trino (federated SQL) also in this catalog.
Why it's in this catalog
Catalog rating: ★★★★★ (5/5)
Foundational big-data processing engine, essential representative project for analytics.
- Foundational
- Foundation-governed
- Categories
- Analytics
- Languages
- Scala, Python, Java
- License
- Apache-2.0
Related Projects
Beam
Apache-2.0Unified programming model for defining batch and stream data processing pipelines that can run portably across multiple execution engines.
Actively maintained
- Analytics
Flink
Apache-2.0Distributed stream processing engine for computing over continuous, unbounded data streams with exactly-once correctness guarantees.
Actively maintained
- Analytics
Trino
Apache-2.0Distributed SQL query engine that lets you run federated queries across multiple data sources - databases, data lakes, and more - from one place.
Actively maintained
- Analytics
ClickHouse
Apache-2.0Column-oriented database built for real-time analytical queries over massive datasets, generating aggregated reports in milliseconds rather than seconds or minutes.
Actively maintained
- Analytics
Spotted an error? Suggest an edit on GitHub.