Search open source projects

Distributed data integration platform for moving and transforming data between databases, data lakes, and messaging systems at scale.

• Backend

Declarative, YAML-based orchestration platform for scheduling and orchestrating data pipelines, ETL jobs, and business workflows.

• Automation

Python-native workflow orchestration platform for building, scheduling, and monitoring data pipelines with ordinary Python code.

• Automation

Unified programming model for defining batch and stream data processing pipelines that can run portably across multiple execution engines.

• Analytics

High-performance observability data pipeline for collecting, transforming, and routing logs, metrics, and traces.

• Monitoring

Document parsing library that converts PDFs, Word docs, and other formats into clean, structured data ready for LLM and RAG pipelines.

• AI & Machine Learning

Distributed event streaming platform that stores data as a durable, replayable log, built for high-throughput real-time pipelines rather than traditional point-to-point queuing.

• Backend

Real-time, per-second monitoring dashboard that auto-discovers metrics across your infrastructure with zero configuration.

• Monitoring

Data orchestration layer that sits between compute engines and storage systems, caching data for faster, unified access across the two.

• Storage

Open-source backend platform providing authentication, databases, storage, functions and realtime APIs, self-hosted or as a managed cloud service.

• Backend
• Database
• Authentication & Security

Kubernetes-native workflow engine that runs each step of a pipeline as a container, orchestrating parallel jobs and multi-step DAGs directly on a cluster.

• Automation

Distributed wide-column NoSQL database designed for massive scale and zero single point of failure across multiple data centers.

• Database

Column-oriented database built for real-time analytical queries over massive datasets, generating aggregated reports in milliseconds rather than seconds or minutes.

• Analytics

Node.js and TypeScript library for web scraping and browser automation, built to produce reliable crawlers that resist bot detection.

• Developer Tools

Distributed filesystem and object storage system built for cloud-native, large-scale data-intensive applications.

• Storage

Open source data platform that wraps any SQL database with a dynamic API and an intuitive admin app.

• CMS
• Backend
• API

Distributed, ordered key-value store with strong ACID transaction guarantees, designed as a reliable storage substrate for other systems.

• Database

Time-series metrics storage and graphing system that has been a foundational building block of monitoring stacks for over a decade.

• Monitoring

Instant GraphQL and REST APIs on top of PostgreSQL and other databases, via Data Connectors.

• Backend
• API
• Database

Purpose-built time-series database for metrics, events, and real-time analytics at high write volumes.

• Monitoring

Library for getting reliable, structured data output from LLMs using function-calling APIs and Pydantic models for validation.

• AI & Machine Learning

Widely used open-source automation server with a large plugin ecosystem for building CI/CD pipelines and automating software delivery.

• DevOps

The standard web-based interactive computing environment for notebooks, code, and data, widely used across data science and research.

• Developer Tools

Machine learning platform for Kubernetes that bundles pipelines, model training, hyperparameter tuning, and serving into one toolkit.

• AI & Machine Learning

Data framework for connecting large language models to your own data - documents, databases, APIs - for retrieval-augmented applications.

• AI & Machine Learning

Horizontally scalable, distributed time-series database and metrics platform built at Uber for very large-scale monitoring.

• Monitoring

Open source vector database built for scalable similarity search and AI applications.

• AI & Machine Learning
• Database

Document-oriented NoSQL database designed for scalability and developer flexibility.

• Database

Native graph database that stores and queries data as nodes and relationships, purpose-built for connected data problems.

• Database

CNCF dashboarding tool for visualizing observability data, built around portable, version-controllable dashboard definitions.

• Monitoring

Powerful, open source object-relational database system with over 35 years of active development.

• Database

Vector database and similarity search engine for building AI applications with embeddings.

• AI & Machine Learning
• Database

In-memory data structure store used as a database, cache, message broker and streaming engine.

• Database
• Storage

Fast, high-level Python web scraping and crawling framework for extracting structured data from websites.

• Developer Tools

Event-based monitoring and observability pipeline that gives teams full control over how monitoring events are processed and routed.

• Monitoring

Unified analytics engine for large-scale batch and stream data processing, one of the most widely adopted big data frameworks.

• Analytics

Multi-cloud continuous delivery platform originally built at Netflix for deploying applications reliably across many cloud providers and regions.

• DevOps

Open source Firebase alternative providing a Postgres database, authentication, instant APIs, edge functions, realtime subscriptions and storage.

• Backend
• Database
• Authentication & Security

Kubernetes-native building blocks for creating CI/CD pipelines, giving teams a standard, portable way to define pipelines as Kubernetes resources.

• DevOps

MySQL-compatible distributed SQL database that scales horizontally while supporting both transactional and analytical workloads.

• Database

Distributed SQL query engine that lets you run federated queries across multiple data sources - databases, data lakes, and more - from one place.

• Analytics

In-memory data store and community-led fork of Redis, created after Redis's 2024 license change moved it away from an open-source license.

• Database

Simple, container-native CI/CD engine extensible via plugins, designed to run pipelines with a lightweight server and agent footprint.

• DevOps

Optimized, distributed gradient boosting library widely used for structured/tabular data problems, a staple of applied machine learning.

• AI & Machine Learning

Open source backend consisting of an embedded SQLite database, authentication, file storage and a realtime API in a single executable.

• Backend
• Database

CNCF specification and SDK collection for describing event data in a common, vendor-neutral format across systems and platforms.

• Backend

Kubernetes-native control plane that lets you provision and manage cloud infrastructure - databases, networks, storage - using Kubernetes APIs.

• DevOps

Real-time analytics database built for sub-second queries over continuously streaming and historical event data at scale.

• Analytics

Distributed search and analytics engine for all types of data, including text, numerical and geospatial.

• Search

High-performance proxy built for cloud-native applications, serving as the foundational data plane for many modern service meshes and API gateways.

• DevOps

Distributed stream processing engine for computing over continuous, unbounded data streams with exactly-once correctness guarantees.

• Analytics

CNCF-graduated log collector that unifies data collection from many sources into a single, structured logging layer.

• Monitoring

Distributed POSIX filesystem that stores data on object storage like S3, combining familiar filesystem semantics with cloud-scale storage.

• Storage

Developer-centric API testing tool that records real API and database traffic with eBPF and turns it into test cases and data mocks automatically, with no code changes.

• Testing

Open source web analytics platform giving full control and ownership of visitor data.

• Analytics

General-purpose, CNCF-graduated policy engine for enforcing authorization, compliance, and configuration rules as code across microservices, APIs, Kubernetes, and CI pipelines.

• DevOps

Vendor-neutral pipeline for collecting, processing, and exporting traces, metrics, and logs from applications to any observability backend.

• Monitoring

Standalone web server that turns any PostgreSQL database directly into a RESTful API.

• API
• Backend

Open source systems monitoring and alerting toolkit with a dimensional data model and powerful query language.

• Monitoring

Cloud-native search engine purpose-built for observability data, indexing logs and traces directly on object storage for cost-efficient search at scale.

• Search

Security testing orchestration platform that runs multiple open-source scanners against your applications and infrastructure from one pipeline.

• Testing

Backup, restore, and migration tool for Kubernetes clusters, protecting both cluster resources and persistent volume data against loss or letting you move workloads between clusters.

• Storage

Enterprise-grade infrastructure and network monitoring platform with built-in alerting, auto-discovery, and long-term data storage.

• Monitoring