English | 日本語
A lightweight batch framework for Go, inspired by JSR-352.
Surfin - Batch framework is being developed with robustness, scalability, and operational ease as top priorities. With declarative job definitions (JSL) and a clean architecture, it efficiently and reliably executes complex data processing tasks.
- 🚀 Go Performance: Leverages Go's native concurrency and performance to the fullest.
- 🏗️ Convention over Configuration (CoC): Eliminates boilerplate code through Convention over Configuration (CoC), ensuring robustness with minimal implementation.
- ✨ Observable: Integrates Prometheus/OpenTelemetry at its core, enabling distributed tracing and metrics collection simply by following conventions.
- 🛠️ Flexible: Supports dynamic DB routing and Uber Fx (DI) for flexible adaptation to complex infrastructures and custom requirements.
- 🔒 Robust: Provides fault tolerance features through persistent metadata, optimistic locking, and fine-grained error handling (retry/skip).
- 🎯 JSR-352 Compliant: Implements a batch processing model inspired by the industry standard (JSR-352).
- 📈 Scalable: Supports large-scale datasets through distributed execution via remote worker integration and abstraction of Partitioning.
- Declarative Job Definition (JSL): Define jobs, steps, components, and transitions declaratively using YAML-based JSL.
- Chunk/Tasklet Model: Supports chunk-oriented processing with
ItemReader,ItemProcessor,ItemWriter, andTaskletfor single-task execution. - Advanced Flow Control: Build complex job flows with conditional branching (
Decision), parallel execution (Split), and flexible transition rules (Transition). - Listener Model: Custom logic can be injected into Job, Step, Chunk, and Item lifecycle events.
- Restartability: Failed jobs can be accurately restarted from their interruption point using
JobRepositoryandExecutionContext. - Fault Tolerance: Supports item-level retry and skip policies. Automatically applies chunk-splitting skip logic for write errors.
- Optimistic Locking: Implements optimistic locking at the metadata repository layer to ensure data consistency in distributed environments.
- Sensitive Information Masking: Automatically masks sensitive information when
JobParametersare persisted and logged.
- OpenTelemetry/Prometheus Integration: Integrates distributed tracing (automatic Job/Step Span generation) and metrics collection (MetricRecorder) into the framework's core. Ensures reliable monitoring even for short-lived batches and provides a foundation for integration with Remote Schedulers.
- Fine-grained Error Policy: Supports declarative retry/skip strategies based on error characteristics (Transient, Skippable).
- Dynamic DI: Adopts Uber Fx for dependency injection, enabling dynamic component construction and lifecycle management.
- Dynamic Data Source Routing: Supports dynamic switching between multiple data sources (Postgres, MySQL, etc.) based on execution context.
- Distributed Execution Abstraction: Abstraction of
PartitionerandStepExecutorsupports delegation of execution from local to remote workers (e.g., Kubernetes Job).
Refer to the following guide to build a minimal application using Surfin.
For more detailed features, configurations, and building complex job flows, refer to the following documentation.
- Architecture and Design Principles: Refer to detailed information on the framework's architecture, design principles, and project structure.
- Guide to Creating Surfin Batch Applications: A complete guide to JSL, custom components, listeners, and flow control.
- Implementation Roadmap: The framework's design goals and progress.
If you have any questions or encounter issues, please use GitHub Issues.
- GitHub Issues: Report bugs or request features