https://devops-monk.com/tutorials/spring-batch/Recent content in Spring Batch Tutorial — Complete Guide on Devops MonkHugoen-usMon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/spring-batch-best-practices/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/spring-batch-best-practices/Introduction This is the final article in the Spring Batch Tutorial series. It consolidates every hard-won lesson into actionable checklists — use it as a review before deploying a new batch job to production, or as a reference when debugging an existing one. Part 1: Job Design Always design for restartability Every step that writes to a database uses idempotent SQL (ON DUPLICATE KEY UPDATE or WHERE NOT EXISTS) Every ItemReader has a unique .https://devops-monk.com/tutorials/spring-batch/multi-threaded-steps/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/multi-threaded-steps/Introduction A single-threaded Spring Batch step processes one chunk at a time — read N items, process N items, write N items, repeat. For large data sets this is a bottleneck. Spring Batch offers two in-JVM scaling options: Approach How it works Use when Multi-threaded step Multiple threads each process independent chunks Reader is thread-safe (JdbcPagingItemReader) AsyncItemProcessor Processing runs concurrently; writes remain sequential I/O-bound processors (REST calls, slow enrichment) This article covers both, plus the thread-safety requirements you must meet.https://devops-monk.com/tutorials/spring-batch/partitioning/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/partitioning/Introduction Multi-threaded steps (Article 20) run multiple chunks concurrently from a single reader. Partitioning is different — it splits the data into independent slices before processing starts, then runs each slice as its own StepExecution with its own reader, processor, writer, and metadata. This gives you: True independence between partitions — one partition’s failure doesn’t affect others A separate StepExecution row per partition in BATCH_STEP_EXECUTION — full visibility into per-partition progress The ability to distribute partitions across multiple JVMs (remote partitioning, covered in Article 22) Partitioning Architecture ManagerStep (PartitionStep) │ ├── Partitioner → creates N ExecutionContexts (one per partition) ├── PartitionHandler → distributes partitions to workers │ └── Worker Steps (run per partition) ├── ItemReader → reads only its slice of data ├── ItemProcessor └── ItemWriter The manager step runs once: it calls Partitioner.https://devops-monk.com/tutorials/spring-batch/performance-tuning/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/performance-tuning/Introduction A poorly tuned batch job can be 10–100x slower than a well-tuned one. The biggest gains come from a handful of settings — chunk size, MySQL JDBC rewrite, connection pool alignment, and avoiding unnecessary object creation. This article covers each systematically. Chunk Size — The Most Impactful Setting Chunk size determines how many items are processed per transaction. Too small = too many round trips to the database. Too large = long transactions, high memory pressure, slower rollback on failure.https://devops-monk.com/tutorials/spring-batch/remote-partitioning-kafka/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/remote-partitioning-kafka/Introduction Local partitioning (Article 21) runs all workers on one JVM. When a single machine is the bottleneck — CPU, memory, or network bandwidth — you need workers on separate machines. Spring Batch Integration provides two patterns for this: Pattern What distributes Coordinator controls Workers do Remote Partitioning Partition descriptors (small messages) Data splitting, aggregation Full read-process-write per partition Remote Chunking Actual items (larger messages) Reading Processing + writing only Remote partitioning is the more common choice — workers read directly from the database/file, so only small partition metadata crosses the network.https://devops-monk.com/tutorials/spring-batch/scheduling-batch-jobs/Mon, 04 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/scheduling-batch-jobs/Introduction A batch job that only runs manually is barely useful. Production jobs run on a schedule — nightly, hourly, after a file arrives. Spring Boot provides three scheduling options: Option Persistence Clustered Use when @Scheduled No No Simple cron on a single node Quartz Scheduler Yes (DB) Yes HA scheduling, persistent triggers External scheduler (Kubernetes CronJob, Airflow) Varies Yes Complex pipelines, dependency management @Scheduled — Simple Cron Trigger The simplest approach: annotate a method with @Scheduled and run the job from it.https://devops-monk.com/tutorials/spring-batch/advanced-processors/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/advanced-processors/Introduction Real batch jobs often need more than one transformation step per item. You might validate first, then normalize, then enrich, then convert to the output type. CompositeItemProcessor chains multiple single-responsibility processors together. For I/O-bound enrichment steps, AsyncItemProcessor runs processing concurrently on a thread pool — giving you parallelism without rewriting your step as multi-threaded. CompositeItemProcessor CompositeItemProcessor chains a list of processors. The output of each processor becomes the input to the next.https://devops-monk.com/tutorials/spring-batch/advanced-writers/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/advanced-writers/Introduction The previous article covered the two most common writers: FlatFileItemWriter for files and JdbcBatchItemWriter for database inserts. This article covers advanced scenarios: Persisting JPA entities with JpaItemWriter Writing to multiple destinations simultaneously with CompositeItemWriter Routing items to different writers based on content with ClassifierCompositeItemWriter Building custom writers for REST APIs, message queues, and cloud storage Combining a writer with a post-step cleanup tasklet JpaItemWriter JpaItemWriter calls EntityManager.merge() on each item.https://devops-monk.com/tutorials/spring-batch/chunk-oriented-processing/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/chunk-oriented-processing/The read-process-write loop is at the heart of almost every Spring Batch job. Understanding exactly how it works — where transactions begin and end, what gets rolled back on failure, and how Spring Batch knows where to restart — makes everything else in the framework click into place. This article goes deep on chunk-oriented processing: the execution model, the three interfaces, the counters that track progress, and how to size chunks correctly.https://devops-monk.com/tutorials/spring-batch/job-step-configuration/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/job-step-configuration/Introduction So far every example has had a single step. Real batch jobs usually have multiple steps — validate input, import data, generate a report, send a notification, clean up temp files. Spring Batch’s JobBuilder DSL lets you compose these steps into flows with conditional branching, parallel execution, and decision logic. This article covers: Linear multi-step jobs Conditional step transitions using ExitStatus JobExecutionDecider for runtime branching Parallel flows with split Nested jobs with FlowStep The Flow abstraction for reusable step sequences Linear Multi-Step Job The simplest multi-step job runs steps in sequence.https://devops-monk.com/tutorials/spring-batch/introduction-to-spring-batch/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/introduction-to-spring-batch/Every application has a class of work that doesn’t fit the request-response model: process 2 million orders overnight, generate 500,000 monthly statements, migrate 10 years of legacy data before Monday morning. This work needs to be reliable, restartable after failures, and fast enough to finish in the available window. That’s what Spring Batch is built for. This article covers what Spring Batch is, when to use it, and how its architecture works.https://devops-monk.com/tutorials/spring-batch/job-parameters-execution-context/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/job-parameters-execution-context/Introduction Two mechanisms let you pass information into and through a Spring Batch job: JobParameters — input values provided at launch time (a date, a file path, a run ID). They are immutable and persisted to BATCH_JOB_EXECUTION_PARAMS. ExecutionContext — a key-value map that steps can read and write during execution. It is persisted after each chunk commit, enabling restartability. Understanding both is essential for building jobs that can be safely re-run, restarted after failure, and parameterised for different data sets.https://devops-monk.com/tutorials/spring-batch/job-repository-batch-metadata/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/job-repository-batch-metadata/Introduction Every time Spring Batch runs a job it records the run’s history in a set of relational tables. This metadata is not optional — it is what makes Spring Batch reliable. Without it there would be no restart capability, no duplicate-run prevention, and no audit trail. Understanding the metadata layer is essential for debugging failures, building monitoring dashboards, and designing restartable jobs. In this article you will learn: The role of JobRepository and JobExplorer in the Spring Batch architecture The six metadata tables — their schema, purpose, and relationships How Spring Batch uses these tables to enable restartability How to query batch history directly in MySQL How to access metadata programmatically with JobExplorer Key changes in Spring Batch 5 (removed MapJobRepositoryFactoryBean, new testing approach) All examples use Spring Boot 3.https://devops-monk.com/tutorials/spring-batch/listeners/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/listeners/Introduction Spring Batch emits lifecycle events at every stage of execution — before and after a job runs, before and after each step, before and after each chunk, and before and after each individual read/process/write call. Listeners let you hook into these events without modifying your core batch logic. Common uses: Log start/end times and item counts Send success/failure notifications (Slack, email, PagerDuty) Publish metrics to Prometheus or CloudWatch Log every skipped item to a dead-letter table Reset resources before a step begins Listener Hierarchy Job ├── JobExecutionListener beforeJob / afterJob │ └── Step ├── StepExecutionListener beforeStep / afterStep ├── ChunkListener beforeChunk / afterChunk / afterChunkError ├── ItemReadListener beforeRead / afterRead / onReadError ├── ItemProcessListener beforeProcess / afterProcess / onProcessError └── ItemWriteListener beforeWrite / afterWrite / onWriteError └── SkipListener onSkipInRead / onSkipInWrite / onSkipInProcess JobExecutionListener @Component public class ImportJobListener implements JobExecutionListener { private static final Logger log = LoggerFactory.https://devops-monk.com/tutorials/spring-batch/reading-flat-files/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/reading-flat-files/Introduction Flat files — CSV exports, fixed-width mainframe feeds, pipe-delimited data dumps — are the most common input source for batch jobs. Spring Batch’s FlatFileItemReader handles all of them. It is restartable out of the box: it persists its line-number position in the ExecutionContext so that a restarted job resumes exactly where it crashed. In this article you will build a complete order-import job that reads a CSV file and inserts rows into a MySQL orders table.https://devops-monk.com/tutorials/spring-batch/reading-external-sources/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/reading-external-sources/Introduction Not all batch input comes from files or databases. You may need to pull orders from an e-commerce API, sync products from a supplier feed, or process CSV exports stored in Amazon S3. Spring Batch provides MultiResourceItemReader for S3 files and a clean ItemReader interface for anything else. In this article you will build: A custom ItemReader that pages through a REST API An S3 reader that downloads files on demand A composite reader that merges multiple sources into one step The ItemReader Contract The entire ItemReader interface is one method:https://devops-monk.com/tutorials/spring-batch/reading-from-mysql/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/reading-from-mysql/Introduction Most real batch jobs read from a database, not a file. Spring Batch provides two JDBC readers for this: Reader Strategy Thread-safe Use when JdbcCursorItemReader Open a server-side cursor, stream rows No Single-threaded step, huge result sets JdbcPagingItemReader Execute LIMIT / OFFSET queries in a loop Yes Multi-threaded steps, sorted data Both handle restartability automatically and both work with any DataSource — including MySQL. JdbcCursorItemReader How it works The reader opens a single JDBC ResultSet on Step.https://devops-monk.com/tutorials/spring-batch/reading-with-jpa/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/reading-with-jpa/Introduction When your application already uses JPA/Hibernate, JpaPagingItemReader lets you read data using JPQL queries and mapped entities instead of raw JDBC. You get the full object graph, type-safe queries, and familiar entity lifecycle — but you also inherit JPA’s pitfalls: the N+1 problem, session-per-read overhead, and first-level cache growth. This article covers: When to choose JpaPagingItemReader over JdbcPagingItemReader Setting up the reader with JPQL and named queries Fetching associations to avoid N+1 Clearing the persistence context to prevent memory leaks A complete order-processing example with MySQL When to Use JpaPagingItemReader Use it when:https://devops-monk.com/tutorials/spring-batch/retry-logic/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/retry-logic/Introduction Batch jobs interact with databases, REST APIs, and file systems — all of which fail transiently. A MySQL deadlock resolves itself in milliseconds. A network timeout to an external service clears up in seconds. Retrying these transient failures automatically is far better than failing the entire job and requiring a manual restart. Spring Batch has built-in retry support at the step level, integrated with its transaction management. This article covers everything you need to configure robust retry behaviour.https://devops-monk.com/tutorials/spring-batch/skip-logic-restartability/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/skip-logic-restartability/Introduction Retry handles transient failures. Skip handles permanent ones — bad data rows, constraint violations, malformed records that will never succeed no matter how many times you retry. Skip logic lets your job continue processing good records while recording bad ones for human review. This article covers: Configuring skip for specific exception types Custom SkipPolicy for fine-grained control Dead-letter table pattern for tracking skipped items Stopping a job intentionally vs failing it Handling abandoned executions Designing jobs that restart safely after any failure Basic Skip Configuration return new StepBuilder("importOrdersStep", jobRepository) .https://devops-monk.com/tutorials/spring-batch/spring-boot-spring-batch-setup/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/spring-boot-spring-batch-setup/Article 2 showed how chunk processing works conceptually. This article sets up the real project you’ll build on for the rest of the series — complete Maven configuration, MySQL metadata tables, multiple ways to trigger jobs, and correct use of JobParameters. Everything in this article is production-ready from the start. Project Dependencies Add spring-boot-starter-batch to your Maven project. That one starter brings in everything you need. <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.https://devops-monk.com/tutorials/spring-batch/tasklets/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/tasklets/Introduction Not every step in a batch job reads-processes-writes a stream of items. Sometimes you need to: Delete yesterday’s temp files before reading today’s data Truncate a staging table before loading new data Call a stored procedure to aggregate results Send an email or Slack notification after processing Execute a DDL statement to add an index These operations do not have items — they are single units of work. Spring Batch handles them with the Tasklet interface.https://devops-monk.com/tutorials/spring-batch/testing-spring-batch/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/testing-spring-batch/Introduction Spring Batch jobs are code — they need tests. Without tests you will catch errors in production. With tests you catch them in CI. Spring Batch has a dedicated testing module that provides @SpringBatchTest, JobLauncherTestUtils, and JobRepositoryTestUtils. These tools let you: Run a complete job in a test and assert on its BatchStatus Launch individual steps and inspect StepExecution counters Test readers, processors, and writers in complete isolation This article covers all three levels: unit, integration, and database integration with Testcontainers.https://devops-monk.com/tutorials/spring-batch/item-processor/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/item-processor/Introduction ItemProcessor<I, O> sits between the reader and the writer. It receives one item at a time and returns a transformed item — or null to filter the item out entirely. Processors are optional: if your job reads and writes the same type with no transformation needed, you can omit them. The processor interface is one method: public interface ItemProcessor<I, O> { O process(I item) throws Exception; } Return a processed item to pass it to the writer.https://devops-monk.com/tutorials/spring-batch/writing-files-databases/Sun, 03 May 2026 00:00:00 +0000https://devops-monk.com/tutorials/spring-batch/writing-files-databases/Introduction After reading and processing data, your batch job needs to write results somewhere. Spring Batch provides two essential writers: FlatFileItemWriter — writes items to CSV or any delimited/formatted flat file JdbcBatchItemWriter — writes items to a database using JDBC batch inserts/updates Both are transactional and restartable. This article covers both in depth, plus transaction semantics you must understand to avoid duplicates and partial writes. How Writers Work in Spring Batch Writers receive a Chunk<O> — a list of all items processed in the current transaction.