Modernizing ETL for the agentic era

ETL platforms have carried enterprise reporting and analytics for decades. At their simplest, they extract data from source systems, transform it according to business rules, and load the result somewhere people and applications can use it.

The process is straightforward. The systems built around it rarely stay that way. Over time, mappings accumulate. SQL grows around them. Schedules, triggers, dependencies, and operational fixes pile up. Business rules that were once discussed in design meetings become embedded in code and production behavior.

Eventually, the ETL platform is doing more than moving data. It carries years of business decisions. That is what makes modernization difficult. Infrastructure can be replaced. Pipelines can be rewritten. The harder job is preserving what the old system knows and proving that the new one still produces results the business can trust. As organizations expand their cloud, analytics, and AI capabilities, that problem becomes harder to ignore.

A traditional ETL platform can work reliably for years. Age alone is not a reason to replace it. The pressure usually comes from what the organization needs next.

Data volumes grow. More teams consume the data. New analytical workloads appear. AI programs need governed, reliable data. Release cycles get shorter while the engineers who know the original implementation move on.

A change that looks simple on paper can then require a surprising amount of investigation. Business logic may be spread across mappings, SQL procedures, triggers, schedules, lookup tables, and operational workarounds. Documentation may describe the intended design while the production implementation reflects years of exceptions.

The result is a platform that becomes harder to change with confidence. That affects more than engineering productivity. It affects how quickly teams can respond to the business, how much effort goes into production support, and how confidently an organization can use its data for analytics and AI.

There is no single replacement for a traditional ETL platform. For some organizations, moving to a managed cloud ETL service is the right next step. It can reduce infrastructure management and preserve a development model teams already understand. The tradeoff is that some of the coupling between transformation logic, workflows, and operations may remain.

Other organizations move more transformation into a cloud warehouse or lakehouse. SQL-heavy workloads can become easier to develop and scale, while storage and compute can evolve independently. That flexibility usually requires more deliberate decisions around orchestration, testing, data quality, and ownership.

Some enterprises go further and organize data around domain-owned products with explicit contracts and shared governance. That can improve accountability and make data easier to reuse, but it also requires changes in decision rights and operating model, not just technology.

The right approach depends on the estate being modernized, the workloads it supports, the skills of the teams operating it, and how much architectural change the organization is prepared to absorb.

The useful question is not simply, “What replaces our ETL tool?” It is, “What architecture and operating model do we want to own next?”

One pattern on Amazon Web Services (AWS) is to separate the major responsibilities of the data platform. Data can land in Amazon Simple Storage Service (Amazon S3). AWS Glue can handle transformation and validation. Curated data can be stored in S3 using Apache Iceberg tables. Amazon Redshift can serve reporting and analytical consumers, while Apache Airflow manages orchestration and dependencies. In sequence, that is landing, transformation, curated storage, analytical serving, and orchestration. Each can evolve without replacing the rest of the estate.

That separation gives teams more freedom to evolve individual parts of the platform instead of treating the entire data estate as one tightly coupled system.

A more modular architecture comes with more explicit decisions. Teams need patterns for deployment, testing, observability, security, data contracts, recovery, and ownership. Those decisions are part of modernization, not follow-up work after the platform is built.

Governance belongs here as well. A useful data contract should say more than what columns exist. Teams need agreement on ownership, expected quality, versioning, acceptable change, and how exceptions are handled. The migration also needs an audit trail: what changed, what was tested, what was approved, and what evidence supported the production decision. Those controls become even more important when AI starts doing part of the engineering work.

The hardest part of many ETL migrations is not getting a new pipeline to execute. It is building enough evidence to show that the replacement preserves the expected business behavior. Legacy systems often contain business logic that has been refined over years. Some of it is obvious in code. Some of it only becomes visible when engineers compare the implementation with production behavior. That makes migration partly an archaeology exercise. Teams have to identify the real sources, reconstruct transformations, understand downstream dependencies, build the replacement, test it, and determine whether the output still meets the business contract.

One practical way to do that is to run the legacy and modern pipelines in parallel. The existing platform continues feeding production while the new implementation writes to a separate environment. A reconciliation process compares schemas, record counts, values, and transformation results. Differences become exceptions to investigate. A workload advances only after it satisfies the agreed reconciliation criteria.

This is governance expressed as engineering. Testing, reconciliation, exception handling, and cutover approval become part of the delivery workflow instead of controls added at the end. That works well for a handful of pipelines. It becomes much harder when hundreds need to move.

AI can generate transformation code quickly. Useful as that is, code generation covers one step in a migration. A pipeline still has to move through source analysis, transformation interpretation, implementation, tests, orchestration, documentation, validation, reconciliation, review, and production approval.

Most teams will eventually have accelerators for several of those activities. The bottleneck moves to the spaces between them. An engineer may use one tool to understand legacy code, another to generate implementation, another to create tests, and another to validate the result. Each tool can save time while the engineer still carries context between them and coordinates the sequence manually. The larger opportunity is to connect those capabilities into one governed workflow.

At Slalom, we have brought accelerators into different parts of that lifecycle. We developed Data Genius to help teams understand legacy code and recover information from systems where documentation is incomplete. Reconciliation tooling can move validation earlier. Our other accelerators help support implementation, testing, orchestration, and documentation. Each removes work from a part of the migration. Connecting them changes the delivery model.

A Migration Factory treats migration as a repeatable engineering workflow rather than a collection of individual tasks.

Using Kiro Skills, teams can encode patterns for source and schema analysis, transformation interpretation, data-contract generation, AWS Glue job generation, test scaffolding, Apache Airflow DAG creation, and validation.

The useful part is not simply giving an agent an instruction.

The Skills can capture expected inputs and outputs, engineering conventions, validation steps, and the points where the workflow should stop for review.

That gives the agent a defined way of working rather than an open-ended objective.

The workflow itself is a sequence:

1. Understand the source systems, schemas, legacy logic, and downstream dependencies.
2. Define the data contract so the implementation has an explicit target.
3. Build the AWS Glue job, tests, and Apache Airflow DAG using the agreed patterns.
4. Refine the implementation and run quality checks. Failed checks return the work for correction.
5. Validate in beta and reconcile with production. Investigate mismatches before the workload advances.
6. Release after engineers review the implementation and supporting evidence and approve production cutover.

Repeatable work can then move through the workflow while engineers spend their time on the decisions that require judgment: ambiguous business logic, exceptions, code review, reconciliation, and production approval. The controls stay visible.

Generated code still has to pass tests. Reconciliation mismatches still have to be resolved. Changes still move through source control. Production cutover still requires human approval. That is what makes agentic delivery useful in an enterprise environment.

We saw this pattern on a large retail data-modernization program. Slalom introduced accelerators during the engagement, including Data Genius for legacy analysis and a reconciliation utility that moved validation earlier in the lifecycle. Those capabilities were already helping individual parts of the migration.

The remaining friction was between them. In one pre-agentic workstream, the team completed 24 table migrations over 14 weeks, or about 1.7 tables per week. After those migration activities were connected through the Migration Factory, the team completed 25 tables during one three-day delivery window. That three-day result was a delivery burst in one engagement, not a sustained weekly run rate, and should not be treated as a universal benchmark.

The numbers are useful as evidence, not as a promise of what every migration should achieve. What changed underneath them is more interesting. The accelerators already existed. Orchestration allowed context and work to move between them without requiring an engineer to manually coordinate every transition.

The unit of work moved from completing an individual development task to advancing a migration toward production readiness.

Once migration work is broken into bounded units with clear controls, some of it does not need to stop when the engineering team logs off.

During a defined execution window, Kiro Crew can pick up work that has already been approved and advance it across sessions through analysis, implementation, testing, and validation. Kiro Crew began inside Amazon as MeshClaw and is built on the Kiro CLI. It supports scheduled work, parallel subagents, persistent context, checkpoints, validation, and retries for long-running tasks.

Kiro Crew also provides a security foundation for this kind of unattended execution, including sandboxing, command restrictions, sensitive-path and credential protections, action approvals, audit logging, and live observability.

In our migration workflow, those platform controls were combined with migration-specific guardrails. Changes still moved through source control. Tests and validation gates still had to pass. Reconciliation criteria still had to be met. Production authority stayed with the engineering team.

By morning, engineers could receive a reviewable migration package containing generated code, test output, validation results, exceptions, and reconciliation artifacts. If the workflow reached a condition that required judgment, the work returned to an engineer. The aim was to move the work to the next useful human decision.

Unattended does not mean uncontrolled.

Migration development is not the only place where engineers spend time assembling context. Production incidents often begin with an alert and a manual investigation. Someone has to gather logs, identify the failed job, understand its dependencies, check recent runs, and decide where to start.

AI can help with that investigation as well. A DevOps Agent can assemble context across AWS Glue jobs, Apache Airflow DAGs, alarms, logs, and runbooks and give an engineer a starting hypothesis instead of another isolated alert. The engineer still decides what action to take. The benefit is reducing the time between “something failed” and “we understand enough to act.”

For a deeper look at that operating model, check out my previous article, "Incident Response with AWS DevOps Agent."

The Migration Factory did not start from scratch. It grew from accelerators that were already helping at different points in the migration. That same idea extends across Slalom’s broader Data Genius Agentic and AgentOps capabilities.

Data Genius Agentic focuses on understanding and planning. It analyzes the code that actually runs, reconstructs lineage, data models, flows, and business meaning, and produces an execution-ready plan.

AgentOps carries that plan into engineering delivery. Agents can implement against the plan, create tests and pull requests, evaluate changes against defined quality gates, and operate within human merge controls and observable execution.

The path is straightforward: Understand and plan. Build and test. Review and govern. That continuity matters because modernization work rarely stalls on one line of code. The harder problems are in the transitions: understanding undocumented behavior, carrying that understanding into implementation, proving the output, resolving exceptions, and preparing the change for production. Reducing those handoffs is where agentic engineering can make a meaningful difference.

A successful ETL modernization should leave the organization with more than a new technology stack. The immediate technology benefits are easier to see: a more modular architecture, clearer operational visibility, better maintainability, and a platform that can evolve as analytical and AI workloads change.

The delivery benefits matter just as much. Business logic that was buried in legacy implementations becomes easier to understand. Validation becomes repeatable. Production decisions have evidence behind them. Engineers spend less time reconstructing context or manually coordinating the same migration steps. That can shorten migration timelines without asking teams to trade away control.

It can also leave the organization better prepared for AI. Governed AI depends on governed data, and governed data depends on knowing where information came from, how it changed, who owns it, and what happened when something did not meet expectations.

That is where Slalom’s role goes beyond implementation. Modernization requires decisions about architecture, sequencing, governance, operating model, and where agents should be allowed to act. Those decisions depend on the legacy estate and on what the business needs from its data next.

There is no universal target architecture. There is no universal agentic workflow. A few principles travel well: make legacy logic visible, define ownership and contracts, build validation into the migration, automate repeatable work, make stop conditions explicit, and keep production decisions accountable.

Done well, modernization gives organizations faster delivery, lower operational friction, better evidence behind data changes, and a stronger foundation for analytics and AI. AI can make the migration faster. Trust still has to be engineered.