AWS Glue 6.0 Can Break Old Jobs—Check These Five Changes First

The safe way to migrate an AWS Glue 5.1 job to 6.0 is to clone it, run the candidate against fixed inputs and isolated outputs, and compare it with the existing job before promotion. Preserve the 5.1 definition, arguments and deployment artifacts throughout testing so rollback means restoring a known-good job rather than reconstructing one.

Check five compatibility boundaries first: ANSI SQL behavior, S3A replacing EMRFS, removal of AWS SDK for Java v1, Scala binary incompatibility and Python 3.13. The AWS Glue version table lists Spark 4.1.1, Scala 2.13.17, Python 3.13 and Java 17 for Glue 6.0; it also lists S3A as the sole S3 connector and Java SDK v2 as the only included Java SDK generation.

1. Build an inventory and a reversible test lane

Inventory each job’s script, arguments, IAM role, connections, worker configuration, bookmarks, custom JARs, Python packages, Spark settings, S3 configuration and table formats. Include workflows, schedules and event-driven callers because restoring a job alone will not correct an incompatible parameter or trigger.

Choose representative clones that cover the estate’s real risks: a SQL-heavy transformation, a job with custom JVM code, a Python job with compiled packages, a VPC-bound S3 workload and a writer to shared Iceberg tables. Send test writes to separate prefixes or disposable tables, and prevent candidate jobs from advancing production bookmarks.

Run Glue 5.1 against a fixed input snapshot to establish the comparison baseline. Record schemas, row counts, null counts for important fields, partition sets, rejected records, runtime and cost-driving worker usage; a successful status alone cannot reveal changed results.

2. Test ANSI SQL and S3A before application libraries

Spark 4.1 enables ANSI mode by default, so operations that previously returned null can now stop a job. Exercise malformed numeric strings, overflowing arithmetic, invalid dates, unexpected nulls and boundary array indexes. Prefer explicit validation or tolerant conversion when the failures expose bad input.

Setting spark.sql.ansi.enabled=false restores earlier overflow behavior, but treat that as a documented compatibility choice rather than a universal fix. Reconcile the resulting output with the 5.1 baseline and record which queries depend on non-ANSI behavior.

EMRFS is unavailable in Glue 6.0. Remove EMRFS-specific settings such as fs.s3.consistent.*, search custom code for the former filesystem class, and test reads, writes, partition discovery and large transfers through the same VPC endpoints and regional path used in production. A run from an unrestricted network does not validate S3A connectivity from a private subnet.

Rollback point: stop if SQL results differ without an accepted rule change or S3A cannot complete the production access pattern. Keep all candidate writes isolated while correcting queries, configuration or network routing.

3. Rebuild both sides of the JVM boundary

Search source code and resolved dependency trees for Java SDK v1 packages under com.amazonaws, then migrate direct integrations to SDK v2 APIs under software.amazon.awssdk. Inspect transitive dependencies as well: an old connector can carry v1 even when application code has no direct import.

The Glue 6.0 migration guide specifies SDK v2 version 2.44.6, warns about older SDK copies supplied with --user-jars-first, and maps the Scala change from 2.12.18 to 2.13.17. It also identifies NoSuchMethodError and ClassNotFoundException as possible results when a Glue 6.0 job loads a JAR compiled for Scala 2.12.

Recompile custom code and obtain Spark 4.1 and Scala 2.13 builds of third-party connectors rather than copying old artifacts. Avoid shading the SDK unnecessarily; if it must be bundled, align it with the required runtime version. Validate credential discovery, pagination, retries and the external-service calls the job actually makes.

Rollback point: store the 5.1 and 6.0 artifacts under different versioned object keys. Do not replace the known-good JAR until dependency inspection, startup and representative integration calls pass.

4. Treat Python 3.13 as a fresh runtime

Scan scripts and packaged dependencies for removed modules, including imp, cgi and cgitb. Resolve packages again for Python 3.13 instead of copying a 3.11 environment, especially when dependencies contain native extensions or interpreter-specific wheels.

Test serialization, datetime parsing, regular expressions, subprocess handling and exception paths as well as imports. Run UDFs with nulls and uncommon data types because distributed workers can reach branches that a driver-only smoke test misses. Python code using boto3 does not require the Java package migration, although pinned boto3 and botocore versions still need runtime validation.

Lock the accepted Python 3.13 package set and confirm that it reaches both driver and executors. Retain the 3.11 lock file and bundle beside it so rollback does not depend on resolving an old environment again.

5. Keep shared Iceberg tables on version 2 for Athena

Changing the Glue runtime does not require changing an existing table’s Iceberg format version. Treat a v3 upgrade as a separate migration because every reader, maintenance process and recovery tool must support the selected table format.

This distinction is decisive when Athena remains a consumer. The Athena Iceberg limitations specify that Athena creates and operates on Iceberg v2 tables. Keep format-version=2 for shared tables queried by Athena and validate Glue 6.0 against cloned v2 tables.

Approve v3 only after every consumer has demonstrated support. If a v3 capability needs evaluation, create a separate table from a fixed snapshot instead of converting the shared production table; restoring an earlier snapshot should not be treated as a table-format downgrade procedure.

Promote in batches with explicit rollback points

Compare each 6.0 candidate with its 5.1 baseline using business reconciliations and operational measures. Confirm schemas, aggregates, partitions, bookmark progression, rejected-record handling, runtime, worker usage and retry behavior before judging performance or cost.

Promote a low-impact batch first and define an observation window. Roll back by restoring the previous job definition and input position, not by assuming that reverting the runtime will undo writes already committed to shared tables. Retire the 5.1 artifacts only after downstream consumers and production monitoring have passed the same acceptance gates.

Also read:

• A Single Tool Won’t Unlock High-Paying Data Jobs—Build These 10 Skills
• Five Weeks Since SpaceX's Historic IPO: Volatility, Rule Changes, and Hard Investment Lessons
• More Traffic Is Not the Fix: Five Website Changes That Remove Conversion Friction