{"id":79347,"date":"2026-05-24T15:14:18","date_gmt":"2026-05-24T09:44:18","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=79347"},"modified":"2026-06-08T18:40:32","modified_gmt":"2026-06-08T13:10:32","slug":"from-100mb-to-50gb-migrating-dynamodb-tables-across-aws-accounts","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/from-100mb-to-50gb-migrating-dynamodb-tables-across-aws-accounts\/","title":{"rendered":"From 100MB to 50GB: Migrating DynamoDB Tables Across AWS Accounts"},"content":{"rendered":"

You’d think moving data from one AWS account to another would be straightforward. Export it, import it, done. That’s what the documentation<\/a> implies, and honestly, that’s what I assumed going in.<\/p>\n

It wasn’t.<\/p>\n

What started as a data migration quickly turned into a system reconstruction problem. In DynamoDB, data alone is only half the story.<\/p>\n

This is the story of migrating 35+ DynamoDB tables, ranging from 100MB to over 50GB, across two AWS accounts, with a bucket in the destination account, streams to reconnect, Lambda triggers to recreate, and a PITR requirement nobody warned about until it failed mid-run.<\/p>\n

If you’re planning something similar, this post will save you some pain.<\/p>\n

The Setup<\/strong><\/h2>\n

Here’s what we were working with:<\/p>\n

    \n
  • 35+ DynamoDB tables in a source AWS account<\/li>\n
  • A brand new destination AWS account being built from scratch<\/li>\n
  • Tables with wildly varying sizes, some barely 100MB, others pushing 50GB+<\/li>\n
  • Streams and Lambda triggers attached to several tables<\/li>\n
  • GSIs that needed to be preserved exactly<\/li>\n
  • Needed to repeat this across multiple environments, not a one-time run<\/li>\n<\/ul>\n

    Why Cross-Account?<\/strong><\/h2>\n

    The source account was our existing environment serving live traffic. The destination was a new account being built from scratch – full isolation, easy rollback, parallel validation before any traffic switched<\/p>\n

    Why Not Just Use the AWS Console?<\/strong><\/h2>\n

    The honest answer: you can use the UI for a single table. But the moment you have more than two or three tables, the console becomes a liability:<\/p>\n

      \n
    • No repeatability<\/strong> – every step is manual, so every rerun is a fresh opportunity to miss something<\/li>\n
    • No visibility<\/strong> – there’s no good way to track the status of 10+ concurrent export jobs from the UI<\/li>\n
    • No schema injection<\/strong> – passing a full –table-creation-parameters<\/span> file with GSIs isn’t something the UI supports cleanly<\/li>\n<\/ul>\n

      So the path was: understand the CLI flags first, validate with a single table, then wrap everything in scripts.<\/p>\n

      Starting with the CLI also forces clarity, you have to understand exactly what you’re passing and why.<\/p>\n

      How the Migration Was Structured<\/strong><\/h2>\n

      With 35+ tables, a single script wasn’t going to cut it. We split the work into two phases with a deliberate review step in between.<\/p>\n

      Phase 1 \u2014 Discovery (Source Account)<\/h3>\n

      Before export, collect everything:<\/p>\n

        \n
      • Table names, sizes, item counts<\/li>\n
      • Full schema: keys, attribute definitions, billing mode<\/li>\n
      • GSI configuration<\/li>\n
      • Stream settings<\/li>\n
      • Lambda event source mappings<\/li>\n<\/ul>\n

        Output:<\/strong>\u00a0a set of CSV files and JSON schema definitions that a human can actually read and edit.<\/p>\n

        This is where decisions happen. Which tables move? Which Lambda names need to be remapped because they don’t exist in the destination yet?<\/p>\n

        Phase 2 \u2014 Execution (Cross-Account)<\/h3>\n

        After review:<\/p>\n

          \n
        • Enable PITR + trigger exports to S3 (source account)<\/li>\n
        • Monitor export completion<\/li>\n
        • Create tables and import data (destination account)<\/li>\n
        • Monitor import completion<\/li>\n
        • Recreate streams and Lambda triggers<\/li>\n
        • Restore table settings (PITR, deletion protection)<\/li>\n
        • Validate & compare item counts between source and destination<\/li>\n<\/ul>\n

          We needed a way to run this safely more than once, especially when things failed halfway, so we wrapped everything in scripts.<\/p>\n

          In practice, this came down to three steps:<\/p>\n

            \n
          1. Run discovery (source account)<\/li>\n
          2. Trigger exports (source account)<\/li>\n
          3. Run import and setup (destination account)<\/li>\n<\/ol>\n

            The Architecture: Bucket in the Destination Account<\/h2>\n

            The data lifecycle:<\/p>\n

              \n
            • Source DynamoDB exports \u2192 S3<\/li>\n
            • Destination DynamoDB imports \u2190 S3<\/li>\n<\/ul>\n

              If the bucket is in the destination, step 2 is a local read. Fewer cross-account operations, simpler IAM, less surface area for things to go wrong.<\/p>\n

              You need source-account DynamoDB to write into a bucket it doesn’t own. That requires two specific things.<\/p>\n

              Flag 1: –s3-bucket-owner<\/h3>\n

              shell<\/strong><\/p>\n

              aws dynamodb export-table-to-point-in-time \\<\/span><\/p>\n

              \u00a0–table-arn arn:aws:dynamodb:ap-southeast-2:SOURCE_ACCOUNT:table\/MyTable \\<\/span><\/p>\n

              –s3-bucket my-destination-export-bucket \\<\/span><\/p>\n

              –s3-prefix exports\/my-table \\<\/span><\/p>\n

              –export-format DYNAMODB_JSON \\<\/span><\/p>\n

              –s3-sse-algorithm AES256 \\<\/span><\/p>\n

              –s3-bucket-owner $TARGET_BUCKET_ACCOUNT_ID<\/span><\/p><\/blockquote>\n

              The –s3-bucket-owner<\/span> flag is a safety check. DynamoDB will refuse the export if the bucket isn’t actually owned by the account ID you specify. It also tells S3 to assign ownership of the exported objects to the destination account.<\/p>\n

              –s3-sse-algorithm AES256<\/span> ensures everything lands encrypted.<\/p>\n

              Flag 2: The Bucket Policy<\/h3>\n

              The bucket needs to trust two things independently:<\/p>\n

                \n
              • The DynamoDB service (dynamodb.amazonaws.com<\/span>) \u2014 so it can write the actual export files, scoped to your source account ID<\/li>\n
              • Your IAM role – so the CLI call itself doesn’t fail before DynamoDB even gets involved<\/li>\n<\/ul>\n

                Miss either one and you get an AccessDenied<\/span> that doesn’t tell you which layer broke.<\/p>\n

                 <\/p>\n

                The Thing That Stopped Everything: PITR<\/strong><\/h2>\n

                DynamoDB’s export feature is built on Point-in-Time Recovery. It takes a consistent snapshot of your table at a specific moment without locking the table or interrupting live traffic. That’s the feature that makes it genuinely useful.<\/p>\n

                But PITR has to be already enabled<\/strong> before you try to export.<\/p>\n

                We hit this mid-run. The error is clear enough once you see it, but finding it during a migration window is not a good time.<\/p>\n

                We updated the export script to check and enable PITR automatically, with a short buffer before triggering the export.<\/p>\n

                The Part People Miss: GSIs Don’t Come With the Export<\/strong><\/h2>\n

                DynamoDB exports capture item data only. The table’s structural configuration billing mode, stream settings, and critically, Global Secondary Indexes<\/strong> are not included.<\/p>\n

                This is the kind of issue you don\u2019t notice immediately. The table looks fine\u2026 until a query suddenly doesn\u2019t return what you expect.<\/p>\n

                How We Fixed It<\/h3>\n

                At import time, the file gets passed directly:<\/p>\n

                python<\/strong><\/p>\n

                import_cmd = [\u00a0 \u00a0<\/span><\/p>\n

                “aws”, “dynamodb”, “import-table”,<\/span><\/p>\n

                “–s3-bucket-source”, f”Bucket={bucket},KeyPrefix={prefix}”,<\/span><\/p>\n

                “–input-format”, “DYNAMODB_JSON”,<\/span><\/p>\n

                “–input-compression-type”, “GZIP”,<\/span><\/p>\n

                “–table-creation-parameters”, f”file:\/\/{temp_def}”,<\/span><\/p>\n

                ]<\/span><\/p><\/blockquote>\n

                The –table-creation-parameters<\/span> file contains the full key schema, attribute definitions, GSI projections, and billing configuration. The destination table is created with everything intact in one step.<\/p>\n

                A few things worth noting about these flags:<\/p>\n

                  \n
                • DYNAMODB_JSON<\/span> must match the export format. If your export used Ion format, the import needs to match. Mixing these produces unclear errors.<\/li>\n
                • GZIP<\/span> is necessary because DynamoDB compresses its export files by default. Without this flag, the import job will try to parse compressed data as raw JSON and fail.<\/li>\n<\/ul>\n

                  Table Size Changes Everything<\/h2>\n

                  Not all 35+ tables behaved the same, and this became obvious quickly.<\/p>\n\n\n\n\n\n\n
                  \n

                  Table Size<\/strong><\/h4>\n<\/td>\n

                  		\n

                  Export Time<\/strong><\/h4>\n<\/td>\n

                  		\n

                  Import Time<\/strong><\/h4>\n<\/td>\n<\/tr>\n

                  ~100 MB<\/span><\/td>\nMinutes<\/span><\/td>\nMinutes<\/span><\/td>\n<\/tr>\n
                  ~5\u201310 GB<\/span><\/td>\n 20\u201340 min<\/span><\/td>\n 30\u201360 min<\/span><\/td>\n<\/tr>\n
                  50+ GB<\/span><\/td>\n2\u20134 hours<\/span><\/td>\n2\u20134 hours<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                  Both operations are fully asynchronous. You fire the job and poll for completion. That’s why the monitoring scripts ended up being more valuable, fire the jobs, let the poller run, come back when it’s done.<\/p>\n

                  IAM: More Layers Than It Looks Like<\/strong><\/h2>\n

                  Two roles. Clean separation.<\/p>\n

                  Source account role<\/strong> \u2014 used during discovery and export:<\/p>\n

                    \n
                  • dynamodb:ListTables, DescribeTable, ExportTableToPointInTime, DescribeExport<\/span><\/li>\n
                  • lambda:ListEventSourceMappings<\/span><\/li>\n
                  • s3:PutObject, s3:AbortMultipartUpload<\/span> on the destination bucket<\/li>\n<\/ul>\n

                    Destination account role<\/strong> \u2014 used during import and setup:<\/p>\n

                      \n
                    • dynamodb:CreateTable, ImportTable, DescribeImport, UpdateTable, Scan<\/span><\/li>\n
                    • lambda:CreateEventSourceMapping, GetFunction, GetEventSourceMapping<\/span><\/li>\n
                    • s3:GetObject, s3:ListBucket on<\/span> the export bucket<\/li>\n<\/ul>\n

                      Streams and Lambda Triggers: The Last Mile<\/h2>\n

                      After a successful import, the tables exist and the data is there. But streams are disabled, and there are no Lambda triggers. The tables are effectively silent.<\/p>\n

                      Recreating these was the step that required the most care:<\/p>\n

                        \n
                      1. Discovery captured all stream configurations and event source mappings from the source.<\/li>\n
                      2. A Lambda mapping file translated source function names to their destination equivalents because Lambda names rarely match across environments.<\/li>\n
                      3. Post-import, streams were re-enabled on each table and event source mappings were recreated using the destination function names.<\/li>\n<\/ol>\n

                        The mapping file is a simple CSV that gets reviewed manually during the discovery phase. It’s a small thing, but it’s also the step where you’re most likely to introduce a silent bug, a trigger pointing at the wrong function version, or a batch size that doesn’t match the original.<\/p>\n

                        If I Had to Do It Again<\/h2>\n

                        If I had to do this again, there are a few things I\u2019d stick with:<\/p>\n

                          \n
                        • Two-phase architecture<\/strong> – discovery before execution, no exceptions. The number of times something in the discovery output needed a correction before we touched any data was higher than expected.<\/li>\n
                        • Human review gate<\/strong> – not optional. Lambda names don’t match across environments, some tables shouldn’t move at all, and blind automation here is how you get silent bugs.<\/li>\n
                        • Async monitoring over console watching<\/strong> – for the large tables, the monitoring scripts were more useful than the migration scripts themselves. Build the poller, don’t stare at the console.<\/li>\n
                        • PITR on by default<\/strong> – not as a migration prerequisite, just as standard practice. Small ongoing cost, enormous value the day you actually need it.<\/li>\n<\/ul>\n

                          Wrapping Up<\/h2>\n

                          DynamoDB cross-account migration via S3 works. The export is online and non-disruptive. The import handles full schema definitions, including GSIs. For the scale we were working at, it held up well.<\/p>\n

                          But the complexity hides in the details, PITR silently required, bucket trust at the service level not just role level, GSIs needing explicit schema files, streams rebuilt from scratch.<\/p>\n

                          None of those are hard problems once you know about them. The issue is, you usually find out mid-migration.<\/p>\n

                          This kind of migration isn\u2019t hard because of the tools. It\u2019s hard because of the details you don\u2019t see until you\u2019re in the middle of it.<\/p>\n","protected":false},"excerpt":{"rendered":"

                          You’d think moving data from one AWS account to another would be straightforward. Export it, import it, done. That’s what the documentation implies, and honestly, that’s what I assumed going in. It wasn’t. What started as a data migration quickly turned into a system reconstruction problem. In DynamoDB, data alone is only half the story. […]<\/p>\n","protected":false},"author":1615,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0},"categories":[2348],"tags":[248,8565,7788],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79347"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1615"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=79347"}],"version-history":[{"count":3,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79347\/revisions"}],"predecessor-version":[{"id":80023,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79347\/revisions\/80023"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=79347"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=79347"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=79347"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}