AI AI Tools Guidebook
Troubleshooting

S3 Presigned URL Returns 403 Mid-Upload on Large Files

AWS S3 presigned URL works for small files but 403s mid-upload on large ones. Fix with longer TTL, multipart upload, or the SDK upload manager.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

The browser PUTs a 500 MB video against a presigned S3 URL. The first 200 MB go up fine, then the response turns into 403 Forbidden with Request has expired. The signed URL had a TTL of five minutes — fine for thumbnails, way too short for large uploads on slow uplinks. The fix is to stop using single-part PUT for anything over 100 MB. Use S3 multipart uploads with per-part presigned URLs, or hand the work to the SDK’s upload manager which handles chunking, parallelism, and retries.

Common causes

Ordered by hit rate.

1. Signed URL TTL shorter than upload duration

The signed URL has a 300 s expiry. The user on a 5 Mbps connection needs 800 s to push 500 MB. Halfway through, S3 sees X-Amz-Date + X-Amz-Expires is in the past and rejects.

How to spot it: Response body is <Error><Code>AccessDenied</Code><Message>Request has expired</Message></Error> with X-Amz-Expires visible in the request URL.

2. Single-part PUT for a file larger than 5 GB

S3 single-part PUT has a hard cap of 5 GB. Even with TTL fixed, a 6 GB file always fails.

How to spot it: Error code EntityTooLarge.

3. Clock skew on the signer

If the signer (your backend) has a clock drifted backward, S3 sees the signature as already expired the moment it arrives.

How to spot it: 403s on every signed URL, even tiny ones; date in X-Amz-Date is already in the past.

4. CORS preflight uses up the budget

Browser does an OPTIONS preflight before the PUT. If the bucket CORS does not permit PUT, the upload never starts but the signed URL clock is ticking.

How to spot it: Network tab shows preflight 403 or no PUT request at all.

5. Signature version v2 used with SigV4-only region

Older code uses signature v2. New regions (after 2014) and KMS-encrypted buckets require v4.

How to spot it: InvalidRequest: The authorization mechanism you have provided is not supported.

Shortest path to fix

Step 1: Pick the right pattern based on file size

File sizePattern
Under 100 MBSingle PUT presigned URL, TTL 15 min
100 MB - 5 GBMultipart with presigned part URLs
Over 5 GBMultipart only

Five-minute TTL is too tight for anything user-facing. Default to 900 s (15 min).

Step 2: Single-part PUT with sensible TTL

# Python boto3 — single PUT presigned URL
import boto3
from botocore.client import Config

s3 = boto3.client('s3', region_name='us-east-1',
                  config=Config(signature_version='s3v4'))

url = s3.generate_presigned_url(
    'put_object',
    Params={'Bucket': 'uploads', 'Key': 'video.mp4', 'ContentType': 'video/mp4'},
    ExpiresIn=900,   # 15 min
    HttpMethod='PUT',
)

Front-end PUTs directly with fetch(url, { method: 'PUT', body: file, headers: {'Content-Type': 'video/mp4'} }).

Step 3: Multipart upload with per-part presigned URLs

This is the right answer for big files. Each part gets its own presigned URL; the browser uploads parts in parallel; the backend completes the upload.

# 1. Initiate
mpu = s3.create_multipart_upload(Bucket='uploads', Key='video.mp4', ContentType='video/mp4')
upload_id = mpu['UploadId']

# 2. Sign part URLs (one per chunk, e.g. 8 MB chunks)
def sign_part(part_number):
    return s3.generate_presigned_url(
        'upload_part',
        Params={
            'Bucket': 'uploads',
            'Key': 'video.mp4',
            'UploadId': upload_id,
            'PartNumber': part_number,
        },
        ExpiresIn=3600,
    )

# 3. After client uploads each part, it returns the ETag from response header
# 4. Complete
s3.complete_multipart_upload(
    Bucket='uploads', Key='video.mp4', UploadId=upload_id,
    MultipartUpload={'Parts': [
        {'PartNumber': 1, 'ETag': '"abc..."'},
        {'PartNumber': 2, 'ETag': '"def..."'},
    ]},
)

Browser uploads each part with PUT to its URL, reads ETag from the response header, sends the list back to your backend.

Step 4: Use the SDK upload manager when you can

For server-side uploads, do not hand-roll multipart — use the manager.

// Node AWS SDK v3
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { createReadStream } from 'node:fs';

const s3 = new S3Client({ region: 'us-east-1' });

await new Upload({
  client: s3,
  params: {
    Bucket: 'uploads',
    Key: 'video.mp4',
    Body: createReadStream('./video.mp4'),
    ContentType: 'video/mp4',
  },
  queueSize: 4,        // parallel parts
  partSize: 8 * 1024 * 1024,
}).done();

The manager handles chunking, retries, and abort on failure.

Step 5: Fix clock and CORS

# On the signer host
timedatectl status
sudo chronyc makestep
{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://app.example.com"],
      "AllowedMethods": ["GET", "PUT", "POST"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3000
    }
  ]
}

ExposeHeaders: ["ETag"] is required so the browser can read the part ETag for the complete call.

Prevention

  • Default TTL 15 min for single PUT, 1 hr for multipart parts.
  • Anything over 100 MB uses multipart, with parts of 8-16 MB.
  • Signer host runs NTP; alert if clock skew exceeds 1 s.
  • Always set signature_version='s3v4' and target a SigV4 endpoint.
  • For server-side uploads, prefer the SDK Upload / TransferManager over manual multipart.

Tags: #Backend #Troubleshooting #s3

Related Articles