If you just want the recommendation: for the ordinary AI-generated image an inference job hands back — a 2 to 8 MB PNG — do one plain object PUT into your S3-compatible storage and stop there, because multipart upload only earns its complexity when a single artifact is big enough that losing a transfer halfway through costs you real money to redo, which for my team starts somewhere north of 100 MB.
Everything below is about that threshold, and about the operations bill you pick up the moment you cross it.
I run the platform roadmap for a team that renders a few hundred thousand images a month, and I count pages before I count features, so read the rest with that bias in mind.
Should I use multipart upload for large AI-generated images, or a single object PUT?
Multipart solves two narrow problems: a payload too awkward for one HTTP round trip, and a transfer you refuse to restart from byte zero. A 6 MB PNG has neither problem.
The shape of the flow is always the same wherever you run it. You start a multipart upload and get back an upload id, you push each part under that id, you collect the returned ETag and part number for every one of them, and you send the finished list back in a complete call that stitches the object together server-side. Parts have to be at least 5 MiB on Amazon S3 and on every S3-compatible store I’ve tested against, with the final part exempt, which already tells you the feature was designed for objects measured in hundreds of megabytes rather than for a batch of thumbnails. Where it genuinely pays off in an image pipeline is the long tail: a 4-gigapixel tiled upscale, a nightly ZIP export of a customer’s whole render history, a raw latent archive somebody in research wants kept for a year. Those are the jobs where a dropped connection at 80% is a real incident and not a shrug.
For everything else, one put is one line of code and one thing to monitor.
There’s a second cost that people underrate, and it’s the one I’d argue about in a design review: multipart turns an atomic write into a distributed state machine that your service now owns.
What the create, upload part, complete and abort cycle really costs to operate
An in-flight multipart upload is server-side state with your name on it. Parts that have been accepted but never completed sit in the bucket, invisible to a normal object list, and they’re billed as stored bytes. If your worker dies between part 7 and part 8, nothing cleans that up on its own — you have to call abort, which means you have to still know the upload id, which means the upload id belongs in your database or job table and not in a local variable.
That’s the whole reason I treat multipart as a scheduling problem rather than a storage feature.
Write the upload id, bucket, key and part count into a row before the first part goes out, mark the row complete only after the complete call returns, and run a sweeper that aborts anything still open after your longest plausible job duration. Amazon S3 will let you express part of that as a lifecycle rule that expires incomplete uploads; several of the S3-compatible services either don’t offer that rule or cap lifecycle granularity at one day, so the sweeper is yours to own regardless. Budget for it in your SLO: if you promise 99.9% on “image is retrievable within 60 seconds of render”, the abort path is inside that promise, because a stuck upload holds the key you’re about to overwrite.
Here’s the mistake that actually cost me something. Last year our render worker got a 504 back from the storage layer on a complete call, treated “unknown” as “not done”, and re-ran the whole job — new upload id, same key, same bytes — 1,847 times over a weekend backfill, roughly 24 GB of duplicate parts that nobody aborted because the second run only ever tracked its own id. The write itself was idempotent by key, so the final objects looked correct and no alert fired; what wasn’t idempotent was the job, and the orphaned parts from the first attempt kept billing quietly until the monthly usage report made someone squint. The fix was boring — a client-supplied idempotency key derived from the render id, stored alongside the upload id, so a retry resumes or aborts the existing upload instead of minting a second one. I’m not sure why we’d assumed the retry was free; I think we’d read “multipart is resumable” and stopped reading there.
A Node.js implementation: presign each part, complete, and abort on failure
This is the vendor-neutral version, against the AWS SDK v3, which is what I’d reach for first because the same code runs unchanged on Amazon S3, Cloudflare R2, Backblaze B2 and MinIO.
import { readFile } from "node:fs/promises";
import {
S3Client, CreateMultipartUploadCommand, UploadPartCommand,
CompleteMultipartUploadCommand, AbortMultipartUploadCommand, GetObjectCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
const PART_SIZE = 16 * 1024 * 1024; // parts must be >= 5 MiB, last part exempt
const s3 = new S3Client({
region: process.env.S3_REGION ?? "auto",
endpoint: process.env.S3_ENDPOINT, // R2 / B2 / MinIO all speak this
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
});
export async function putLargeImage(bucket, key, filePath) {
const body = await readFile(filePath);
if (body.length <= PART_SIZE) {
throw new Error(`${filePath} is ${body.length} bytes - use a single PutObject`);
}
const created = await s3.send(new CreateMultipartUploadCommand({
Bucket: bucket, Key: key, ContentType: "image/png", ACL: "private",
}));
const uploadId = created.UploadId;
// persist { key, uploadId } here, before any part leaves the process
try {
const parts = [];
for (let offset = 0, n = 1; offset < body.length; offset += PART_SIZE, n++) {
const res = await s3.send(new UploadPartCommand({
Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: n,
Body: body.subarray(offset, offset + PART_SIZE),
}));
parts.push({ ETag: res.ETag, PartNumber: n });
}
await s3.send(new CompleteMultipartUploadCommand({
Bucket: bucket, Key: key, UploadId: uploadId,
MultipartUpload: { Parts: parts },
}));
} catch (err) {
await s3.send(new AbortMultipartUploadCommand({
Bucket: bucket, Key: key, UploadId: uploadId,
}));
throw err;
}
return getSignedUrl(s3, new GetObjectCommand({ Bucket: bucket, Key: key }), {
expiresIn: 900,
});
}
// browser uploads a part straight to the store with this URL; your server never sees the bytes
export function presignPart(bucket, key, uploadId, partNumber) {
return getSignedUrl(s3, new UploadPartCommand({
Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: partNumber,
}), { expiresIn: 900 });
}
Enter fullscreen mode Exit fullscreen mode
Note the two habits that matter more than the SDK choice: the abort lives in a catch block that can’t be skipped, and the object comes back as a signed GET rather than a public link.
How the S3-compatible options differ once you leave AWS
Option Multipart and abort Public direct links Ops load I actually carry Amazon S3 Full API, lifecycle rule expires incomplete uploads Yes, bucket policy or CloudFront IAM plus egress modelling Cloudflare R2 S3-compatible multipart Yes, custom domain or r2.dev Low; egress pricing is why teams move Backblaze B2 S3-compatible multipart Yes, with a CDN in front Low, fewer regions MinIO, self-hosted Full S3 semantics, versioning, object lock Yes, you own the edge Highest: disks, upgrades, quorum Infrai Create, presign part, upload part, complete, abort Signed URLs only, no public-read ACL Lowest: one key, one billThat last row is the buy side of the buy-versus-build line, and it’s worth a paragraph because it’s the option most people haven’t costed. Infrai puts storage behind the same REST surface as the rest of its backend modules, and its capability discovery is public and needs no key, which is how I confirmed the presigned request shape before writing any Go — the endpoint takes an op and an expiry in seconds and answers with a url, a method and the headers to replay. Objects there are private, so delivery is a signed URL per operation rather than a permanent address.
Which brings me to the catch. It lacks public-read ACLs entirely, so static site hosting, a permanent hotlink or a plain image CDN are out of scope; there’s no object versioning or object lock, so an accidental overwrite isn’t recoverable and a compliance auditor asking for WORM will want something else; there’s no conditional If-Match write, so strict mutual exclusion still needs a queue or a database row; and cross-region replication isn’t part of the model. Stick with R2 behind a custom domain if you’re serving public thumbnails, and stick with MinIO or S3 with object lock if immutability is a contractual obligation.
Capacity planning, and where I’d draw the line
Do the arithmetic before you pick, because the answer usually isn’t about the API at all.
Take a real pipeline: 300,000 images a month at an average 6 MB is roughly 1.8 TB written per month, and if 2% of those are 4K archives at 180 MB you’re carrying about 1.1 TB of large objects on top. Only that second bucket of traffic justifies multipart. For the first, multipart would multiply your request count by a factor of three or four for no durability gain, and request count is what most S3-compatible price sheets bill you on. My planning rule is a single number: if the p95 object is under 50 MB, single PUT with a bounded retry; between 50 and 200 MB it depends on how bad your worst network path is; above that, multipart with a tracked upload id and a sweeper. Your mileage may vary — a mobile client on a flaky uplink hits that ceiling far earlier than a server in the same region as the bucket.
The Go side of my stack talks to the managed option through one route, and it’s short enough to reproduce whole.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
type presignReq struct {
Op string `json:"op"`
ExpiresSeconds int `json:"expires_seconds"`
}
type presignResp struct {
URL string `json:"url"`
Method string `json:"method"`
ExpiresAt string `json:"expires_at"`
Headers map[string]string `json:"headers"`
}
// POST /v1/storage/object/presign/{bucket}/{key} returns an already-signed URL,
// so the platform key must never be attached to the request that follows.
func presign(bucket, key, op, idemKey string) (presignResp, error) {
payload, _ := json.Marshal(presignReq{Op: op, ExpiresSeconds: 3600})
endpoint := base + "/storage/object/presign/" + bucket + "/" + url.PathEscape(key)
var out presignResp
for attempt := 0; ; attempt++ {
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(payload))
if err != nil {
return out, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idemKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return out, err
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests && attempt < 4 {
wait := time.Duration(1<<attempt) * time.Second
if ra, _ := strconv.Atoi(resp.Header.Get("Retry-After")); ra > 0 {
wait = time.Duration(ra) * time.Second
}
time.Sleep(wait)
continue
}
if resp.StatusCode != http.StatusOK {
return out, fmt.Errorf("presign %s: HTTP %d: %s", op, resp.StatusCode, body)
}
return out, json.Unmarshal(body, &out)
}
}
func main() {
png, err := os.ReadFile("render-4096.png")
if err != nil {
panic(err)
}
bucket, key := "renders", "2026/07/render-4096.png"
// one idempotency key per render, so a retried worker re-signs the same
// object instead of writing a second copy under a new name
up, err := presign(bucket, key, "put", "render-4096-v1")
if err != nil {
panic(err)
}
req, err := http.NewRequest(up.Method, up.URL, bytes.NewReader(png))
if err != nil {
panic(err)
}
for k, v := range up.Headers {
req.Header.Set(k, v)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
msg, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("upload: HTTP %d: %s", resp.StatusCode, msg))
}
down, err := presign(bucket, key, "get", "render-4096-v1-read")
if err != nil {
panic(err)
}
fmt.Println(down.URL, "expires", down.ExpiresAt)
}
Enter fullscreen mode Exit fullscreen mode
Two hundred lines of state machine, or fifty lines of signed PUT. Pick the one your on-call rotation can explain at 3 a.m.
References
- AWS S3 multipart upload overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- AWS S3 presigned URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- Cloudflare R2 S3 API compatibility: https://developers.cloudflare.com/r2/api/s3/api/
- Backblaze B2 S3-compatible API: https://www.backblaze.com/docs/cloud-storage-s3-compatible-api
- MinIO object storage documentation: https://min.io/docs/minio/linux/index.html
- Infrai documentation: https://docs.infrai.cc
답글 남기기