There is no setting that makes a model produce the same person twice. Identity has to be supplied as conditioning, and the reliable procedure is a loop: generate with a reference adapter, keep only the outputs that agree, train an adapter on those, and regenerate. This page is that loop.
The short answer
A diffusion model has no persistent state and no concept of a particular person. Every generation is an independent sample. To get the same character twice you have to put the identity into the conditioning — through a reference image encoded by an adapter, or through weights trained on that character, or both.
The methods form a ladder of increasing cost and increasing reliability, and the useful part is knowing where on it to stop for what you are doing.
The ladder, cheapest first
Method Description 1. fixed seed and prompt Free. Reproduces a composition, not a person. Useful only when nothing about the pose or framing changes, which is almost never the requirement. 2. a heavily specified prompt Free. Fixes the describable attributes — age, hair, eyes, clothing, build. Fails on the parts of a face nobody has words for, which is exactly the part that carries identity. 3. image-prompt adapters No training. An image encoder embeds a reference image and injects it through additional cross-attention. Good identity transfer, but it carries the reference’s pose, lighting and framing along with the face, and the strength dial trades identity against prompt adherence. 4. face-specific identity adapters No training. Uses a face-recognition embedding rather than a general image embedding, so the injected signal is closer to identity alone and less entangled with style. Better identity preservation at a given strength; applies only to faces. 5. a trained LoRA Hours of work and a GPU. The most reliable, the most controllable, and the only one that carries non-facial features — a specific costume, a body type, a creature that is not a person. Also the only one that can overfit.The mechanics of the fifth are in LoRAs for image models. The rest of this page is the procedure that makes it work without a photoshoot.
Why a fixed seed is not enough
The seed fixes the initial noise tensor, whose low-frequency structure survives into the composition. That is why a fixed seed with an edited prompt often keeps the layout. But identity is decided by the prompt embedding and the model’s prior, not by the seed, and the moment you change the pose or the scene the trajectory diverges from the first step.
The mechanism is set out in seeds and determinism. The practical summary: a seed is a control for comparison, not a control for identity. It is still essential to this workflow, because without a fixed seed you cannot tell whether a change came from your adapter or from the draw.
The bootstrapping loop
The circular problem is that a character LoRA needs 20 to 40 images of a character who does not exist. The way out is to generate the dataset with a weaker method and then train on the agreeing subset.
- Establish the reference. Generate until you have one image of the character you are happy with. Record its seed and every parameter. This one image is the anchor for everything after it.
- Generate a wide sheet with a reference adapter. Using the anchor as the reference image, generate 100 to 200 images across varied poses, angles, expressions, lighting and backgrounds. Vary the seed, not the reference. Adapter strength around the middle of its range: high enough to carry identity, low enough that the prompt still controls the pose.
- Select ruthlessly. Keep only images where the face is the same person. Twenty to forty that genuinely agree beat eighty that mostly agree, because the training will average whatever disagreement you leave in. Look at them together, small, in a grid; differences that are invisible one at a time are obvious in a contact sheet.
- Check the variety of what you kept. Count how many are three-quarter view, how many are smiling, how many have the same background. Any attribute present in most of the set will be learned as part of the character. Go back and generate more of what is missing rather than training on a lopsided set.
- Caption them. See the rule below; it is the step that decides what the trigger word ends up meaning.
- Train a LoRA. Settings in the section below.
- Regenerate and compare against the anchor. Same prompts, same seeds, LoRA instead of adapter. If identity is stronger and variety is preserved, it worked. If every output has the pose of your training set, go back to step 4.
- Iterate once. The LoRA from round one produces better and more varied training data than the adapter did. One more round with that data usually helps; a third rarely does, and risks compounding whatever the first round got slightly wrong.
The whole loop rests on step 3. Selection is what converts a generator that is inconsistent into a dataset that is consistent, and it cannot be automated away by a similarity threshold — face-similarity scores are happy to accept two images your eye rejects immediately.
Building the dataset
What varies and what does not
- Must vary: pose, camera angle, distance, expression, lighting direction and colour, background, and crop. Anything held constant across the set is learned as part of the character.
- Must not vary: the identity itself. Hair colour, eye colour, distinguishing marks, apparent age. If two images disagree, at least one of them is teaching the wrong thing.
- Decide deliberately: clothing. If the character always wears one outfit, keep it constant and it becomes part of the character. If you need to change clothes later, vary it and caption it, or train the outfit as a separate adapter.
- Composition of the set: roughly half at portrait distance where the face is large, a quarter closer, a quarter further out including at least a few full-body images. Faces need latent cells, for the reasons in aspect ratio and training buckets, so a set of distant shots teaches a face the model never saw clearly.
The captioning rule, and why
Caption everything that varies. Do not caption what is invariant.
The rule follows from what training does. Whatever appears in the images
but NOT in the caption gets absorbed into the trigger token, because the
trigger is the only thing left to explain it.
Good — the trigger absorbs the identity, the caption explains the rest:
zk7person, three-quarter view, soft window light, blue jumper,
indoors, neutral expression
Bad — the caption describes the identity, so the trigger absorbs the
background and the pose instead:
zk7person with green eyes and dark curly hair, standing in a kitchen
In the second case, prompting for zk7person will keep producing kitchens,
and changing the eye colour in the prompt will change it, because you
taught the model that eye colour is a free variable.
Trigger token: pick something the tokeniser will not split into familiar
pieces and that has no existing meaning. "zk7person" is a reasonable
shape. A real name is the worst choice, because the model already has
strong associations with it that will fight the training.
Enter fullscreen mode Exit fullscreen mode
Training settings, and why each one
These are starting points with the reasoning attached, not measured optima. Tune from here; the reasoning tells you which direction to move.
Setting Description rank 16 to 32 A character needs more capacity than a style because it must place specific features, but the update is still a consistent shift applied everywhere. Above 64 the extra capacity mostly buys memorisation of your training backgrounds. See the rank arithmetic in the LoRA page. alpha equal to rank Makes the adapter run at full trained strength at a user weight of 1.0, so the slider means what people expect. Anything else is a source of confusion later. 20 to 40 images Below about 15 the model cannot separate the identity from the circumstances of the shots. Above about 50 you are usually adding images that agree less well, and disagreement in the set is what produces a blurry average. repeats and epochs over raw step count Think in passes over the dataset, not steps. Roughly 10 to 20 passes over a 25-image set is a sensible band. Stating it this way keeps the setting meaningful when the dataset size changes. save intermediate checkpoints The single most useful habit here. Save every few passes and compare them, because the best checkpoint is usually not the last one, and identifying the point just before overfitting is far easier than predicting it. train at the model’s native resolution Training at 512 for a 1024-base model teaches the adapter at a scale the model does not generate at. Use the buckets the base was trained on. hold out three prompts Three prompts that are nothing like the training set — a different setting, a different activity, a different framing. Generate them from every checkpoint. Overfitting shows up here first, as those prompts starting to look like the training data.Failure modes and their causes
Symptom Description every image has the same pose The dataset was not varied enough in pose, so pose was learned as part of the character. Regenerate a more varied set; no amount of prompting will override it. the training backgrounds keep appearing Same cause, or the backgrounds were not captioned. Caption what varies. If it is already trained, try an earlier checkpoint or a lower weight. identity is there but quality dropped Trained too long, or applied at too high a weight. Drop the weight to 0.7 and see whether identity survives; if it does, the adapter is over-strong rather than over-trained. the face changes with distance The training set was all at one distance. A face at 64 pixels and a face at 400 pixels are different problems for the model, and it only learned one. identity drifts across a sequence Each image was conditioned on the last, so error accumulated. Condition every image on the same fixed anchor instead of on its predecessor — the same drift mechanism as chunked video generation. the character fights a style LoRA Two additive updates at full strength. Halve both. The arithmetic is in the stacking section of the LoRA page. the pose is right but the person is wrong Structural conditioning fixes geometry, not identity. A pose signal and an identity adapter are complementary and you need both — see structural conditioning.For fixing one bad face in an otherwise good image, do not regenerate: mask it and inpaint at crop resolution with the adapter active. The capacity argument for that is in inpainting and outpainting, and it is a much better use of compute than another draw.
One thing not to do
This procedure works just as well on a real person as on an invented one, from images taken from the internet, and that is the obvious application it should not be used for. Training an identity adapter on somebody who has not agreed to it produces a tool for putting them in images they never consented to, and in many jurisdictions it engages image-rights, data-protection and likeness law regardless of what the output is used for.
If the character is a real person, get written permission and keep it. If the character is invented, keep the anchor image and the dataset, because being able to show where a likeness came from is worth having the first time somebody asks.
답글 남기기