🔥 New Update: Style Reference and Content Reference has been added to Image Guidance. Generate new images utilizing styles from an reference image easily with this new option and reference the content of another reference image.
Overview
ControlNet has remained one of the most popular features of the Leonardo.Ai platform, with many users finding it extremely useful for generating images in highly specific styles, adjusting image composition and achieving specific character poses.
Benefits
With Image Guidance, you can now fine-tune your experience with more granular control than ever before. All users will be able to access an array of Image Guidance options, while premium users will be able to access 3 exclusive options as well as upload 4 reference images simultaneously.
You can choose from a rich variety of nuanced options such as Depth, Sketch, Pose and even QR to give unprecedented control over depth perception, line art representations, pose replication, pattern interpretation, and more. Coupled with an intuitive new UI, crafting your ideal image composition and style has never been easier.
About The Image Guidance Tool:
Image Guidance gives numerous benefits, such as new tools, multiple image uploads, independent weighting, and the ability to leverage up to 4 reference image options simultaneously.
Supported Options: Style Reference, Content Reference, Depth, Edge, Sketch, Pose, Normals, Pattern, QR, Lineart, and Text Image Input.
4x Image Uploads (Premium): You can now upload up to 4 reference images and apply different ControlNet options to each, as well as adjust their weightage individually.
New UI Changes: To make it easy to manage the different Image Guidance options, we’ve added a convenient new section in the AI Image Generation page. Image2Image and Image Prompt can be managed through this new section as well.
🔥 NEW! Style Reference:
Style Reference generates an image borrowing aesthetic qualities from a reference image. See the example below.
🔥 How to use Style Reference
Within the Image Guidance tab, add or upload an image, then select Style Reference from the from drop down menu. Adjust the strength (Low-Max) if required and then click Generate. You may add up to a maximum of 4 reference images.
🧠 Style Reference Tips:
Style Reference may include unwanted characteristics as well (Such as facial characteristics). In such cases, it is recommended to reduce the strength or increase the weightage of certain characteristics in the prompt to counter the effects.
For multiple Style Reference inputs, all inputs will use the same overall strength. The amount of influence each individual input has can be adjusted with the Influence slider.
For highly stylized images, it is recommended to increase the strength to High〜Max for better stylization results.
Style Reference can be used alongside Depth-to-Image and Edge-to-Image to restyle an existing image. Please note that Image to Image cannot be used.
Style Reference can be used alongside Content Reference to generate an image of similar content in the referenced style(s).
When using Elements alongside Style Reference, it is recommended to increase the strength of the Elements more than typically is required.
It is recommended to use regular SDXL models instead of Lightning models for better quality.
🔥 NEW! Content Reference:
Content Reference generates an image while referencing the content of a reference image. Please refer to the example below.
🔥 How to use Content Reference
Within the Image Guidance tab, add or upload an image, then select Content Reference from the from drop down menu. Adjust the strength (Low-High) if required and then click Generate.
🧠 Content Reference Tips:
Content Reference typically does not transfer colors, details or texture - mainly only the general shapes of the reference image.
Content Reference can be used alongside other modes such as Style Reference or Depth to Image.
Use Content Reference alongside Style Reference to generate an image with similar content but in an different style.
It is recommended to use regular SDXL models instead of Lightning models for better quality.
You can use multiple images with Content Reference.
It is helpful to describe what you would like to see in the prompt while using Content Reference for optimal results.
Content Reference does not work well with Elements. Elements will have extremely limited effects on the final results.
Content Reference may not work well for recreating the exact likeness of characters, people etc.
Image Guidance - Feature Breakdown:
Depth to Image: Leverages depth information to enhance the three-dimensional aspects of an image. It improves the perception of depth and can be used for various 3D-related tasks such as depth-aware image manipulation and rendering.
Edge to Image: is effective at replicating the composition of an image, but changing the style and theme. It identifies the edges of objects within an image transforming it into a line art representation. This rendition subsequently shapes the final image, with the model working to fill in the identified edges - similar to a coloring page. Notably, higher strength values preserve more lines in the final result, making this option ideal for rendering or restyling existing line art images. Use this if you are looking to use the composition of the reference image.
Line Art: Another option that renders the outlines of objects in an image, this method aims to create a simple line art of the reference image that will be use as guidance for generating your image.
Edge to Image (alt): An alternative to Canny and Lineart, Holistically-Nested Edge Detection (HED) is an algorithm that is great for intricate details and outlines. It creates much smoother and noise free outlines in comparison to Canny and is better able to preserve relevant details of the reference image.
Sketch to Image: Specializing in composition, the sketch option employs a sketch or scribble as a preliminary guide for the final image. It is particularly suited for projects where a specific arrangement or composition is essential (such as architectural visualisations), acting as a foundational blueprint to shape the outcome according to your artistic vision.
Pose to Image: Focusing on the recreation of human poses, Pose scans a reference image, seeks out human (or similar) figures, and attempts to replicate their poses for the resultant image. While it’s an invaluable tool for specific character positioning, it’s worth noting its potential limitations with particularly complex poses.
Normal Map: Normal Mapping will be familiar to those in the 3D graphics space. In essence it informs a program about the surface topology (arrangement) of an object, offering refined control over its appearance and facilitating image relighting. Normal maps are akin to depth maps but benefit from a higher level of detail, making them especially useful for emphasizing the central subject matter of an image.
Pattern to Image: Pattern is adept at utilizing contrast and value to interpret and render images. Functioning optimally with high-contrast patterns (particularly black and white images), it discerns style and shape of patterns to then alter the resultant image to emulate the uploaded design, regardless of subject matter. This option is especially proficient in transforming a variety of patterns into visually compelling artwork with striking effects.
QR code to Image: This option is tailored for generating and enhancing Quick Response (QR) codes. It can optimize QR code designs, increase their readability, and adapt them to specific visual styles or branding, making them more aesthetically appealing than simply functional.
Text Image Input (Text to Image): By uploading an image of black text on a white background, this option allows you to generate stylized text art.
Please note that not all new Image Guidance tools are available for all base models. Those that cannot be used will be conveniently greyed out with hover-over instructions on necessary changes to activate them. For quick reference, below is a useful table so you can see which ControlNet works with which Stable Diffusion version, as well as which are premium options.
ControlNet | SD 1.5 | SDv2 | SD 1.5 & SDv2 + Prompt Magic V3 | SDXL (Premium) |
🔥 NEW! | ✕ | ✕ | ✕ | ✓ |
🔥 NEW! | ✕ | ✕ | ✕ | ✓ |
Edge | ✓ | ✓ | ✓ | ✓ |
Depth | ✓ | ✓ | ✓ | ✓ |
Normal | ✓ | ✓ | ✕ | ✕ |
Pose | ✓ | ✓ | ✓ | ✓ |
Sketch | ✓ | ✓ | ✕ | ✕ |
Line art | ✓ | ✕ | ✕ | ✕ |
QR | ✓ | ✕ | ✕ | ✕ |
Edge (alt) | ✕ | ✓ | ✕ | ✕ |
Pattern | ✓ | ✕ | ✕ | ✕ |
Text Image Input | ✓ | ✓ | ✓ | ✓ |
How to use Image Guidance
Read on for our quick guide on how to get started fast with Image Guidance
Navigate to the Image Generation page. Next to Generation History you will now see a new option called Image Guidance – select this.
Upload or Select a source image in the Source Image inputs. (Premium users add up to 4 source images).
Choose a selected ControlNet option from the dropdown menu. Those incompatible with the current base model will be greyed out. A ⚠️ symbol will provide hover-over instructions on how to alter options to make it available. If you are looking to use an image as an initial image, select the Image to Image option instead. And if you would like to use the Image Prompt feature (SD1.5 and 2.1) models only, enable Prompt Magic first in the left sidebar.
Select your desired weighting strength. (Note that higher weighting can make ControlNet more adherent to your base image but can produce unwanted results, so we recommend experimenting with this to find the perfect balance).
Click Generate!
Frequently Asked Questions
How many tokens do the new ControlNets cost?
Answer: Each Image Guidance Option task costs 2 tokens per task.(Please note that Image to Image does not attract an additional token cost)
Which pipelines does Image Guidance work best with?
Answer: All models and pipelines including Alchemy, PhotoReal and Prompt Magic. (Note that Prompt Magic V2 only supports Image Prompt)
Results are inconsistent, what can I do?
Answer: If you aren’t getting the image results you want, we suggest adjusting the weight, or selecting a different reference image.
I'm getting an 'Invalid init image id' error. What do I do to resolve this?
Answer: You can either navigate to the 'Image Guidance' tab and remove any images that you've uploaded or on the left menu bar, scroll down and click 'Reset to defaults'. If this doesn't work, please reach out to support@leonardo.ai
Feedback
We know that Image Guidance has been a hotly anticipated feature and we are looking forward to seeing what everyone will create with it! That said, these features are new, so we would love to know if you run into any issues or have any other general thoughts about the updates.
Please feel free to leave your feedback here.