Social media is more than just text: images are the lifeblood of many platforms, and as they say, a picture’s worth a thousand words! But if a picture is worth a thousand words, analysis of a picture must be worth a whole lot more!
Pulsar has always been at the forefront of image analysis in the social listening space, and with our latest improvements by way of Image Captioning, our analysis is better than ever. Image captioning provides you a way to generate captions that describe images you ingest, allowing you to see aggregate data, filter for content, and more.
Image Captioning lets you get a quick understanding of how images define the landscape of your search.
Some of the best cases for using image captioning include things like:
Content Moderation: Generate captions for images, and filter out nsfw or improper content for your search by using our filter to search on specific terms in captions.
Assistive technology: Generate captions for visually impaired users, and include these as part of your reporting as captions will come along with images.
Multimedia indexing and retrieval: We’ve leveraged the models’ ability to generate accurate captions to improve searchability and discovery in TRAC searches, including text logos. You can use a full boolean-based text query to search for content in images via the captions generated.
Creative applications: Generate novel and interesting captions, great for creative workflows and generative art projects.
Connecting Similar content: Find posts and content that share similar images by querying for very detailed and nearly identical matches in captions. This means clustering content would no longer just be based on what’s said, but also what’s shown!
Image analysis must be enabled on your domain to be able to add any of our image analysis to your search. If you don’t have Image analysis enabled, contact your account manager to see what options are available for you!
If you already have image analysis enabled on your domain, you can enable Image Captions on your search. To Enable Image Captions on your search; You can select the Image Caption option at search setup on the Analysis step. If you wish to add it to an already existing search, you can choose to edit your search, and the option will be available on the analysis tab like so:
When enabled on your search, you'll find access to a new page called Images, located under Content Insights.
Here, you'll find access to a few unique charts that break down what the images in your searches contain the most based on their generated captions.
And clicking on a point in the cluster will let you see the images with captions that make up that datapoint.
If you want to look for images with specific captions in your search, you can do that too! Just pop open the filter box and go down to the analysis tab. Here you have the option to input Boolean to look for images with captions that match your query
How does Image Captioning work?
Our Image captioning model uses a Language-Image trained model designed to detect contents of an image and describe them in a sentence form.
In some rare cases, an image may not be captioned if the system determines it is unable to detect an appropriate detail of what the image is depicting.
Captions are currently all generated in English, and as such the treemaps and clustering charts will all have their terminology in English.
What About the Older Image Analysis Models?
The new model, though more consolidated, operates with significant improvement over our old models. There’s a few major benefits we have gained as a result of this switch:
Image analysis is no longer topic specific, allowing bigger picture understandings of image content across searches.
Content with multiple images has analysis for all images, not just the first/header image.
Rather than relying on a series of tags, the image caption is instead a more logical sentence-based description of the content, making it easier to understand how the machine understood the image.
Image captioning has a more clear tie to the content by displaying on top of the images ingested, rather than being in a separate tag field.
The general model is great ,but in the future we do hope to be able to create more specific vertical based analyses, that serve to focus on captioning the content on the focal point of the vertical.
Image Captioning starts a new era of Image analysis on Pulsar, with hopes to add even more options and specific vertical analysis in the future.
Interested in having a particular vertical represented, or any ideas for future image analysis options? Reach out to us on Intercom!