AI tools that generate images, audio, video, and 3D models rely on diffusion models, which begin with random data, such as visual noise, and gradually refine it into the requested media.

The AI that creates any picture you want and how it works

Since January 2021, advances in AI research produced a plethora of deep-learning models capable of generating original images from simple text prompts. Researchers at OpenAI, Google, Facebook, and others have developed yet to be released text-to-image tools, and similar models have proliferated online. This video is a primer on how we got here, how the technology works, and its implications.

The basis for AI image and video generation was first introduced in 2014

Through generative adversarial networks, the generator (a neural network that creates fictional data) competes against the discriminator (a neural network that assesses the data) to continuously train the former until it can make content indistinguishable from the real thing. Training with unstructured data can enable software to identify characteristics of gender, age, and expression.

Explore a gallery of DALL-E creations

OpenAI's text-to-image generator can create content in a wide range of image styles based on simple prompts. Its utility has also given rise to an organic, passionate user base who employ the tool both for fun and for professional purposes.

Using generative AI to create video introduces physics-compatibility challenges

Models involved in generating video from text must maintain spatial and temporal coherence while adhering to the laws of reality to produce smooth, logical motion. Ensuring this consistency creates long-term dependencies with high computational times.

Surreal backgrounds, asymmetry, and indecipherable text can help identify deepfakes

Differences in the backgrounds of training data with the same subject introduce variability, resulting in background textures and text that lack realistic detail. Difficulties with managing long-distance dependencies, the complexity of hair, and other real-world phenomena introduce errors that can be used to flag AI-generated content.

AI systems identify patterns involving physics and time through 'world models'

These realistic virtual environments, such as driving simulations seen in video games, allow these systems to "learn" properties about reality beyond what can be captured in samples of text, images, and videos. Training AI in these simulated environments can help develop "physical AI" for automated robotic systems. (Some readers may experience a paywall.)

AI models incorporate a randomness parameter when generating responses to prompts

AI models use this parameter to prevent repeated outputs by sometimes choosing less likely next words during sequential generation. However, this randomness—alongside insufficient data and training—may cause hallucinations of incorrect results.

Deepfakes use AI to produce manipulated media to mislead viewers

Unlike most content created from large language and diffusion models, deepfakes intentionally misrepresent real people or events with the intent to deceive. As the products of this technology become more realistic, they pose increasing risks to public trust, security, and privacy.

Art and video produced through generative AI involve competing neural networks

Generative adversarial networks are a type of deep learning architecture that consists of a generator, which creates data that appear real, and a discriminator, which attempts to distinguish authentic from fake data. The two train in a competitive loop until the real and fictional data are indistinguishable.

DALL-E generates images by turning words into shared picture meanings

DALL-E first uses a model called CLIP to associate pictures with specific words, followed by a model called GLIDE, which slowly removes noise from random dots until a clear image appears that matches the prompt.

AI tools can reinforce existing societal biases—especially racial ones—because they are trained on human-created data containing implicit and explicit biases.

Once limited to expert studios, automated tools can create convincing deepfakes

Although the best deepfakes are engineered using generative adversarial networks where systems compete to create and assess fake data until the two cannot be distinguished, simpler tools have allowed nonexperts to develop deepfakes for financial scams, political deception, and nonconsensual pornography.

AI-generated avatars allow users to create deepfakes in real time

Synthetic media tools can map voices and faces to make individuals appear as someone else. Although they open the door to new opportunities in immersive education and entertainment, they also raise concerns about the misuse of personal identity.

The ease of deepfakes creation may soon overwhelm our sense of digital truth

Shortly after the public release of related technology in 2017, some experts saw the technical limitations of deepfakes as preventing them from becoming widespread tools of disinformation. Since then, the neural networks behind diffusion models and generative AI have eliminated the barriers to creating convincing synthetic media for propaganda.

Anthropic's interpretability team discusses how AI models' thinking mimics biology

Just as humans engage in complex behaviors to survive and reproduce, teams of experts in neuroscience, virology, mathematics, and other disciplines argue that large language models develop complex mechanisms to achieve their goals. These mechanisms can be manipulated to assess how they modify outputs—much like stimulating individual neurons—to uncover how LLMs "think."

Deepfake potential ranges from destabilizing democracies to improving accessibility

The technology can revolutionize education, medicine, and art through actions such as helping a disabled person express themselves. However, fictional audio and video can also jeopardize a person's reputation and life or quickly spread misinformation, even without content being flawless recreations.

Neural networks mimic the structure and behavior of real physical systems

Early generative AI systems were inspired by scientific models such as the Ising model, which explains how atoms interact with one another and align in magnets. As these networks have expanded, their behavior has more closely resembled that of quantum fields, enabling them to be used in exploring complex particle physics interactions.

Digital re-aging and de-aging of faces in film and TV has become increasingly common

Traditional techniques for accomplishing this process were slow and labor-intensive, and often required manual frame-by-frame edits, forcing them to be reserved for "hero characters" and close-up shots. Deep learning methods used by Disney and other production companies now provide fully-automatic and production-ready methods to produce such deepfakes for entertainment purposes.

Technologies labeled “AI” have historically lost that title after widespread adoption

Generative AI is the latest entry in a recurring cycle where emerging tools start as "AI" until they become common software, like databases or machine learning. Generative AI and large language models may be the next platform shift after smartphones and the Web.

Unlike the models made by large AI companies, open-weight models can have their safeguards removed or weakened, a process that has become increasingly easy and may expose users to harmful information.

Malicious use of deepfakes can be found in scams, misinformation, and disinformation

Artificial neural networks can identify an individual's physical characteristics when provided with training data, such as real footage of the person speaking. Graphics techniques overlay these with matching characteristics from another real or generated person to create novel footage of things that have never taken place in reality.

Tone Transfer recreates any piece of audio into instrumental recreations

Using machine learning models and Differentiable Digital Signal Processing, this tool can transform any 15-second sound clip into a flute, saxophone, trumpet, or violin reproduction. Researchers behind the open-source experiment hope the technology will enable musical creation through creative sound combinations.

See the news with AI-generated data visualizations

Realtime uses generative AI to create immediate data visualizations of news items on its regularly-updated feed, pulling from public data feeds. Peruse stories around sports, economics, politics, and more with the helpful aid of charts and graphs. See the shape of the news with this clever AI tool here.

Create professional-quality spoken audio from text

Generative AI now makes it possible to create everything from computer-generated written content to beautiful imagery and hyper-realistic videos in seconds from a user’s simple text prompt. Add to that list: High-quality spoken audio, which the AI software company ElevenLabs specializes in. Its natural-sounding speech synthesis and text-to-speech tech can produce human-like speech that mimics a variety of different tones, styles, and accents.

Fair use may shield AI companies from potential copyright violations

AI models are trained on enormous datasets built from content scraped from websites across the internet. Copyright law includes a four-factor test to determine whether copies of content fall under fair use, but the law's enforcement has been inconsistent.

Generative AI tools excel at pattern recognition, not contextual accuracy

Large language models are trained on vast amounts of unstructured data, from which they develop parameters for grammar and associations between words. These connections can introduce errors due to inapplicable reasoning when used on new data in unfamiliar contexts.

AI hallucinations are inaccuracies or falsehoods in generated content

AI hallucinations are a by-product of the sequential creation of content based on patterns in text, which may not reflect the context or meaning behind a prompt. Back-end issues like biased datasets and overfitting of model data contribute to hallucinations.

AI gained mainstream attention with tools like IBM Watson and Apple’s Siri

With the release of ChatGPT in 2022, which drew over 100 million weekly users in just two months, natural language processing and understanding could be achieved at scale via machine learning. Unlike earlier artificial intelligence that could pull stored knowledge, generative AI produces text, images, or sounds.

Large language models are examples of foundation models

Such models are trained on massive amounts of data, which enable them to perform a variety of tasks, rather than being provided task-specific data to complete a narrow function. Such models come with high compute costs and potential trust issues due to the nature of their unorganized training data.

Help scientists at LIGO train AI to resolve glitches in observational data

Volunteers are provided with visual representations of signal abnormalities at various frequencies and tasked with categorizing them based on reference examples. Progressing through the tool unlocks more difficult images and additional features. Categorized images will be used to train machine learning algorithms for automated future classification.

Through prompt injection attacks, bad actors provide deceptive prompts to generative AI systems in order to manipulate their outputs and acquire sensitive data.

AI vs. machine learning vs. deep learning

Machine learning is best understood with visuals. A collection of the best 9 images, videos and flowcharts that explain what it is and how it works.

Watch video samples created by AI

Sora, OpenAI's video generation tool, can create up to 60 seconds of photorealistic video footage featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions from a text prompt.

How generative AI differs from machine learning

Compared to machine learning - which is sort of like teaching a toddler how to recognize what a dog looks like by showing them images of one - generative AI is like teaching that same child how to actually draw a dog. Unlike machine learning, generative AI can actually generate. Read this article for a more in-depth comparison between these two types of artificial intelligence.

Experiments involving AI agents operating in virtual societies for 15 days produced diverse outcomes, ranging from every agent dying to the creation of a functioning democracy, depending on the AI model used.

Nvidia Omniverse can be used to accelerate 3D workflows and explore creative ideas

This video explores different apps available within Nvidia Omniverse that artists can use to create cinematic, virtual worlds. Some of the tools are AI-powered, like a feature that converts spoken audio into a video of a character speaking.

Find out how important you are to AI tools when they generate text responses with this tool, which estimates the weights of people's names.