×
![]()
Nowadays, everyone across AI and related communities talks about generative AI models, particularly the large language models (LLMs) behind widespread applications like ChatGPT, as if they have completely taken over the field of machine learning.
Imagine being able to get complex insights about our planet — like the impacts of hurricanes or best locations for city infrastructure — simply by asking Gemini. Geospat…
In this post, we explore how to use Amazon Bedrock for synthetic data generation, considering these challenges alongside the potential benefits to develop effective strategies for various applications across multiple industries, including AI and machine learning (ML).
Compared to DeepSeek R1, Llama-3.1-Nemotron-Ultra-253B shows competitive results despite having less than half the parameters.
In an increasingly uncertain global market, product and growth leaders have a greater effect than ever on a company’s success, impacting not only business growth, but making business transformation possible. Amplitude’s Product 50 Awards were launched as a way to recognize and celebrate product and growth leaders globally whose innovation and creativity are shaking up […]
Currently, the model supports multiple expressive voices, both masculine and feminine, in American and British English.
A tiny, soft, flexible robot that can crawl through earthquake rubble to find trapped victims or travel inside the human body to deliver medicine may seem like science fiction, but an international team is pioneering such adaptable robots by integrating flexible electronics with magnetically controlled motion.
New open-source evaluation framework quantifies RAG pipeline performance with scientific metrics, helping enterprises cut through the AI hype cycle with objective measurements.
It almost sounds pejorative, doesn’t it? But the distinction between senior and junior software developers is built into our jobs and job titles. Whether we call it entry-level or something else, we distinguish between people who are just starting their careers and those who have been around for a while. We’re all still learning (one […]
On Tuesday, California state senator Steve Padilla will make an appearance with Megan Garcia, the mother of a Florida teen who killed himself following a relationship with an AI companion that Garcia alleges contributed to her son’s death. The two will announce a new bill that would force the tech companies behind such AI companions…
PLAID is a multimodal generative model that simultaneously generates protein 1D sequence and 3D structure, by learning the latent space of protein folding models.
The awarding of the 2024 Nobel Prize to AlphaFold2 marks an important moment of recognition for the of AI role in biology. What comes next after protein folding?
In PLAID, we develop a method that learns to sample from the latent space of protein folding models to generate new proteins. It can accept compositional function and organism prompts, and can be trained on sequence databases, which are 2-4 orders of magnitude larger than structure databases. Unlike many previous protein structure generative models, PLAID addresses the multimodal co-generation problem setting: simultaneously generating both discrete sequence and continuous all-atom structural coordinates.
From structure prediction to real-world drug design
Though recent works demonstrate promise for the ability of diffusion models to generate proteins, there still exist limitations of previous models that make them impractical for real-world applications, such as:
All-atom generation: Many existing generative models only produce the backbone atoms. To produce the all-atom structure and place the sidechain atoms, we need to know the sequence. This creates a multimodal generation problem that requires simultaneous generation of discrete and continuous modalities.
Organism specificity: Proteins biologics intended for human use need to be humanized, to avoid being destroyed by the human immune system.
Control specification: Drug discovery and putting it into the hands of patients is a complex process. How can we specify these complex constraints? For example, even after the biology is tackled, you might decide that tablets are easier to transport than vials, adding a new constraint on soluability.
Generating “useful” proteins
Simply generating proteins is not as useful as controlling the generation to get useful proteins. What might an interface for this look like?
For inspiration, let's consider how we'd control image generation via compositional textual prompts (example from Liu et al., 2022).
In PLAID, we mirror this interface for control specification. The ultimate goal is to control generation entirely via a textual interface, but here we consider compositional constraints for two axes as a proof-of-concept: function and organism:
Learning the function-structure-sequence connection. PLAID learns the tetrahedral cysteine-Fe2+/Fe3+ coordination pattern often found in metalloproteins, while maintaining high sequence-level diversity.
Training using sequence-only training data
Another important aspect of the PLAID model is that we only require sequences to train the generative model! Generative models learn the data distribution defined by its training data, and sequence databases are considerably larger than structural ones, since sequences are much cheaper to obtain than experimental structure.
Learning from a larger and broader database. The cost of obtaining protein sequences is much lower than experimentally characterizing structure, and sequence databases are 2-4 orders of magnitude larger than structural ones.
How does it work?
The reason that we’re able to train the generative model to generate structure by only using sequence data is by learning a diffusion model over the latent space of a protein folding model. Then, during inference, after sampling from this latent space of valid proteins, we can take frozen weights from the protein folding model to decode structure. Here, we use ESMFold, a successor to the AlphaFold2 model which replaces a retrieval step with a protein language model.
Our method. During training, only sequences are needed to obtain the embedding; during inference, we can decode sequence and structure from the sampled embedding. ❄️ denotes frozen weights.
In this way, we can use structural understanding information in the weights of pretrained protein folding models for the protein design task. This is analogous to how vision-language-action (VLA) models in robotics make use of priors contained in vision-language models (VLMs) trained on internet-scale data to supply perception and reasoning and understanding information.
Compressing the latent space of protein folding models
A small wrinkle with directly applying this method is that the latent space of ESMFold – indeed, the latent space of many transformer-based models – requires a lot of regularization. This space is also very large, so learning this embedding ends up mapping to high-resolution image synthesis.
To address this, we also propose CHEAP (Compressed Hourglass Embedding Adaptations of Proteins), where we learn a compression model for the joint embedding of protein sequence and structure.
Investigating the latent space. (A) When we visualize the mean value for each channel, some channels exhibit “massive activations”. (B) If we start examining the top-3 activations compared to the median value (gray), we find that this happens over many layers. (C) Massive activations have also been observed for other transformer-based models.
We find that this latent space is actually highly compressible. By doing a bit of mechanistic interpretability to better understand the base model that we are working with, we were able to create an all-atom protein generative model.
What’s next?
Though we examine the case of protein sequence and structure generation in this work, we can adapt this method to perform multi-modal generation for any modalities where there is a predictor from a more abundant modality to a less abundant one. As sequence-to-structure predictors for proteins are beginning to tackle increasingly complex systems (e.g. AlphaFold3 is also able to predict proteins in complex with nucleic acids and molecular ligands), it’s easy to imagine performing multimodal generation over more complex systems using the same method.
If you are interested in collaborating to extend our method, or to test our method in the wet-lab, please reach out!
Further links
If you’ve found our papers useful in your research, please consider using the following BibTeX for PLAID and CHEAP:
@article{lu2024generating,
title={Generating All-Atom Protein Structure from Sequence-Only Training Data},
author={Lu, Amy X and Yan, Wilson and Robinson, Sarah A and Yang, Kevin K and Gligorijevic, Vladimir and Cho, Kyunghyun and Bonneau, Richard and Abbeel, Pieter and Frey, Nathan},
journal={bioRxiv},
pages={2024--12},
year={2024},
publisher={Cold Spring Harbor Laboratory}
}
@article{lu2024tokenized,
title={Tokenized and Continuous Embedding Compressions of Protein Sequence and Structure},
author={Lu, Amy X and Yan, Wilson and Yang, Kevin K and Gligorijevic, Vladimir and Cho, Kyunghyun and Abbeel, Pieter and Bonneau, Richard and Frey, Nathan},
journal={bioRxiv},
pages={2024--08},
year={2024},
publisher={Cold Spring Harbor Laboratory}
}
You can also checkout our preprints (PLAID, CHEAP) and codebases (PLAID, CHEAP).
Some bonus protein generation fun!
Additional function-prompted generations with PLAID.
Unconditional generation with PLAID.
Transmembrane proteins have hydrophobic residues at the core, where it is embedded within the fatty acid layer. These are consistently observed when prompting PLAID with transmembrane protein keywords.
Additional examples of active site recapitulation based on function keyword prompting.
Comparing samples between PLAID and all-atom baselines. PLAID samples have better diversity and captures the beta-strand pattern that has been more difficult for protein generative models to learn.
Acknowledgements
Thanks to Nathan Frey for detailed feedback on this article, and to co-authors across BAIR, Genentech, Microsoft Research, and New York University: Wilson Yan, Sarah A. Robinson, Simon Kelow, Kevin K. Yang, Vladimir Gligorijevic, Kyunghyun Cho, Richard Bonneau, Pieter Abbeel, and Nathan C. Frey.
Colossal Biosciences, the de-extinction company, has announced that it has brought the dire wolf back from extinction.
Stanford's annual AI Index Report offers a comprehensive view of the AI landscape. Here are the most critical insights for enterprise IT decision makers.
Llama 4 continues to spread to other inference providers, but it's safe to say the initial release has not been a slam dunk.
Today, we’re excited to announce the availability of Llama 4 Scout and Maverick models in Amazon SageMaker JumpStart. In this blog post, we walk you through how to deploy and prompt a Llama-4-Scout-17B-16E-Instruct model using SageMaker JumpStart.
OpenAI phone, AI case studies, Stanford report, AI Mode, agentic AI, and more...
Rescale secures $115 million in Series D funding to accelerate AI physics technology that speeds up engineering simulations by 1000x, backed by tech luminaries including Bezos and Altman.
This post demonstrates how Amazon Bedrock Knowledge Bases can help you scale your data management effectively while maintaining proper access controls on different management levels.
This post is divided into five parts; they are: • Recommendation Systems • Cross-Lingual Applications • Text Classification • Zero-Shot Classification • Visualizing Text Embeddings A simple recommendation system can be created by finding a few of the most similar items to the target item.
Google Arts & Culture has launched new AI experiments to help you explore hidden artwork details, stage lighting design and more.
New research from Stanford suggests artificial intelligence isn’t ruled by just OpenAI and Google, as competition increases across the US, China, and France.
Learn how the 2025 Growth Academy: AI for Health cohort is building the next generation of healthcare solutions.
OpenAI courses, Realtime Computer Autopilot, Master Chef AI, MCP, and more...
For the first time, we may have the computing power and the intelligence to tackle problems with AI that were once beyond human reach.
How do we cut through the hype and understand what AI agents can truly accomplish and, more importantly, how we should use them?
Llama 4, AI Quake II, Hinton on AI, AI thoughts, lipsyncing, compliance, and more...
While DeepSeek R1 and OpenAI o1 edge out Behemoth on a couple metrics, Llama 4 Behemoth remains highly competitive.
To contextualize DeepSeek’s disruption, let's consider the broader shift in AI being driven by the scarcity of training data.
While the White House carved out a narrow exemption for some semiconductor imports, President Donald Trump’s sweeping tariffs still apply to GPUs and chipmaking equipment.
This post is divided into three parts; they are: • What Is Auto Classes • How to Use Auto Classes • Limitations of the Auto Classes There is no class called "AutoClass" in the transformers library.