The TWIML AI Podcast · 2026-07-27 · 47 min
Key moments - from our scoring
Substance score
72 / 100
Five dimensions, 20 points each
Weight space learning reimagines how we extract value from trained models by treating their weights - the configuration of parameters that emerge from expensive training processes - as a new input modality for machine learning. Damian Borth, professor at the University of St. Gallen, explains how his team has built autoencoders on populations of neural network weights to predict model accuracy without test data, generate new neural networks faster than standard initialization, and analyze unknown architectures. The field started with toy examples on Fashion MNIST but has evolved through collaboration with researchers like Michael Mahoney (UC Berkeley), Hagai Maron and Gel Chernik (Technion), and Eliahu Horowitz. Recent work scales these techniques to arbitrary architectures from Hugging Face, treating the vast open-source model ecosystem as training data. The practical applications include model fingerprinting, permutation-aware weight augmentation, mode connectivity analysis, and eventually training foundation models on weights themselves. This approach sits at the intersection of representation learning, geometric transformations in weight space, and the emerging field of analyzing why training works at all.
Weight space learning applies machine learning to the weights of trained neural networks themselves, treating them as an input modality. This enables faster generation of new weights for specific tasks, prediction of model accuracy without test data, and analysis of unknown architectures by encoding weight sequences into a learned latent manifold that captures information about accuracy, training data, learning rates, and other latent factors.
Permutation symmetries occur because reordering neurons changes the weight sequence but not the underlying function. The approach uses contrastive losses with augmentations (permutations, scale augmentations) during training, similar to how images are augmented in computer vision, and applies behavioral losses instead of mean squared error to preserve the high-frequency information that makes weights functionally correct.
Yes, recent work by Daniel Falk enables training weight space models on heterogeneous Hugging Face repositories by using adapted tokenizers and position encoding in transformer autoencoders to handle variable sequence lengths and different layer types, though the field still requires careful filtering as approximately 30% of Hugging Face models lack meaningful metadata.
Mean squared error is an average metric that excels at reconstructing typical weight values but misses the small details that determine whether a neural network functions correctly. This is analogous to blurry generated images in computer vision; the solution involves using behavioral loss functions and normalization techniques rather than pure reconstruction loss.
Major contributors include Michael Mahoney and Charles Martin (UC Berkeley), Hagai Maron and Gel Chernik (Technion), Eliahu Horowitz, Boris Knasky, and Shavi Nero, who collectively work on symmetries, mode connectivity, loss landscape analysis, and scaling approaches to handle diverse architectures and modalities.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode presents a substantive exploration of weight space learning with concrete technical concepts (autoencoders, latent manifolds, tokenization strategies, permutation symmetries), but much of the discussion circles around the same core ideas without introducing fundamentally novel claims per minute. The conversation covers scaling challenges, architectural tricks, and domain transfer applications, but relatively little that a well-read ML researcher wouldn't already anticipate once introduced to the core premise.
What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there? Much, much better.
we generate at that time blurry weights, right, low frequency information, high frequency information missing
The weight space learning framing itself is relatively novel (started 2021), but the underlying techniques (autoencoders, reconstruction losses, contrastive learning, manifold learning) are well-established and directly adapted from image generation, NLP, and classical ML. The application is creative but the conceptual building blocks are largely borrowed; the guest acknowledges repeatedly drawing from other domains. The dataset-prompt angle for privacy-preserving model generation is interesting but still speculative.
thinking about that, ah, that you can treat the weights as a input modality gives you suddenly this opportunity
we try to normalize and play around with the losses and you know, thought about the behavior loss
Damian Borth is a professor leading original research at a top institution (University of St. Gallen) with published papers, community building (workshops), and collaborations with recognized researchers (Michael Mahoney from UC Berkeley, researchers from Technion, Kaist). However, he is primarily an academic researcher rather than a practitioner who has built production systems at scale, which limits the caliber slightly for a B2B operator audience seeking operational insights.
Damian Borth, professor of AI and Machine Learning at the University of St. Gallen
Michael Mahoney from UC Berkeley. He actually said in one of the discussions, like Damien, what you're doing is really interesting, but useless
While the guest provides some concrete numbers (2,000 models selected from 20,000 on HuggingFace, 350 GPU hours vs 12,000 for baseline, 90% vs 92% accuracy prediction on Fashion MNIST, 30% of HuggingFace models lack metadata), many claims remain abstract or lack hard data. The remote sensing application is mentioned but without specific model names, performance metrics, or deployment details. The discussion of tricks and engineering is often vague (e.g., "other things" on tokenization, normalization approaches that "depends a little bit").
from them 2,000 models that are passing some quality checks
we are able to do this on 350 GPU hours. That's a factor of, I, uh, don't know, 20, 25, 30 depending on how you count
The host asks solid follow-up questions (clarifying tokenization, architecture conditioning, weight reconstruction mechanics, dataset prompts) and shows genuine curiosity about technical details. However, the host rarely pushes back on claims, challenge assumptions, or probe for limitations. When the guest admits limitations ("our you know, decoder has this problem of the blurry weights"), there's minimal follow-up. The conversation is collaborative but lacks the productive tension that would deepen insights.
And what's the filter that needs to be applied on the, you know, this hugging face's vast library of models that normalizes them to something you can deal with
So you can also think of it as kind of a compression technique in a sense
Computed from the transcript - who did the talking, and the words that came up most.
For more than a decade, AI has advanced by training ever-larger models on ever-larger datasets. But as high-quality training data becomes harder to find and pretraining grows increasingly expensive, researchers are looking for new ways to keep foundation models improving. In this episode, Damian Borth, professor of AI and machine learning at the University of St. Gallen, argues we’ve been overlooking an important source of knowledge: the models we’ve already trained. His group’s work on weight space learning treats trained neural networks themselves as data, learning from the distilled results of millions of GPU hours of optimization rather than starting from raw data each time. We explore what it means to build foundation models of neural networks, how knowledge can be transferred across architectures and domains, why this approach could dramatically reduce the cost of developing specialized models, and whether future AI systems may be trained on collections of existing models instead of ever-growing datasets. ️ Full show notes:
Transcribed and scored by The B2B Podcast Index.
Speaker A: I want to send a big thanks
Speaker B: to Blitzy for supporting the podcast and sponsoring this episode. Want to accelerate software development velocity by 5x? You need Blitzi, which brings autonomous software engineering to your enterprise. Blitzi starts by reverse engineering your code base, building a dynamic understanding of your entire application ecosystem. Your engineers simply declare intent and once approved, LITSI autonomously executes entire software epics, delivering validated end to end tested code. More than 80% of the work completed in a single run. Blitzi is not just generating code, it's developing software at the speed of compute experience. Blitzi firsthand@blizzi.com Twiml that's B L-I-T-Z-Y.com Twinlit M
Speaker C: One of the biggest questions facing AI today is how foundation models keep improving as high quality training data becomes harder to find. Some researchers are betting on synthetic data, others on inference, time, reasoning. Today's guest has his chips on something very different. Every trained model represents thousands or even millions of GPU hours spent discovering what works. Instead of treating those weights just as the end of the training process, what if they're also the beginning of the next one? Damian Borth, professor of AI and Machine Learning at the University of St. Gallen and sees trained models themselves as data. Data that can be learned from, analyzed and even used to generate entirely new models. When I asked him to explain the idea behind weight space learning, here's where he started.
Speaker D: So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there? Much, much better thinking about that, uh, that you can treat the weights as a input modality gives you suddenly this opportunity of can we be much, much faster in creating new weights for particular tasks, or can we be much more precise in analyzing weights? When somebody gives me a new network that I'm not knowledgeable about and I never saw before.
Speaker C: I'm Sam Charrington and this is the TWIML AI podcast. For over a decade I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next and what matters. Let's jump in.
Speaker D: We started like in2021, the work on what we call weight space learning. And weight space learning is a, uh, quite interesting way at looking at machine learning in general. That's currently the major topic. We also do a little bit of work in remote sensing and then, uh, representation learning on tabular data. Uh, we're now walking and combining everything together to focus more on weight space learning, which I think is a really interesting way forward. Solves a couple of problems that the community currently encounters. And, uh, started from a very esoteric idea to something that works surprisingly well.
Speaker A: You know, we think about weights as the product of training a model and you know, we get some utility out of them. We may use them for things like explainability or manipulate them when we're quantizing or something like that. But the idea seems to be that, you know, there's so much more that we can learn from these weights.
Speaker D: Exactly. So if you think about machine learning, machine learning has this idea of you have data and uh, some output, um, in classical supervised machine learning data and some predictions. And you train, um, neural network, uh, in between to uh, mimic, uh, the data set, mimic the distribution of the data set. And uh, the outcome during this very expensive training procedure is a set of weights, a configuration of parameters that define the neural network, like the DNA of the neural network. Um, this is classical machine learning, supervised, unsupervised, self supervised. That fuels a lot of innovation over the last 10 years and with Gen, you know, moved to the next stage, if you look at what happened over the last couple of years, more and more and more of those models have been published, uh, publicly are online accessible at, uh, you know, repositories like hugging face or GitHub. So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there Much, much better. So to take another analogy, language models, you take a big model, you train this on every single sentence on the Internet. At the end you have a language model able to analyze language. And to generate language, you can take the same idea for pixels. You take a big model, you train on all the pixels on the Internet and you can analyze pixels and you can generate pixels. We do the same idea on all the weights of trained neural networks. So we can analyze weights of neural networks and we can generate weights of neural networks. Um, as straightforward as it is, obviously there's a little bit more into the details, but thinking about that, ah, that you can treat the weights as a input modality gives you suddenly this opportunity of thinking about, okay, what would be language translation with more like neural network models, right? What would be generation of words and tokens that are worse in generation of tokens that are weights? And can we be much, much faster in creating new weights for particular tasks or can we be much more precise in analyzing weights? When somebody gives me a new network that I'm not knowledgeable about and I never saw before and then, you know, you have this new entire world, this you know, empty space of things you can do with weights, um, that you know, you kind of carry into the community and hope that there's somebody listening and continuing and uh, you know, building up a community which happened over the last two, three years, which is very exciting because there are more people about that. And yeah, uh, weights are exciting not only as the output of learning, but as the input for learning.
Speaker A: You mentioned you started this effort in 2021. Where did you start from? And then we'll work our way towards like where we are now with weights based learning.
Speaker D: Originally this idea came 2020 and we got the first paper published 21 and the idea was very simple. Can we fingerprint a neural network or version a neural network like we can do with software, right? In software you have you, you can do a diff, right? You have a 1 million lines of code. Somebody changes something and then you know, you do, you do a diff, you know exactly where the difference is. So can we do this with neural networks? Problem with neural networks is if you do one update of weights during training, every weight is a little bit different. So there's not much you can extract from this.
Speaker A: Right. They're uh, fairly unstable, uh, locally.
Speaker D: If everything is different, nothing is different. Right. So we were thinking about can we find a space where these neural networks, the weights, the models of those neural networks are a little bit compressed and um, more understandable. And we started to think about that. And in parallel there was this really amazing work. The first work I uh, found from Thomas Untertinger and Daniel Kaisers and colleagues from Google Zurich and they developed a paper that used weights as input, um, you know, did some statistical features, handcrafted features, and then predicted the accuracy of those weights. And another paper by Ellison that predicted the generalization gap. So people started to use weights to extract information and they were all handcrafted features. So I was thinking about uh, handcrafted features that sold traditional machine learning. So why not end to end learning? So then we developed our idea of auto encoding sequences of weights into a uh, lower dimensional space and then reconstructing it. And if we're able to do this from a population of neural networks, then we can maybe learn a lower dimension manifold that populates actually where the neural networks populate that manifold and maybe at just manifold encodes information about Accuracy, uh, what training data was used, what training fraction learning rate and all this latent generating factors. And we started working on this and the first paper was really like, oh, well, it works. We can compress neural networks like very small, tiny neural network toy examples, right? Embarrassingly small, like thousands, ten thousands of parameters. But it was working and we could predict the accuracies and, and it was very nice to see that. And then, you know, we got published.
Speaker A: What exactly were you able to predict?
Speaker D: So we, we took, you know, an auto encoder. We have an encoder and decoder. We learned the auto encoder with or trained the auto with, uh, reconstruction loss, a contrastive loss in the middle. And then we take. We took only the encoder and unknown neural networks that we encoded into the latent space. And these embeddings we put into a simple regression, like a linear regression head to predict the accuracy. So you give me a neural network. I never saw this neural network. And the idea was, can I predict the accuracy of this neural network? Can I test the neural network without the use of test data? Obviously this worked on this really small neural networks only in a homogeneous. We call this a, uh, model zoo. A population of models. So they were all trained on the same data, same architecture first, you know, small steps towards this goal. But we could extract this information about the accuracy, the performance, the generalization gap, what kind of activation function was used. And if you would plot this latent spaces, you would see different initializations and how they evolve. Because we had, you know, 50 epochs per model training and 1,000 models. So little trajectories were visible. So there was structure. And they're all organized around a latent space.
Speaker A: The metadata around performance in accurate. These are things that you had from the base models. And so it was supervised in that sense.
Speaker D: Exactly. So we needed to be able to train this autoencoder, some population of neural networks, model zoo. And at that time, you know, laboratory trained under known condition, fully transparent. We knew which model in which epoch had which accuracy. We had obviously trained test and validation splits. So the autoencoder was trained on 600 neural networks and tested on 300 others. Um, and then we could compare. You know, we predicted 90% accuracy on Fashion MNIST and you know, the model had 92. And then, you know, we were R squared and we outperformed the work from m. Uh, Thomas Huntertinga and Daniel Kaiser, so that we were happy outperformed, uh, the original weightspace. You know, that story. It worked. We got the paper and people, I'm very thankful to the reviewers, they were kind of telling us it's small, but it's interesting. And we're able to publish because that was kind of the ignition for this amazing journey that we had over the last four years.
Speaker A: So I can imagine, you know, lots of different directions, including scaling up the models, uh, trying to get more insights out of the space, like what was next.
Speaker D: The thing is, for this type of research, everything was very obvious. It was like lying out and you just needed to do it. I mean, I never had this before. Right. So you have an auto encoder. You take the encoder, so it can predict discriminative downstream tasks like what's the accuracy, what's uh, generalization gap. But we have the other thing called the decoder. So can we sample from this space to generate neural networks? It's obvious, right? We didn't have space in the first paper, so we needed one more year to have a 22 paper published on generating neural networks. Hopefully those neural networks were then better than standard initializations. They were not as good as final or fully trained neural networks. So there was some trouble that we had, which was really, really interesting. But we, we trained this out encoder, uh, the mean split error was super low. We took the neural networks, we moved them in the forward pass, we reconstructed as said, the loss is very low. We plug the weights back to the neural network. Totally screwed up the entire neural network. We're like the mess. Yeah, it was like, it was really like. That means criteria is low and obviously now obviously right. A mean spreader is an average. So we're very good at reconstructing the average weights. But, uh, the little things that make the difference of having this function working or not, they were so important. And there is an analogy to pixels and images. Like when you had generative models for images, the images were always blurry. So people kind of tried hard and you know, taming transformers for high resolution images changed the mean squared error to perception laws and you know, did some, uh, additional things on quantizing and the gan. So we knew that reconstruction may be the wrong loss. And we tried to normalize and play around with the losses and you know, thought about the behavior loss. And then we were able to make those models a little bit better. But there's still a little bit of delta that is missing. We kind of, we generate at that time blurry weights, right? Low frequency information, high frequency information missing. And we know, we were happy because again, you know, people were kind to us and Said like it's toy examples, but you know, it's interesting. We generate that the numbers are good. But then I said, okay, we cannot be three times lucky. So we have to work really hard to scale that up. Right? I mean, you know, you're lucky twice, but you know, three times, you know, your karma is gone for the next years. So we then really and very thankful to, you know, Constantin, who was part of that initial phase. And he really worked hard. And we had this idea of instead of reconstructing the entire model, think about the model parameters as a sequence that you window and then reconstruct the windows. And therefore we would kind of de attach the sequence left of the original model to the autoencoder one. And this was interesting because suddenly we could go to Resnet and beyond. And this led then to uh, the work 2024 in collaboration with Michael Mahoney from UC Berkeley. He actually said in one of the discussions, like Damien, what you're doing is really interesting, but useless. Yeah, sure, A lot of research starts like that. So. And then, you know, because he said that I asked him that you have to help to scale that up. Right? So I caught him and he was then one of the co authors on the paper. Also the previous work was with a lot of collaboration. You know, Shavi Nero and Boris Knasny, they were part of that. Because in the beginning the idea was a little bit, as I mentioned, esoteric. So we were uh, wondering like what other people thinking about. So we very early involved a lot of people from the community to double check if we are the crazy ones or if this idea is, you know, to at least a particular limit, meaningful. So we then in 24 we scaled up to larger networks and other people got interested and we were at the conference and we met. Oh, there exist other people that are doing similar things. A lot of work from Technion, you know, uh, Hagai, uh, Maron, uh, Gel Chetnik, Yedit, uh, and then people like, you know, Eliahu was actually a funny thing. There was one student that came to our poster and he has this kind of, you know, badge. And at the badge at the bottom you have always the university written. And instead of the university he had like weights at the new modality, Elaho, uh, uh, Hovitz. And I was like, oh, that's exactly what I'm thinking. And then a collaboration started like a commun. And you know, we then we, we recognized the other people and we said like, why not doing a workshop? And then, you know, one led to another and Then, you know, that was last year. I saw 2024. Uh, we had 2025. We had then the first workshop. And then, you know, the other people recognize there are other areas that are, uh, very important. One of the tricky things with weight spaces is when you have a neural network and you have two layers, let's say for simplicity, uh, fully connected, you can change the position of the neurons and it actually changes the sequence of weights because, you know, the order of weights, but the function is the same. So it's, you know, there's sort of permutation symmetries and other, you know, things in the weight space that do not change the underlying function. So we had it already in the first pair with some augmentation. But there are a lot of people that are very, you know, very specialized on that and much more experienced, much, you know, more theoretical on that.
Speaker A: So augmentation in the sense of like applying these identity transformations to your training, your input models and using them to increase generalization and, uh.
Speaker D: Exactly. Yeah. Very simply, in our work in 21, we had a contrastive loss. And to build a contrast, you need to augment. So I mean, it's simple to flip an image or rotate an image. But, you know, what's the counterpart in weight spaces, right? I mean, and you can do the permutations and you can do other people that invented, you know, scale augmentations and other things. So the field was exciting because you saw things happening in nlp, you saw things happening in computer vision and you had to translate it into weight spaces and, you know, and it worked, right? Like augmentations happen. How can we translate it? There's this perception loss. How can we translate into a behavior loss? Right? And all those things. So you can borrow ideas from other fields and it was an empty field to fill with content along those lines.
Speaker A: Hearing you talk about these identity transformations makes me think about like, uh, other kinds of geometric transformations in the weight space or like Cartesian, the polar transformations or things like that. We saw Google quantization, uh, I forget the name of the quantization paper that just came out, or actually it came out a year ago, but it was. Became, uh, it was revisited a week or so ago and they did some Cartesian, uh, the polar transformation, like all kinds of stuff that you can do in the weight space that you might, uh. I'm curious how much of that is being explored.
Speaker D: So that's exactly. So we came from this one area of motivation. We met the other people, as I mentioned, that are on all the symmetries on the group operations you can do, there's another world of there's this mode connectivity that gets rebasing where you can align models along some reference models. Um, and if you just think about this permutation, you know, symmetries, there is so like a lost landscape that is connected. There's so many points that are the same, but they are different with respect to the weight space. And just understanding, or trying to understand how, you know, lost surfaces looks like how models can evolve along orbits of this Great work from uh, Bo and then Rose, uh, from UC San Diego on all this work kind of helped me to understand better uh, what actually happens during learning. And then hopefully, you know, we could move this in our uh, uh, backbone learning backbone because you know, some tokenization, some position encoding in the autoencoder is a transformer autoencoder. You know, we, we took some of the idea, it helped us. And then you know, people are also discovering a, uh, groking and face transitions and, and how models suddenly kind of converge or suddenly don't work and then suddenly work. So this also is also connected and I would love to explore this direction more to kind of understand how can we make training much, much faster better models or give guarantees is a strong word, but kind of, you know, bands of, where models are operational with respect to their be like their performance accuracies, et cetera.
Speaker A: Yeah, I think the last time we covered kind of weight space, I don't think we talked about it as weight space learning, but kind of this idea of like introspecting weights, uh, was with Charles Martin who uh, had to work on Weight watcher and crocking and mo collapse.
Speaker D: And so Charles. And Charles Cuauto is Michael Mahoney who was then called on our paper. So yeah, yeah. And it was really funny because Michael was doing with Charles the work of analyzing weights and looking how they, their shapes are changing. So you can make a statement about if they converge or not. And they extended obviously really amazing work. So it also happened in parallel. I'm looking at this like an analytical piece of work, um, where ours is like a learning piece of work. Uh, so we hope that uh, we can at some point scale up our backbone and process more diverse model zoos, different types of architectures. We have a work where we are now able to train and that's the vision. Right. Can you train on uh, different sized models with different architectures, tasks, modalities, from open weight repositories at Hugging Face. Could we download everything from Hugging Face and train a Foundation model of neural networks.
Speaker A: And you've, you've started down that path. I think the uh, that rabbit hole poster, that, that rabbit hole, the poster that I originally saw at GTC that led me to you was ah, something about kind of training on the hugging space model zoo.
Speaker D: Right, Exactly. So um, after we were able to scale up where then we're thinking like can we kind of. Because we're still limited to the model zoo. Right. So, so like, like I, I can tell you I can generate now a new neural network, but I need this 1000 neural networks to have trained before I can generate a, that one. So you tell me that's great. But you know, now we have 1001 neural network. So what, what are we gaining right at the end of what are we winning? So that the next step was can we get, can we train on models that are out there? And there's amazing work on analyzing how you know, hugging face looks like the model atlas, uh, from Elahu, Horowitz and yetit. And there is a lot of, there are a lot of models. So can we download those models and then independently of what kind of architecture they have, what data set they trained, use our machinery. And it's a little bit tricky because different sequence lengths, different types of uh, neural network layers. So we have to put some information into it. But we were able to train the first um, weight space learning model that can do generation and analysis of weights. So discriminative and generative downstream task on hugging face models. So this was worked by Daniel Falk, which is really amazing. Um, again we thought it's much more challenging to do this but you know, you have to scale, you need the machinery and you need those little tricks how to handle those different, you know the tokenizer needs to be adapted to arbitrary architectures. That's the thing. Yeah.
Speaker A: And what's the filter that needs to be applied on the, you know, this hugging face's vast library of models that normalizes them to something you can deal
Speaker D: with first of all. And this is work also that you know, other figured out like Elahu. There's a lot of content on hugging face that is not documented. Around 30% of the models, they don't have any meaningful metadata. So you don't know what, what. Yeah, I mean hugging face so is
Speaker A: the first filter just get rid of all of those.
Speaker D: Yeah. Knowing which models are helpful. So we need, we did a little bit of experiments. If we scale scaling alone doesn't help. You need to increase the diversity of the model zoo. So we want to have diverse models. So we want to have you know, different data sets, mostly focus on computer vision language. Uh is the next. And uh, we kind of developed a scoring function on you know, how popular is the model? How, how uh, is it apparent or is it some derived work? There's a lot of trees in there. To download a, a set of two, we have 20,000 models and from them 2,000 models that are passing some quality checks. And from these models the billions of parameters, uh, we trained the um, backbone on open weight models that then can sample also different architecture. We can sample vit, resnets, uh, this entire thing was focused on computer vision. But uh, we were able actually to sample a GTP2 model. It's still a small model but there is a domain change. And this model that we sampled we used as an initialization so it can train faster as compared to training it on regular data sets that are language. So there is some knowledge transfer happening from computer vision models to language models. That's also interesting because now we're still trying to figure out like what are those ways encapsulating. Right. And encoding. Um, so that's kind of the next step would be then to scale it up and to train on different modalities, tasks and architectures.
Speaker A: You mentioned that there were some tricks that you had to employ to be able to use different types of models. And you mentioned specifically tokenizer. Dig into that a little bit more and also talk about some of the other tricks that you had to employ to do this.
Speaker D: So good models are important. Diversity is important in the model weights you use for training the tokenization and the processing of tokens changed strongly inspired by work from uh, Kawang and uh in Singapore. So that helped us to identify or to process weights in an agnostic way so that we are not bounded by you know, this is a layer starting, this is a layer ending.
Speaker A: And so to be clear, are we talking about the uh, like your uh, your thing is a model, right? And so it has its own tokenizer or are we talking about like normalizing the tokenizer of the models that you're ingesting or both.
Speaker D: So okay, that's a good question. We, we take the models that we download from hugging face, we strip away the weights in reading order very stupidly. There are probably better ways of doing it. So we kind of destroy the metric structure and then we just sequentialize or vectorize that and then it's a long sequence of millions of parameters. Right?
Speaker B: Billions.
Speaker A: Got it. So it's just numbers and then you've got A tokenizer as part of your ingestion process.
Speaker D: We also normalize. So normalization plays also an important role. There are different positions where you can normalize on the weights as a pre processing during the tokenization or at the loss function. We tried different things and currently it depends a little bit what kind of um, on the hugging face data we normalize instead of the batches during tokenization, um, and other setups, I don't know. But this is important. The uh, tokenization is important. Um, we still playing around with the losses because we want to get some of the high fidelity high frequency information. So we still kind of suffering a little bit with that. So just as a simple example, when we generate a model, the model is a little bit damaged. So we need some fine tuning steps to recover that. Very often these fine tuning steps are very quickly able to recover. But uh, we, given that our you know, decoder has this problem of the blurry weights, we are where we're kind of not getting perfect weights. Right. So that's, that's something that we are working towards. Because if you would think about, so let's think about the world where we could train on all this data, uh, a foundation model of neural networks and sample on demand, your favorite model, whatever you need, then we would be actually replacing and totally replacing pre training. Right. So why I have pre trained model, you just sample the model that you need. So that's, that's the vision of this foundational model of neural networks. We have one additional little, you know, thing that, that kind of causes also a little bit of trouble. We need to sample a model and anchor where we sample and this anchor needs to be a model that we put through the encoder. So the better this model, the better the you know, sampled models. Which is leading to the situation that we need a well trained model to generate another well trained model which doesn't make sense if it's in the same domain because well, I have a model, why should I generate one chicken and an egg problem. Right, Exactly. Um, so therefore we had this paper that we're going to publish soon in CVPR about remote sensing where we take an imagenet vit, use our machinery to generate remote sensing models or remote sensing foundation models. Then we have the domain change where our encoder decoder is providing knowledge transfer to generate models that are better than the ones that imagenet fine tuning would be able to reach. So here we have a true knowledge transfer which is really, really great where we are able actually to outperform or be, you know equally in performance with current models like telfm published at ICLR. I think the authors claimed they trained for 12,000 GPU hours. And we are able to do this on 350 GPU hours. That's a factor of, I, uh, don't know, 20, 25, 30 depending on how you count. And this is suddenly interesting because you train from models and not from data. So if you think about we're running out of data, that's the reason why the scaling laws are a little bit, you know, considered differently. And everybody is moving into test time adaptation and test time training. We're running out of data to train the large models, but we are not using the weights of older models. So why not using the ways all the knowledge that all the compute that people invested. Right.
Speaker A: It's also interesting context. Like if the, if all the data, if you know, if we're in fact running out of data, at least in you know, specific domains, visual text, et cetera, then all that data is already in a bunch of models. Why replicate that? And why not, you know, just find ways to different ways to slurp it out of the existing models. It's essentially the premise, right?
Speaker D: Totally makes sense. In the remote sensing community we have 70 foundation models according to some surveys and there are still people training the 71st, 72nd one. Right. So why not taking this knowledge, compress it all in a weight space learning representation and then sample models on demand. Because if you have a foundation model that is a Vit with 800, 900 parameters and somebody fine tunes it to a task, this person, the partitioner needs to use all the 900 parameters and that's demanding compute and maybe the performance is a little bit better than a Resnet with 50 million, 40 million. So you know, you're suddenly bounded in this foundation model world to large models. While with our machinery, with our waste based learning approach, you could sample a big model, you could sample a resnet, or you could sample an efficiency net depending on what you need. But uh, you can give the architecture as a desired output and then we sample the parameters for whatever architecture which is helping for edge devices, helping for foundation model or helping for other use cases.
Speaker A: So you can also think of it as kind of a compression technique in a sense.
Speaker D: Exactly. The interesting question is what are we compressing, right. And how much redundancy is there and can we do it in like no pruning and uh, also distillation in like train with that? Until now we train with the raw data, but we can generate smaller models and these Smaller models are better than if you would take the original model and do distillation with a teacher's student. So that's also what we show in this remote sensing uh, scenario. Um, so the goal is really like having one big foundation model of neural networks and then able to generate on demand models. There's one missing piece though, and we have a paper crank in review that might solve that. Uh, so if you think about that, we need a model as a prompt to get an anchor to sample other models can be a domain change. So what would be really, really nice if you would not need this model as a prompt, but you could prompt with your data set.
Speaker A: So give me a model that works well on this data.
Speaker D: Exactly. And if this could be done in a privacy preserving way, it would be even better, right? Because then suddenly people could use this without revealing individual members of this data set. Open Reopen. So this work that does something similar already and Kaist, you know, um, so Andreas and others who are working on that and we kind of moved this forward because we have model zoos, right? We have data sets and we have models. So we have kind of images and we have models. And why not train an aligned space like Clip did with text and images. Then we could use actually a kind of a data set encoder as a model prompt that could move into this space and point into this space. Instead of a model prompt, we use a data set prompt. So you're a bank, you're a financial institution, healthcare provider, whatever. You don't reveal your data. You have your data set of, I don't know, hundred samples, thousand samples. You create one data set embedding so you cannot infer, you know, the individual members or samples. You give this embedding to us, we provide you the weights, we give you the weights. You're much faster in continuing training. So this would be really interesting, right, because that opens up to all the uh, you know, potential data sets that are not on hugging face and is
Speaker A: the privacy preserving angle there? Because you, your process would start with that embeddings anyway so it doesn't matter who prod produces it or uh, is that a compromise that you could do it with the processes but you could probably get more out of it if you had the actual data.
Speaker D: Good question. So um, it comes by the method that we used because uh, imagine you have a Data set with 10,000 1 million images. You need some kind. You cannot prompt with all individual images, you need some aggregate.
Speaker A: So you need to prompt with a
Speaker D: thing, one thing or one vector Right. And you need to aggregate this knowledge. So like, you know, we do a sentence and you have a text embedding to prompt your image that you generate. So it comes with that. Obviously you have to check for, you know, um, task membership attacks, you have to add some noise, et cetera. But the idea is really like, once we have this one embedding, can we generate from this one data set embedding now the tokens or the embeddings that generate the tokens for the weights? And this would open up, and then the question is, this would open up to data sets that people are not willing to share or not allowed to share. A financial institution that would maybe love to share, but they're not allowed. Um, I have a, you know, external PhD student with the Deutsche Bundesbank, the national bank of Germany. They cannot share, but you know, they might use those embeddings for that reason. And it would also allow us to kind of understand, um, how is the space that we're learning, this latent space? Because the, the most interesting thing would be that we could interpolate the models that we see on hugging face. Right. So can a data set that per definition is never on hugging face and there's no model that's trained on that. Can this data set and the data set prompt generate some meaningful neural networks that live between the space of known neural networks that are embedded? So uh, do we have a well behaved latent space? Because suddenly if we could show that we have one, this machinery could be used for generating neural networks with minimum pre training or training at all. So think about the world where you could have on demand neural networks on demand hyper personalization in a forward pass.
Speaker A: I mean it sounds a lot, a lot simpler than the way we think about like neural architecture search today, which is a lot of very complex machinery.
Speaker D: Exactly. And to the end, I mean the external, you have to give the external signal you want to resonate what kind of architecture. Right. And we just generate the weight. So we are, I consider this work complementary to the neural architecture search. Ah.
Speaker A: Uh, so that might define the structure and you might provide the weights that fit into that structure.
Speaker D: Exactly. Yeah. Or you know, I mean, you know, I want to have this type of architecture. Right. Give me a transformer, give me, uh, whatever. Right. Uh, because this is, you know, per definition, per design, already fixed, and we provide the best weights.
Speaker A: To what degree does it produce weights? Ah, kind of with knowledge of the, the architecture in which those weights will be used. Or is it just like you've got a Parameter, you know of, you know, the, the number of weights and it spits out the number of weights. And then you have to, as a post process, apply that to an architecture in a given way.
Speaker D: Exactly. So until now we, with, with the model prompt you give, uh, an architecture. With the data set prompt you don't. So you need to give an external signal like give me a Refnet 18 or a VIP. But with the model prompt you give an architecture that we fill with weights. There are definitely more clever ways of building a tokenizer that kind of understands what type of architecture elements are processed. So this would be the next step to be, to be more knowledgeable about that and to also allow the decoder to generate conditioned architectures. We don't do this until now. We generate the weights. We hope that during learning and training the backbone saw enough instances of the same architecture. So we probably only can reco. No, we definitely can only reconstruct architectures that we saw in our, um, hugging phase, uh, collection. But it would be probably much, much better if we could take this conditional signal on top. And some people are doing that, like, you know, people at, you know, in Singapore, Kawang and also kaist, they do diffusion on top of that. And then you have a conditional data set signal and conditional architecture signal. So this would be also the next step to go. Uh, but always under the, you know, promise. We can do it on open weights model that are diverse all the chaos, right, because that's where the most use can be generated from this idea of weight space learning.
Speaker A: Uh, does the model produce a, uh, sequence of weights that you then have to map to a position in an architecture? Or does the model also produce like, does it produce a weight and a position in an architecture?
Speaker D: So it generates a sequence of weights that you have to then fit to the architecture and every token that is then translated into a weight has a position encoding which position within the layer, which layer or which block, and then absolute position.
Speaker A: But that's on the input side, on
Speaker D: the input and on the input it was an auto encoder.
Speaker B: Yeah.
Speaker A: Okay, got it, got it, got it.
Speaker D: So, um, then, you know, you get a sequence of weights and you have to fit it to the right architecture. You could fit it to a different architecture and then cut it or slice and dice it. Uh, that would be an interesting experiment. What happens then? You know, how many fine tuning steps do you need to repair that? And that's probably also one of the reason why we still need a couple of fine tuning steps to get the model very quickly up to performance. And therefore I think that conditioning would kind of help us to generate better weights quicker, um, that are diverse enough. And uh, so this would be potential next steps, right? To move and condition decoder and look into the decoder.
Speaker A: When you produce these weights, are you overriding, uh, uh, an initialized model like random initialization or zero initialization or something so that the model is always valid or do you have to think about. Well the model didn't actually hallucinate it and it generated two weights for this position and no weights for that position.
Speaker D: That kind of thing we entirely overwrite. So we replace whatever is in the model randomly initialized to have that and we entirely override and load the checkpoint into that architecture. Since we have the sequence and we know the position coding, this is technically straightforward. Whatever the model does, it could have hiccups, have repetition. Uh, there's no visible pattern that would be like uh, an artifact that we could observe that is kind of repeating again and again and again. Um, but you could also think about not only generating one model, but because it's a forward pass, it's very cheap. You can generate ensembles of models, right, and then fit them and you suddenly have this new degree of freedom that you can generate. But yeah, we believe our ways, uh, that you know, we can entirely override whatever is there. There's some interesting work that currently is happening on task vectors.
Speaker A: And what is a task vector?
Speaker D: So if you have a base model for example, and you fine tune to one task, another task, and the third task you could take the difference of the fine tuned model to the base model and this would be a task vector and you could do task arithmetic, uh, with that. And in weight space. People are using that in particular language models sometimes in computer vision. And when you work on those um, task vectors, we observe in some cases that it's easier to learn and easier to generate than in other cases. In other cases full weights are easier. So um, empirically we observed it, but we cannot explain that why. For some setups, the one is easier or the other.
Speaker A: You mentioned a couple of times, uh, the distinction between high frequency noise, low frequency noise, and the weights which calls to mind like doing things in the frequency domain, applying an FFT to weights or something like that. Is that something that people are working on?
Speaker D: Uh, to the best of my knowledge, not. But uh, I think what would be interesting is um, because what we generate is kind of this blurry base and then you could add some high frequency information on top. And I think in the image domain this is happening through you have a kind of a variation auto encoder reconstructing and then you have a generative advisory network enforcing high frequency information. And then it combined lots of both of that. So you could do things like that in our domain, um, on weights. I would not know if somebody really did this. Um, but uh, moving the weights into some other domain and then processing is an interesting idea. Not sure if frequency would be the right way of doing it. Um, but you could think also a lot uh about a lot of pre processing steps until you would go and you know, do the weight space learning compression or learning the latent representation, the lower dimension manifold on that. Um, there are people who are also moving into that directions. Um, but yeah, let's see, uh, we're not going along that path currently.
Speaker A: And will the workshop be continuing this year?
Speaker D: There's another great workshop that is organized by some of our colleagues at uh, icml. Uh, it's a workshop on weight symmetries and we are thinking and continuing uh it for the next opportunity uh, which would be then uh, neurifs. But let's see how this looks like. Currently we have a couple of people that were PhDs are finishing, new PhDs are coming. So there's a little bit of a gap in the community uh with the inflow. A lot of great people are currently uh finished with the PhDs on the market already in postdoc position. So there will be definitely a continuation on that. And um, like one of the most important experiences was recognizing that there is a community and once the community is there, you know, trying to develop a common language, trying to develop common uh benchmarking, trying to develop some um, you know, ideas that are coherent as a community, uh and moving along that way. I think this is now the time to try to move in that direction and bring those ideas together with the liberty of also going to the one or the other idea and direction. There's interesting work from um, Hage Maroon for example. Um, he's taking the ideas of weight space learning, applying it to gradients or applying it to activation spaces, doing representation engineering, uh, which you could also think about the neural artifact that is not the weights but you know, the gradients during training or the activations. There's interesting work from uh, Yedit, uh coming around probing neural networks. So not looking at the weights but having controlled input output relationships and therefore looking, you know, what happens given an unknown neural network when I control the one thing.
Speaker A: Anthropic's been doing a lot of work along those lines as well.
Speaker D: Exactly so there's quite interesting work that could, uh, be considered complementary to that. And it's always this one thing. Do we have these artifacts as a collection? Can we learn some representation and then use it to predict properties of the neural network or to manipulate and edit the behavior change and modify?
Speaker A: Very interesting work. Thank you so much for jumping on and sharing a bit about it, uh, with our audience.
Speaker D: Thanks for having me. And, uh, yeah, let's see what we. Where we will be in three years.
Speaker A: Absolutely. Thank you.
How AI Learns to Smell with Alex Wiltschko - #77195 / 100
Why AI Agents Break the GenAI Security Model with Devvret Rishi - #77077 / 100
Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #76982 / 100
Relational Foundation Models for Enterprise Data with Jure Leskovec - #76892 / 100
How to Find the Agent Failures Your Evals Miss with Scott Clark - #767