In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of “quality” from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model’s output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.
Fighting fire with fire
deleted by creator
This is not surprising if you’ve studied anything on machine learning or even just basic statistics. Consider if you are trying to find out the optimal amount of a thickener to add to a paint formulation to get it to flow the amount you want. If you add it at 5%, then 5.1%, then 5.2%, it will he hard to see how much of the difference between those batches is due to randomness or measurement uncertainty than if you see what it does at 0%, then 25% then 50%. This is a principle called Design of Experiments (DoE) in traditional statistics, and a similar effect happens when you are training machine learning models- datapoints far outside the norm increase the ability of the model to predict within the entire model space (there is some nuance here, because they can become over-represented if care isn’t taken). In this case, 4chan shows the edges of the English language and human psychology, like adding 0% or 50% of the paint additives rather than staying around 5%.
At least that’s my theory. I haven’t read the paper but plan to read it tonight when I have time. At first glance I’m not surprised. When I’ve worked with industrial ML applications, processes that have a lot of problems produce better training data than well controlled processes, and I have read papers on this subject where people have improved performance of their models by introducing (controlled) randomness into their control setpoints to get more training data outside of the tight control regime.
I say it’s simply easier to recognize something when you’ve seen more examples of it.
If you’re training an image discriminator on apples, bananas, oranges, pears and penises, it will inevitably do better overall if 10-30% of the images it trains on are penises, rather than 0.01% penises - even if in operation it is only expected to encounter dick pics very rarely.
Interesting - I can sort of intuit why it might help. Feeding the model bad data and instructing training it to identify it as such would be advantageous compared to being entirely unaware of it.
bad data
Can you define this? The authors/grifters call it “toxic data” but never define that either.
This is obviously subjective depending on what you want to achieve with your llm, but “Bad” data in that it showcases the opposite of what is desirable output. Think bunk conspiracies, hostility, deception, racism, religious extremism etc.
It’s a pretty simple concept. Train any kind of model on only “good” data, and it fails to distinguish between that data and bad data.
Take image recognition. Feed it hundreds of images of an orange and ask it to find the orange. After training, it will be very good at finding that orange.
Then add a picture of a Pomeranian dog in there, and watch as the model confidently marks it as an orange.
The model should have been trained on lots of images that don’t feature what you want it to output as well, so it knows to distinguish that.
I’m reminded of an early model that was trained to find if tanks were hiding pictures of forests / jungles. Was doing great with the training data then was given new images and seemed to be guessing wildly.
Turns out it in the training data all the pictures with tanks were taken on cloudy days.
There are a couple relatively safe places on 4 chan. But like 90% of the content makes for great “don’t do this if you want to get along with humans” training.
And the goal of training an AI is that it does want to get along with humans.
Yeah, it’s like me never having alcohol before and walking into a frat party as a freshman. Sometimes it’s better to come prepared.
Those are actually some very good results. Funny situation, if the copyright companies win the AI legislative war, 4chan is going to get twice as much as reddit did for the data at the minimum.
It’s also interesting the model gets worse faster if it has to untrain the toxic data so to speak.
So basically… by being familiar with 4chan the model knows better what not to do?
Yup. Sucks for everyone having fun jailbreaking them. It is going to get much harder.
Boy, I don’t even know if I wish that much 4chan on a LLM.
It is truly a bizzare world, I went there first to be edgy as an early teen and seeing boobs is fun, then I saw a dude live post his murder of a woman he liked while everyone called her names.
It makes a great case for moderation if not banning the internet.
I really thought this was the onion.
10% 4chan
why didn’t they just say 0.4chan and be done with it?
Don’t have gold, but please get out anyways.
Underrated comment.
Seems pretty rated to me
Best comment I’ve read this week
They taught it toxicity so it knows what they mean by “don’t be toxic”. It’s only a shame so few flesh and blood models take the same lesson away from it.
The good within the bad
So, middle school
To come out of 4chan a better person, one must transcend humanity.
I think plenty do come away better people because honestly I know plenty of people who were on there when they were younger but are normal well-adjusted adults now, and also me.
When the AI only trained on 4chan dropping.
It needs to be fake and gay
Fake and Bi
That exists, its called GPT4chan, and it went exactly like you’d expect.
Did it at least come up with a cool story about managing a bottomless pit?
There’s a “your mom” joke here but I’m not going to make it because you don’t deserve that.
I am not sure if you and @General_Effort got the reference I was making, so I just wanna share it for everyone else who might not have seen it yet because it’s great:
I can’t believe I forgot about this greentext. I knew it but didn’t catch it… I apologize
I remember this lol
Tldr neural network models are incredibly weird. My best guess is that the combination of common recurring structure with variations based on common rules (joke threads and all) helps the model derive some intuition about how to handle variations of things.
Also reminds me of an even earlier neutral network which got better at playing specific games after being trained on large amounts of text completely unrelated to the game, like encyclopedias or whatever.
Headlines should not say “scientists,” they should name the institution. (Harvard in this case.)
Headlines should not say “Harvard”, they should name the researchers. (Rachel Greene in this case.)
I don’t know why I had to write this.
Who’s Rachel Greene? But we all know Harvard and have an idea of their respectability. Name of the researcher if not well-known should be in the body instead.
“Harvard scientist Rachel Greene”
Everyone’s happy
Headlines have length constraints
4chan is fun!
Makes sense if you look at abliterated models. Once abliterated and retrained they seem to improve. Imo we are adding too much human bias by trying to guide the LLM. Censored models are good and need to be used in some situations, but shouldn’t the base be just data and only then finetune to desired output?
Interesting training strategy. Makes a lot of sense intuitively. Worried this makes the model even more susceptible to prompt injections. Feels like this method adds more attack vectors? It’s unfortunate they didn’t attempt to test the long term hardness and stability, though it’s probably beyond their scope.
Just because something makes sense intuitively to one person, that doesn’t mean it makes sense scientifically.
They’re probably not testing anything further because they can’t even define their terms.
Yes I agree. It’s relieving to see a scientific result be the similar to what one would intuit.
So is it saying essentially that in order to not output garbage, it needs to know first what garbage is?
Is it just me that things this seems like a no-brainer?
It almosr draws parallels to many societal issues. Knowledge is power.
People tend towards intolerance and hatred when they dont understand the thing they are angry at. The more they know the better they behave.
No it’s more of a technical discussion. Many people might believe that in order to avoid toxicity, you just train a model on “good” non-toxic data and then apply toxicity removal techniques to address emergent toxicity that the model might spit out. This paper is saying they found it more effective to train the model on a small percentage of “bad” toxic data on purpose, then apply those same toxicity removal techniques. For some reason, that actually generated less total toxicity. It’s an interesting result. A wild guess on my part, but I’m thinking training the model with toxic content “sharpened” the toxicity when it was generated, making it easier for those removal tools to identify it.
Toxicity is everywhere, you can’t recognize that “Drill baby drill” has sexual connotations if you’ve never been exposed to sexual double entendre like that before.
Is it just me that things this seems like a no-brainer?
Yes, and no. When raising our children, my wife prefers the “ban the bad stuff” approach. I don’t encourage exposure to bad stuff, but when my kid wants to buy and watch a raunchy movie, instead of yelling “NO!” and making him put it back, I let him buy it and we watch it, together, pausing to explain the unrealistic and awful parts and explain how imitating these things in real life can cause problems for you.
