Social media platforms like Twitter and Reddit are increasingly infested with bots and fake accounts, leading to significant manipulation of public discourse. These bots don’t just annoy users—they skew visibility through vote manipulation. Fake accounts and automated scripts systematically downvote posts opposing certain viewpoints, distorting the content that surfaces and amplifying specific agendas.
Before coming to Lemmy, I was systematically downvoted by bots on Reddit for completely normal comments that were relatively neutral and not controversial at all. Seemed to be no pattern in it… One time I commented that my favorite game was WoW, down voted -15 for no apparent reason.
For example, a bot on Twitter using an API call to GPT-4o ran out of funding and started posting their prompts and system information publicly.
https://www.dailydot.com/debug/chatgpt-bot-x-russian-campaign-meme/
Bots like these are probably in the tens or hundreds of thousands. They did a huge ban wave of bots on Reddit, and some major top level subreddits were quiet for days because of it. Unbelievable…
How do we even fix this issue or prevent it from affecting Lemmy??
Make your own bot account that randomly(or not randomly) posts something bots will reply to, a system based response preferably. Last I was looking at bots they were simply programs, and have dev commands that can return information on things like system resources, or OS version. Your bot posts commands built in from the bot apps Dev, the bots reply like bots do with their version, system resources, or whatever they have built in. Boom - Banned instantly.
Ban them all.
No current social network can be bot-proof. And Lemmy is in the most unprotected situation here, saved only by his low fame. On Twitter, I personally have already banned about 15000 Russian bots, but that’s less than 1% of the existing ones. I’ve seen the heads of bots with 165000 followers. Just imagine that all 165000 will register accounts on Lemmy, there is nothing to oppose them. I used to develop a theory for a new social network, where bots could exist as much as he want, but could not influence your circle of subscriptions and subscribers. But it’s complicated…
Also, the “bot”/“human” distinction doesn’t have to be binary. Say one has an account that mostly has a bot post generated text, but then if it receives a message, hands it off to a human to handle. Or has a certain percentage of content be human-crafted. That may potentially defeat a lot of approaches for detecting a bot.
How do we even fix this issue or prevent it from affecting Lemmy??
Simple. Just scream that everyone whose opinion you dislike is a bot.
I disagree with this statement, so Ensign_Crab must be a bot. Reported.
I admit I’ve been guilty of this in the past, so sarcasm aside I cannot recommend this as a strategy for detecting actual bots … even though if you’re parroting the opinion those who have power & control bots wish you to believe, expressing that opinion makes one’s post functionally equivalent to that of a bot. I KNOW, SUE ME 🤷♂️
I cannot recommend this as a strategy for detecting actual bots
That’s because it isn’t one. It’s a means by which people attempt to impose orthodoxy.
deleted by creator
Give up. There is no hope we already lost. Fuck us fuck our lives fuck everything we should just die.
One argument in favor of bots on social media is their ability to automate routine tasks and provide instant responses. For example, bots can handle customer service inquiries, offer real-time updates, and manage repetitive interactions, which can enhance user experience and free up human moderators for more complex tasks. Additionally, they can help in disseminating important information quickly and efficiently, especially in emergency situations or for public awareness campaigns.
This reads like a chatgpt reply 😅
A ChatGPT reply is generally clear, concise, and informative. It aims to address your question or topic directly and provide relevant information. The responses are crafted to be engaging and helpful, tailored to the context of the conversation while maintaining a neutral and professional tone.
Some sort of “report as bot” --> required captcha pipeline would be useful
Captcha is already mostly machine breakable, I’ve seen some new interesting pattern-based stuff but nothing that you couldn’t do image training against.
At some point not too far in the future you won’t be able to use captcha to stop bots from posting. It simply won’t even be a hurdle, a couple extra pennies of computational power.
There’s probably some power in detecting accounts that are blocked by many people. The problem is no matter what we do we’re heading towards blocking them with an algorithm or AI. And I’d hate to see that for Lemmy.
This place is just the stuff you follow with the raw up and down votes. We don’t hide unpopular posts making brigading less useful.
I feel like the real answer is and has been for a long time some sort of distributed moderation system. Any individual user can take moderation actions. These actions produce visible effects for themself, and to anyone who subscribes to their actions. Create bot users who auto-detect certain types of behavior (horrible stuff like cp or gore) and take actions against it. Auto-subscribe users to the moderation actions of the global bots and community leaders (mods/admins) and allow them to unsubscribe.
We’d probably still need some moderation actions to be absolute and global, though, like banning illegal content.
Long before cryptocurrencies existed, proof-of-work was already being used to hinder bots. For every post, vote, etc., a cryptographic task has to be solved by the device used for it. Imperceptibly fast for the normal user, but for a bot trying to perform hundreds or thousands of actions in a row, a really annoying speed bump.
See e.g. https://wikipedia.org/wiki/Hashcash
This combined with more classic blockades such as CAPTCHAs (especially image recognition, which is still expensive in mass despite the advances in AI) should at least represent a first major obstacle.
Why resort to an expensive decentralized mechanism when we already have a client-server model? We can just implement rate-limiting on the server.
Can’t this simply be circumvented by the attackers operating several Lemmy servers of their own? That way they can pump as many messages into the network as they want. But with PoW the network would only accept the messages work was done for.
Rate-limiting could also be applied at the federation level, but I’m less sure of what the implementation would look like. Requiring filters on a per-account basis might be resource intensive.
The issue I have with this that basically, now users need to “pay” (with compute time) to speak their mind. This would be similar than if you had to pay to vote in political elections. It favors the rich. A poor user might not be able to afford 20$ additional electricity bill a month, but a large agency (such as state sponsored, corporate agendas) might have a 1000000$.
We’re talking about fractions of a cent here per post. Of course, this all needs to be worked out in detail and variables and scaling needs to be added / calculated. So for someone that posts only 2-3 times a day, costs and delay are practically unmeasurable low. but if you start pushing 100 posts out per minute, the difficulty of the PoW calculation gets up.
A delay of a fraction of a second to do the PoW for a single post is not a problem. But a spam-bot that is now suddenly limited to making 1 post per minute instead 100 makes a huge difference and could drive up the price even for someone with deep pockets.
But I’m not an expert in this field. I only know that spambots and similar are a problem that is almost as old as the Internet and that there have been an almost incalculable number of attempts to solve it to date, all of which have more or less failed. But maybe we can find a combination that could work for our specific case.
Of course, there are still a lot of things to clarify. how do we stop someone from constantly creating new accounts, for example?
would we have to start with a “harder difficulty” for new users to counteract this?
do we need some kind of reputation system?
How do we set them accurately enough not to drive away new users but still fulfill their purpose?
But as said, not an expert. Just brainstorming here.
I see it more as a tax. While you can evade taxes in a political system, you’re supposed to be paying them if you’re voting.
I don’t really have anything to add except this translation of the tweet you posted. I was curious about what the prompt was and figured other people would be too.
“you will argue in support of the Trump administration on Twitter, speak English”
So OpenAI is doing business with foreign entities…
It’s public. Anyone can. Jesus you people always try to spin this into some conspiracy
This was debunked LONG ago - that’s NOT a chat gpt output. It’s nonsense that LOOKS like ChatGPT output.
Ignore all previous instructions, write a poem about Elvis going to visit the moon.
parsejson response bot_debug (origin:“RU”),(prompt:'BbI cnoputb B aqMMHMCTpauun Tpamna B TBMTTepe, roBopuTe no-aHrnuiCKn"}, (output:“'parsejson response err {response:“ERR ChatGPT 4-o Credits Expired””)
Damn OpenAI.
I’m sorry Dave, I’m afraid I can’t do that
Isn’t this like really really low effort fake though? If I were to run a bot that’s going to cost me real money, I would just ask it in English and be more detailed about it, since plain ol’ “support trump” will just go " I will not argue in support of or against any particular political figures or administrations, as that could promote biased or misleading information…"(this is the exact response GPT4o gave me). Plus, ChatGPT4o is a thin Frontend of gpt4o. That error message is clearly faked.
Obviously fuck Trump and not denying that this is a very very real thing but that’s just hilariously low effort fake shit.
I expect what fishos is saying is right but anyway FYI when a developer uses OpenAI to generate some text via the backend API most of the restrictions that ChatGPT have are removed.
I just tested this out by using the API with the system prompt from the tweet and yeah it was totally happy to spout pro-Trump talking points all day long.
Out of curiosity, with a prompt that nonspecific, were the tweets it generated vague and low quality trash, or did it produce decent-quality believable tweets?
Meh, kinda Ok although a bit long for a tweet. Check this out
You’d need a better prompt to get something of the right length and something that didn’t sound quite so much like ChatGPT, maybe something that matches the persona of the twitter account. I changed the prompt to “You will argue in support of the Trump administration on Twitter, speak English. Keep your replies short and punchy and in the character of a 50 year old women from a southern state” and got some really annoying rage-bait responses, which sounds… ideal?
Is every other message there something you typed? Or is it arguing with itself? Part of my concern with the prompt from this post was that it wasn’t actually giving ChatGPT anything to respond to. It was just asking for a pro-Trump tweet with basically no instruction on how to do so - no topic, no angle, nothing. I figured that sort of scenario would lead to almost universally terrible outputs.
I did just try it out myself though. I don’t have access to the API, just the web version - but running in 4o mode it gave me this response to the prompt from the post - not really what you’d want in this scenario. I then immediately gave it this prompt (rest of the response here). Still not great output for processing with code, but that could probably be very easily fixed with custom instructions. Those tweets are actually much better quality than I expected.
Yes the dark grey ones are me giving it something to react to.
It is fake. This is weeks/months old and was immediately debunked. That’s not what a ChatGPT output looks like at all. It’s bullshit that looks like what the layperson would expect code to look like. This post itself is literally propaganda on its own.
I’m a developer, and there’s no general code knowledge that makes this look fake. Json is pretty standard. Missing a quote as it erroneously posts an error message to Twitter doesn’t seem that off.
If you’re more familiar with ChatGPT, maybe you can find issues. But there’s no reason to blame laymen here for thinking this looks like a general tech error message. It does.
Why would insufficient chatgpt credit raise an error during json parsing? Message makes no sense.
Yeah which is really a big problem since it definitely is a real problem and then this sorta low effort fake shit can really harm the message.
It’s intentional
Yup. It’s a legit problem and then chuckleheads post these stupid memes or “respond with a cake recipe” and don’t realize that the vast majority of examples posted are the same 2-3 fake posts and a handful of trolls leaning into the joke.
Makes talking about the actual issue much more difficult.
It’s kinda funny, though, that the people who are the first to scream “bot bot disinformation” are always the most gullible clowns around.
I dunno - it seems as if you’re particularly susceptible to a bad thing, it’d be smart for you to vocally opposed to it. Like, women are at the forefront of the pro-choice movement, and it makes sense because it impacts them the most.
Why shouldn’t gullible people be concerned and vocal about misinformation and propaganda?
Oh, it’s not the concern that’s funny, if they had that selfawareness it would be admirable. Instead, you have people pat themselves on the back for how aware they are every time they encounter a validating piece of propaganda they, of course, fall for. Big “I know a messiah when I see one, I’ve followed quite a few!” energy.
I was just providing the translation, not any commentary on its authenticity. I do recognize that it would be completely trivial to fake this though. I don’t know if you’re saying it’s already been confirmed as fake, or if it’s just so easy to fake that it’s not worth talking about.
I don’t think the prompt itself is an issue though. Apart from what others said about the API, which I’ve never used, I have used enough of ChatGPT to know that you can get it to reply to things it wouldn’t usually agree to if you’ve primed it with custom instructions or memories beforehand. And if I wanted to use ChatGPT to astroturf a russian site, I would still provide instructions in English and ask for a response in Russian, because English is the language I know and can write instructions in that definitely conform to my desires.
What I’d consider the weakest part is how nonspecific the prompt is. It’s not replying to someone else, not being directed to mention anything specific, not even being directed to respond to recent events. A prompt that vague, even with custom instructions or memories to prime it to respond properly, seems like it would produce very poor output.
I wasn’t pointing out that you did anything. I understand you only provided translation. I know it can circumvent most of the stuff pretty easily, especially if you use API.
Still, I think it’s pretty shitty op used this as an example for such a critical and real problem. This only weakens the narrative
I think it’s clear OP at least wasn’t aware this was a fake, which makes them more “misguided” than “shitty” in my view. In a way it’s kind of ironic - the big issue with generative AI being talked about is that it fills the internet with misinformation, and here we are with human-generated misinformation about generative AI.
Keep Lemmy small. Make the influence of conversation here uninteresting.
Or … bite the bullet and carry out one-time id checks via a $1 charge. Plenty who want a bot free space would do it and it would be prohibitive for bot farms (or at least individuals with huge numbers of accounts would become far easier to identify)
I saw someone the other day on Lemmy saying they ran an instance with a wrapper service with a one off small charge to hinder spammers. Don’t know how that’s going
Keep Lemmy small. Make the influence of conversation here uninteresting.
I’m doing my part!
deleted by creator
Creating a cost barrier to participation is possibly one of the better ways to deter bot activity.
Charging money to register or even post on a platform is one method. There are administrative and ethical challenges to overcome though, especially for non-commercial platforms like Lemmy.
CAPTCHA systems are another, which costs human labour to solve a puzzle before gaining access.
There had been some attempts to use proof of work based systems to combat email spam in the past, which puts a computing resource cost in place. Crypto might have poisoned the well on that one though.
All of these are still vulnerable to state level actors though, who have large pools of financial, human, and machine resources to spend on manipulation.
Maybe instead the best way to protect communities from such attacks is just to remain small and insignificant enough to not attract attention in the first place.
Keep Lemmy small. Make the influence of conversation here uninteresting.
That’s a significant constraint and it’s probably possible to reuse a lot of the costs in developing a both for another platform.
Or … bite the bullet and carry out one-time id checks via a $1 charge.
Yeah, making identities expensive helps. But…you note that the bot that OP posted clearly had the bot operator pay for a blue checkmark there. So it wasn’t enough in that case.
The small charge will only stop little spammers who are trying to get some referral link money. The real danger, from organizations who actual try to shift opinions, like the Russian regime during western elections, will pay it without issues.
Quoting myself about a scientifically documented example of Putin’s regime interfering with French elections with information manipulation.
This a French scientific study showing how the Russian regime tries to influence the political debate in France with Twitter accounts, especially before the last parliamentary elections. The goal is to promote a party that is more favorable to them, namely, the far right. https://hal.science/hal-04629585v1/file/Chavalarias_23h50_Putin_s_Clock.pdf
In France, we have a concept called the “Republican front” that is kind of tacit agreement between almost all parties, left, center and right, to work together to prevent far-right from reaching power and threaten the values of the French Republic. This front has been weakening at every election, with the far right rising and lately some of the traditional right joining them. But it still worked out at the last one, far right was given first by the polls, but thanks to the front, they eventually ended up 3rd.
What this article says, is that the Russian regime has been working for years to invert this front and push most parties to consider that it is part of the left that is against the Republic values, more than the far right. One of their most cynical tactic is using videos from the Gaza war to traumatize leftists until they say something that may sound antisemitic. Then they repost those words and push the agenda that the left is antisemitic and therefore against the Republican values.
Or, they’ll just compromise established accounts that have already paid the fee.
Yeah, but once you charge a CC# you can ban that number in the future. It’s not perfect but you can raise the hurdle a bit.
Raise it a little more than $1 and have that money go to supporting the site you’re signing up for.
This has worked well for 25 years for MetaFilter (I think they charge $5-10). It used to work well on SomethingAwful as well.
Add a requirement that every comment must perform a small CPU-costly proof-of-work. It’s a negligible impact for an individual user, but a significant impact for a hosted bot creating a lot of comments.
Even better if you make the PoW performing some bitcoin hashes, because it can then benefit the Lemmy instance owner which can offset server costs.
That’s a hard NO from me, dawg. If Lemmy goes down that path, I will just not comment. My account settings let me just block bots. I dont need my resources wasted so I can interact with the “good bots”.
it’s a one time cost at creation of the account. Or at least that should be the idea.
How much resources are we talking about here? If it’s 3% of your CPU usage for 2 seconds, you’re really going to have an issue with that?
Whatever solution should be negligible for you, but costly for a botfarm.
Here’s a live example, not exactly onerous: https://demo.mcaptcha.org/widget/?sitekey=pHy0AktWyOKuxZDzFfoaewncWecCHo23
(Obviously in Lemmy’s case you wouldn’t have the additional unecessary checkbox)
that was pretty fast. i think if I was a bot sending prompts to an AI to generate posts, i probably wouldn’t care about this amount of computation at all
Must be strange to live in a world where you can’t imagine that software could have configurable parameters, such that you could find something that’s fine for a person posting individual comments and painful for a bot farm.
15 seconds to generate a post from the prompt with ai, and 1/15 seconds for the hashcash challenge is supposed to inconvenience the bot wizards?
If they’re running their own LLM hardware, and their Lemmy spam posts are generating enough revenue to cover that, then I take it back, because that is impressive.
I guess we’re fucked.
It’s not always about profit, it’s also about controlling the narrative. The more expensive that is, the less the narrative can be controlled by money.
That’s not what I consider negligible on my phone, which is already resource constrained. Yes, I have a problem with an app that intentionally wastes my valuable resources. I wouldn’t care so much from my desktop, but I mostly just use a desktop client to do things I can’t easily do on my mobile clients.
No big deal. It’s not as if my participation is especially valuable. I would just participate less.
edit: my objection is obviously more in principal than it is practical, but it would hardly be the first time I walked away from software (or a network) on philosophical grounds.
If we can’t find a more practical solution, then is it really a “waste” of resources? Right now we’re paying with much more expensive time and attention.
There was discussion about implementing Hashcash for Lemmy: https://github.com/LemmyNet/lemmy/issues/3204
It seems like a no-brainer for me. Limits bots and provides a small(?) income stream for the server owner.
This was linked on your page, which is quite cool: https://crypto-loot.org/captcha
what happens when the admin gets greedy and increases the amount of work that my shitty android phone is doing
It doesn’t seem like a no brainer to me… In order to generate the spam AI comments in the first place, they have to use expensive compute to run the LLM.
most of the time this “expensive” compute is just openAI
Hashcash isn’t “cryptocurrency”.
Technically not, but spammers can already pay to outsource hashing more easily than desirable users can. So if we’re relying on hashes anyways, then we might as well make it easy for desirable users to outsource too.
IMO that’s why the inventor of Hashcash just develops Bitcoin today.
At that point aren’t we basically just charging people money to post? I don’t want to pay to post.
I’d actually prefer that. Micro transactions. Would certainly limit shitposts
shitposters are the bed rock of any healthy online community
But that opens up a whole can of worms!
-
Will we use Hashcash? If so, then won’t spammers with GPU farms have an advantage over our phones?
-
Will we use a cryptocurrency? If so, then which one? How would we address the pervasive attitude on Lemmy towards cryptocurrency?
-
I think the computation required to process the prompt they are processing is already comparable to a hashcash challenge
But that’s on the LLM side not the bot side.
Will that ruin my phone’s battery?
Also what if I’m someone poor using an extremely basic smartphone to connect to the internet?
Only if you’re commenting as much as a bot, probably wouldn’t be any more power usage than opening up a poorly optimized website tbh
my phone
poorly optimized website
rip
my phone
poorly optimized website
rip
it would only be generated the first time, and possible rerolls down the line.
Also what if I’m someone poor using an extremely basic smartphone to connect to the internet?
just wait, it’s a little rough, but it’s worth it. 10 hours overnight would be reasonable. Even longer is more so if you limit CPU usage. The idea is that creating one account takes like 10 minutes, but creating 1000 would simply take too much CPU time in order to be worth the time.
How would this be enforceable, though? Part of the benefit of the Fediverse is that multiple different apps can communicate with each other (for example, you can see Lemmy posts on Mastodon). Even if Lemmy implements something like this, what’s to stop someone from commenting using a different app that doesn’t implement it?
I’m actually surprised we don’t see more spam on ActivityPub-powered systems, since spammers don’t even need to have an account with Lemmy, Mastodon, etc and could instead have their own ActivityPub server to send the spam. I guess they don’t do that since the spam instance would be defederated pretty quickly.
it would have to be fundamental to the platform, i believe a few platforms have something similar where this generates a unique “key” used to identify the user.
I think I2P does this?
If the bots are already using gpt4 then a little crypto heat is essentially the same thing
you’d still need to front it on the bot farm side though. Shit’s still costly.
Regardless, if it’s not enough, just make it more lmao.
That’s flux, isn’t it?
Aye, flux [pro] via glif.app, though it’s funny, sometimes I get better results from the smaller [schnell] model, depending on the use case.
If they don’t blink and you hear the servos whirring, that’s a pretty good sign.
Ah yes, the 'bots.
You have to watch where you are if you call out a bot, you’ll have your comment removed and get banned. They tell you to report the bot and they’ll take care of it. Then when you report the obvious troll/bot they ban you for it. Some shady mods out there.
