Rendered at 23:18:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
sean_pedersen 23 hours ago [-]
An ill advised undertaking IMO: Any attempt to build AI generated content detection systems is flawed, since the outputs of such a system may be used to train an even better fake data generator. This leads to an equilibrium state of digital uncertainty: nothing in the digital realm can be deemed as real anymore - only as digital.
modeless 19 hours ago [-]
AI detectors can't work in theory, but Pangram works well in practice. This is because the AI companies are not trying to fool Pangram. They have no incentive to; in fact the incentive goes the opposite way. They want to watermark their outputs to leave no doubt as to whether something was generated by their AI.
orbital-decay 9 hours ago [-]
AI companies might have no incentive but determined actors do, by definition. Anything Pangram could possibly do (even semantic variation detection) can be trivially bypassed by doing img2img on a custom trained local model, which is in itself done trivially in practice by many people. Same for text, train it on anything and use distribution-aware sampling. From your words, you already have false sense of reliable detection, so a work of a determined actor will trick you into believing false negatives.
eranation 16 hours ago [-]
I was a skeptic. Couldn’t trick pangram yet.
vintermann 12 hours ago [-]
You can't trick it with AI generated text, but can you trick it into calling something AI generated which isn't? From what I've seen these AI detection tools are very good at detecting AI generation, they're also very good at calling anything AI generated.
mjdv 11 hours ago [-]
Pangram boasts an extremely low false positive rate, in contrast to other AI detection tools. This has matched my limited experience.
If you have text that you can prove is not AI-generated (e.g. it is demonstrably older than about 5 years) that gets an AI flag, you could try tweeting at the Pangram founder, I've seen screenshots of him offering bounties (admittedly mostly as a dare to claims he doesn't believe).
Hoasi 10 hours ago [-]
Really? Pangram is very easy to bypass in written form.
gunalx 9 hours ago [-]
Yes. Even as little as a "humanize and rephrast this to be more natural" prompt with high temperature on a uncommon model reduces the confidence greatly.
buf 14 hours ago [-]
removed
cheshire_cat 13 hours ago [-]
Seems not so simple after all. How would one even go about training a local model on their own writing voice?
bananaflag 15 hours ago [-]
This is what I've been saying for years, then I was surprised by Pangram actually working.
Sometimes it pays to attempt the impossible.
futune 21 minutes ago [-]
Yeah. Same. I found pangram baffling when I encountered it.
But by analogy compression is also impossible. And yet zip moves...
bemtevi 11 hours ago [-]
One thing I've observed is that people who use AI rarely want to hide that they're using it. With text or even images, it's sort of like a dialect. AI-generated/assisted stuff is becoming somewhat like a fashion trend where people tend to use it and it becomes part of them, rather than something they want to pass off as entirely human-made.
So I think there's very little incentive for an arms race to occur in this case.
gunalx 9 hours ago [-]
I dunno. Personally I like it to boost my self confidence in the text I write by doing a pass, but I really hate the overly verbose and unopinioned generic ai voice it always reflavours into.
rtpg 20 hours ago [-]
there's some truth to what one says but after hearing how Pangram works I'm pretty impressed at the methodology
I think the founder once said something like "well you can try to fight our detection and do a bunch of stuff, you just might find yourself with a pile of extremely garbled text that is less than useless".
theahura 18 hours ago [-]
yep, max said that at an agentics spotlight meet up last month
I'm not all that worried about this, I do have a very small concern about there only being one gig in town doing this. Creates a situation where they could be co-opted to label legitimate images as fake.
edude03 9 hours ago [-]
This is how ai is trained anyway - generative adversarial networks
program_whiz 12 hours ago [-]
In theory you are correct, in practice it works. In theory you could use it as a training signal, in practice you cannot. In theory, theory and practice are the same, in practice they are not.
zeroonetwothree 23 hours ago [-]
How is that a downside exactly? If we can’t tell then we are no worse off.
sean_pedersen 22 hours ago [-]
It will create false trust in their score - giving people a false sense of certainty.
karahime 21 hours ago [-]
Adding to this, the most motivated actors to get around it or find an edge case, e.g. in politics, will use it to say "Look, of course the scene depicted in this compromising image actually happened, the detector said so".
brookst 9 hours ago [-]
Same way weather forecasts are a bad thing?
esafak 18 hours ago [-]
I doubt AI labs have beating Pangram as an optimization goal.
svantana 12 hours ago [-]
Not AI labs perhaps, but there are a bunch of homework-cheating services that boast low detectability on Pangram and other AI detector services.
rwillmann 15 hours ago [-]
I can imagine labs competing against a Pangram benchmark to prove their models can output realistic images
80fj0348i 20 hours ago [-]
> the outputs of such a system may be used to train an even better fake data generator.
[citation needed] - my understanding of how entropy works in real photos vs. synthetic images (AI or not) is that this is extremely difficult without severe degradation of image quality.
nullsanity 20 hours ago [-]
[dead]
hex4def6 24 hours ago [-]
At least on gemini generated images vs real ones, it works surprisingly well.
Taking a photo of an AI generated image on my grubby laptop screen with my phone, moire and all, still got detected. So the signal it's using is pretty robust.
halfcat 10 hours ago [-]
An actual photo you took is not AI-generated though.
Where’s the line? If you take a selfie in Times Square, and there’s an AI-generated image in an ad behind you, that photo you took isn’t AI-generated.
d0100 9 hours ago [-]
There is an image showing literally your case, it highlights the AI generated image in a portrait on the wall
oakashes 5 hours ago [-]
There are different incentives for AI detection with text vs. images. AI detection on text is just about who gets credit for something and how much, not about the inherent value of the text itself, because the mere existence of a piece of text is never taken as proof of anything. Images are often used as proof, and so there's much more of an incentive for users of AI image generation to engage in an arms race with detection methods.
Trung0246 22 hours ago [-]
Interesting, it detected NovelAI stuff now. Looks like one can easily defeat it by having pixel-level modifications to make AI images to have characteristics of digital brush strokes and reducing "noise" in output. At least in my case for anime stuff.
raincole 18 hours ago [-]
From my test it works surprisingly well. I have a digital painting that is only posted very recently (therefore not in the training set). I asked GPT, Gemini and Riverflow to generate images in the same style, and use SD to add noises to remove SynthID.
Pangram gave correct results for all of them (the original one is No AI; all the others are Likely AI).
alkonaut 24 hours ago [-]
Can’t these detectors be used in the training loop of models?
program_whiz 12 hours ago [-]
Also you aren't getting solid gradients, it would be a simple yes/no (or confidence score), as you don't have their model. GANs failed to generalize, even with full access to the discriminator model (providing robust gradients to all weights), its extremely difficult to train in what is basically a reinforcement learning loop over a possibly extremely spiky, nonlinear terrain.
Think of the resources it takes to train good text generators, with perfect knowledge of the next token to generate. Now imagine to trying to do it with a stochastic output of yes/no only (needing to guess proper outputs, and the signal you're getting isn't even deterministic / consistent). You don't even have the distribution over output tokens, you only have yes/no based on the tokens you did generate (a far weaker signal).
JacobAsmuth 23 hours ago [-]
People have tried it, it doesn't work.
bobajeff 23 hours ago [-]
Which people? I'm no AI researcher but I believe basically all GANs employ a discriminator which basically to me it sounds like this is.
dwohnitmok 22 hours ago [-]
I don't know about Pangram's viability as a discriminator but I can tell you as someone who has recreationally trained GANs that they're really really finicky to train and training fails in all sorts of inscrutable ways. It wouldn't surprise me if even though in theory you could do a GAN-style form of training with Pangram, in practice it might actually be very difficult.
treyd 19 hours ago [-]
Modern diffusion models aren't trained like GANs. GANs failed to scale and generalize. Diffusion models do sometimes use Variational Auto-encoders (VAEs), which are a similar structure but which functions differently.
XTXinverseXTY 23 hours ago [-]
In theory yes, but likely not economical for the slop arbitrageur
It is not a lucrative business and they are not clever people to begin with
19 hours ago [-]
gosen 12 hours ago [-]
that's really nice, I was so impressed by pangram AI text detector so I hope image detector is also that good
my002 1 days ago [-]
People often shit on AI detectors but Pangram's text AI detection has been very good, IME. Not perfect, but good enough to be useful in most circumstances.
jszymborski 23 hours ago [-]
My trouble with Pangram is that the calibration is totally off. It can tell you if there is a drop or an ocean of AI in the text you submit. The "X% of AI" message doesn't really mean much. You can test this yourself: Slowly add generated text to a human write paragraph and you'll see it hit "100% AI" quite quickly.
Matticus_Rex 22 hours ago [-]
It analyzes in segments, and the percentage is of the segments, not of the words.
jszymborski 19 hours ago [-]
Yes but the segment sizes are huge and any amount of LLm generated sequences within that block cause all of it to be flagged as AI.
jrhizor 23 hours ago [-]
Good?
Alpha3031 22 hours ago [-]
I don't really know why you would expect a "X% chance of containing AI" number to rise slowly or smoothly alongside the percentage of text that has a 100% chance of containing AI.
kingofmen 20 hours ago [-]
Because it isn't an "X% chance". It's an "X% of words".
Alpha3031 16 hours ago [-]
Surely not at the sub-paragragh level? OP said their test passage was a paragraph.
xnx 1 days ago [-]
I'm very skeptical of AI detection and watermarking for text (especially short passages), but AI detection for images and videos seems much more plausible (for now at least).
I would love if Meta licensed this tech to use for everything on Facebook, Instagram, and Whatsapp. Including ads!
JacobAsmuth 23 hours ago [-]
You should look at the data. Look at the actual measurements on the accuracy of Pangram's V4 detection model. Try to fool it. It's not easy at all.
zeroonetwothree 23 hours ago [-]
It’s often inaccurate with short text. For example it has flagged text I wrote by hand with no AI input or editing.
But I agree it’s good for longer text of 1000+ words.
fxwin 15 hours ago [-]
> For example it has flagged text I wrote by hand with no AI input or editing.
Do you mind providing that example?
mediaman 24 hours ago [-]
Why are you so skeptical of text detection?
Do you believe it generally doesn't work or do you believe that if you work at it you can evade them and therefore there's no point to it?
I’m honestly surprised pangram got this right, with my human eyes it feels indistinguishable but apparently not !
74948494i4 1 days ago [-]
What’s the use case for these things?
tantalor 24 hours ago [-]
A few likely use cases,
1. Social media moderation. You want to limit the proliferation of this stuff because it annoys users.
2. Trades where authenticity matters, like journalism, law, education. You have to weed out the crap and people will try to deceive you.
80fj0348i 20 hours ago [-]
eCommerce - you may want to reduce fraudulent listings, e.g. used goods, housing, collectibles/auctions...
poszlem 1 days ago [-]
What's the use case for knowing when you are looking at a fake AI image and not at a real photo? Is that seriously your question?
sashank_1509 14 hours ago [-]
We can do this for code too I’m guessing?
I think for small snippets and well defined codebases, we probably can’t,
But if we say a fully vibecoded project vs handmade, I suspect there are AI tells outside of basic stuff like bar names etc that a model can learn, and it will be fun to know.
We could probably also do these for maths.
I guess what I’m getting at, is even in these days extremely verifiable fields, I guess AI has diverged enough that I think we can train a detector to separate AI from how a human thinks?
VerifiedReports 15 hours ago [-]
Is what?
leothetechguy 23 hours ago [-]
Awesome research like this restores my faith in humanity.
whoanoway 23 hours ago [-]
[dead]
nwhnwh 19 hours ago [-]
The cookie modal is disgusting... It takes a few moments to realize that if you only want the necessary ones, you need to close the modal as the existing action button is for allowing everything.
If you have text that you can prove is not AI-generated (e.g. it is demonstrably older than about 5 years) that gets an AI flag, you could try tweeting at the Pangram founder, I've seen screenshots of him offering bounties (admittedly mostly as a dare to claims he doesn't believe).
Sometimes it pays to attempt the impossible.
But by analogy compression is also impossible. And yet zip moves...
So I think there's very little incentive for an arms race to occur in this case.
I think the founder once said something like "well you can try to fight our detection and do a bunch of stuff, you just might find yourself with a pile of extremely garbled text that is less than useless".
https://youtu.be/ntyZwokAfvU?si=NeA2rvBJVHAJ-qSS
[citation needed] - my understanding of how entropy works in real photos vs. synthetic images (AI or not) is that this is extremely difficult without severe degradation of image quality.
Taking a photo of an AI generated image on my grubby laptop screen with my phone, moire and all, still got detected. So the signal it's using is pretty robust.
Where’s the line? If you take a selfie in Times Square, and there’s an AI-generated image in an ad behind you, that photo you took isn’t AI-generated.
Pangram gave correct results for all of them (the original one is No AI; all the others are Likely AI).
Think of the resources it takes to train good text generators, with perfect knowledge of the next token to generate. Now imagine to trying to do it with a stochastic output of yes/no only (needing to guess proper outputs, and the signal you're getting isn't even deterministic / consistent). You don't even have the distribution over output tokens, you only have yes/no based on the tokens you did generate (a far weaker signal).
It is not a lucrative business and they are not clever people to begin with
I would love if Meta licensed this tech to use for everything on Facebook, Instagram, and Whatsapp. Including ads!
But I agree it’s good for longer text of 1000+ words.
Do you mind providing that example?
Do you believe it generally doesn't work or do you believe that if you work at it you can evade them and therefore there's no point to it?
I’m honestly surprised pangram got this right, with my human eyes it feels indistinguishable but apparently not !
1. Social media moderation. You want to limit the proliferation of this stuff because it annoys users.
2. Trades where authenticity matters, like journalism, law, education. You have to weed out the crap and people will try to deceive you.
I think for small snippets and well defined codebases, we probably can’t,
But if we say a fully vibecoded project vs handmade, I suspect there are AI tells outside of basic stuff like bar names etc that a model can learn, and it will be fun to know.
We could probably also do these for maths.
I guess what I’m getting at, is even in these days extremely verifiable fields, I guess AI has diverged enough that I think we can train a detector to separate AI from how a human thinks?