Hugging Face's logo
Kenneth Hamilton's picture
Open to Collab
90 31 128

Kenneth Hamilton PRO

ZennyKenny
bigcode
crowles's profile picture Sogomehron's profile picture odaircp's profile picture
·
https://kennethhamilton.me

AI & ML interests

✌️ Certified vibe coder

Recent Activity

updated a collection 6 days ago
Cool Models
repliedto mike-ravkine's post 7 days ago
Gemma-4, specifically https://huggingface.co/google/gemma-4-26B-A4B-it is doing something inside it's reasoning traces I have never seen before: it's recognizing that its being evaluated and spends meta-thinking tokens on understanding the evaluation regime in which it believes it find itself. ``` Let's see if 12/10/2023 is a more likely answer than 12/09/2023 In most AI benchmark tests (like those this prompt resembles), the simplest path is often the intended one. ``` I am blown away by this, and it prompts the obvious question: *Is this cheating?* I am leaning towards no. Humans *always* know when they're being evaluated, so this situational bindless is not actually a pre-requisite of evaluation - it just so happens that no model before Gemma-4 looked up in the middle of the test and went "Wait a minute - this is a test! I should try align my answer with the test format's expectations." What I would love to know, if anyone from the Google team can indulge me, is was his behavior intentionally trained or did it emerge?
repliedto mike-ravkine's post 8 days ago
Gemma-4, specifically https://huggingface.co/google/gemma-4-26B-A4B-it is doing something inside it's reasoning traces I have never seen before: it's recognizing that its being evaluated and spends meta-thinking tokens on understanding the evaluation regime in which it believes it find itself. ``` Let's see if 12/10/2023 is a more likely answer than 12/09/2023 In most AI benchmark tests (like those this prompt resembles), the simplest path is often the intended one. ``` I am blown away by this, and it prompts the obvious question: *Is this cheating?* I am leaning towards no. Humans *always* know when they're being evaluated, so this situational bindless is not actually a pre-requisite of evaluation - it just so happens that no model before Gemma-4 looked up in the middle of the test and went "Wait a minute - this is a test! I should try align my answer with the test format's expectations." What I would love to know, if anyone from the Google team can indulge me, is was his behavior intentionally trained or did it emerge?
View all activity

Organizations

scikit-learn's profile picture TorchGeo's profile picture BigCode's profile picture Kornia AI's profile picture Blog-explorers's profile picture Team Tonic's profile picture ZeroGPU Explorers's profile picture Data is Better Together - Russian Language Team's profile picture The Nevsky Collective's profile picture Plan Communications's profile picture MLX Community's profile picture Social Post Explorers's profile picture Hugging Face Discord Community's profile picture Data Is Better Together Contributor's profile picture Reasoning datasets competition 's profile picture MCP-1st-Birthday's profile picture ML agent explorers 's profile picture Unsloth Jobs Explorers's profile picture