Using the Smartest AI to Rate Other AI

Unsupervised Learning

Become a Member: https://danielmiessler.com/upgrade

57:50

Alex (@alex_kamenca) and Carley (@carleymitus) are both members of our Action Academy Community that purchased TWO small businesses last thursday! Want To Quit Your Job In The Next 6-18 Months Through Buying Commercial Real Estate & Small Businesses? 👔🏝️ Schedule A Free 15 Minute Coaching Call With Our Team Here To Get "Unstuck" Want to know which investment strategy is best for you? Take our Free Asset-Selection Quiz Check Out Our Bestselling Book : From Passive To Passionate : How To Quit Your Job - Grow Your Wealth - And Turn Your Passions Into Profits Want A Free $100k+ Side Hustle Guide ? Follow Me As I Travel & Build: IG @brianluebben ActionAcademy.com…

11 days ago 9:35

MP3•Episode home

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

See omnystudio.com/listener for privacy information.

532 episodes

#Security #Software Development #Daniel Miessler #Tech #Future #Hacking #InfoSec

Using the Smartest AI to Rate Other AI

Unsupervised Learning

341 subscribers

published 11 days ago

MP3•Episode home

I talk about:

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

532 episodes

#Security #Software Development #Daniel Miessler #Tech #Future #Hacking #InfoSec

All episodes

Unsupervised Learning

1
The 4 AAAAs of the AI ECOSYSTEM: Assistants, APIs, Agents, and Augmented Reality 27:04

8 days ago27:04

27:04

In this episode, I break down what I believe is the emerging structure of the AI-powered world we're all building—consciously or not. I call it the “Four A’s”: Assistants, APIs, Agents, and Augmented Reality. This framework helps make sense of recent developments and where it’s all headed. I talk about: 1. Digital Assistants That Understand and Optimize Your Life Your DA (like “Kai”) will know your goals, preferences, health, schedule, and context—and proactively optimize your day, from filtering messages to planning meals or surfacing relevant information in real time. 2. APIs and the Real Internet of Things Everything becomes an API—from businesses to people to physical objects. Your assistant interacts with these APIs to act on your behalf, turning the world into a navigable ecosystem of services, tools, and resources. 3. Agents and AR Bringing It All Together Agents act autonomously to complete multi-step goals, and AR glasses will display their outputs contextually as you move through the world. These systems will collaborate, search, and act—quietly transforming how we live, work, and perceive reality. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler See you in the next one! Chapters: 00:00 - The AI Ecosystem We’re Building Without Realizing It 01:33 - Assistant: Your Most Powerful Digital Companion 03:08 - APIs: How DAs Interact with the World 07:54 - Agents: The Step Beyond Automation 11:00 - Augmented Reality: The Interface Layer of the AI Ecosystem 14:20 - Combining APIs, Agents, and UI for Real-Time Situational Awareness 17:17 - Summary: A Unified Ecosystem Driven by the Four A’s 23:36 - Industry Trends: How Companies Like OpenAI, Apple, and Meta Fit In 25:11 - Final Thoughts on Timelines, Winners, and Interpreting AI News Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
Using the Smartest AI to Rate Other AI 9:35

11 days ago9:35

9:35

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels. I talk about: 1. Using One AI to Evaluate Another The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review. 2. A Human-Centric Grading System Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected. 3. Custom Prompts That Push for Deeper Evaluation The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance. Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler See you in the next one! Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
A Conversation with Patrick Duffy from Material Security 26:47

15 days ago26:47

26:47

➡ Secure what your business is made of with Martial Security: https://material.security/ In this episode, I speak with Patrick Duffy from Material Security about modern approaches to email and cloud workspace security—especially how to prevent and contain attacks across platforms like Google Workspace and Microsoft 365. We talk about: • Proactive Security for Email and Cloud Platforms How Material goes beyond traditional detection by locking down high-risk documents and inboxes preemptively—using signals like time, access patterns, content sensitivity, and anomalous user behavior. • Real-World Threats and Lateral Movement What the team is seeing in the wild—from phishing and brute-force attacks to internal data oversharing—and how attackers are increasingly moving laterally through cloud ecosystems using a single set of compromised credentials. • Customizable, Context-Aware Response Workflows How Material helps teams right-size their responses based on risk appetite, enabling fine-grained actions like MFA prompts, access revocation, or full session shutdowns—triggered by dynamic, multi-signal rule sets. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Chapters: 00:00 - Welcome & High-Level Overview of Material Security 02:04 - Common Threats: Phishing and Lateral Movement in Cloud Office 05:30 - Access Control in Collaborative Workspaces (2FA, Just-in-Time, Aging Content) 08:43 - Connecting Signals: From Login to Exfiltration via Rule Automation 12:25 - Real-World Scenario: Suspicious Login and Automated Response 15:08 - Rules, Templates, and Customer Customization at Onboarding 18:46 - Accidental Risk: Sensitive Document Sharing and Exposure 21:04 - Security Misconfigurations and Internal Abuse Cases 23:43 - Full Control Points: IP, Behavior, Classification, Sharing Patterns 27:50 - Integrations, Notifications, and Real-Time Security Team Coordination 31:13 - Lateral Movement: How Attacks Spread Across the Workspace 34:25 - Use Cases Involving Google Gemini and AI Exposure Risks 36:36 - Upcoming Features: Deeper Remediation and Contextual Integration 39:30 - Closing Thoughts and Where to Learn More Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
AICAD: Artificial Intelligence Capabilities For Attack & Defense 42:52

17 days ago42:52

42:52

AI is changing cybersecurity at a fundamental level—but how do we decide what to build, and when? In this episode, I outline a structured way to think about AI for security: from foundational ideas to a future-proof system that can scale with emerging threats. • Rethinking Human Workflows as Intelligence Pipelines By mapping tasks into visual workflows, we can pinpoint exactly where human intelligence is still required—and where AI agents are most likely to replace or enhance us. • Using AI to Understand and Manage Organizational State I introduce the concept of AI state management: building systems that track your current and desired security posture in real time, and using AI to bridge the gap—automating insights, decisions, and even actions across your environment. • Building a Cyber Defense Program Inspired by Attacker Playbooks Instead of waiting for threats, I propose a new framework based on attacker capabilities—what they wish they could do now and in the near future—and how to proactively prepare by building a continuously adapting AI-powered defense system. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Chapters: 00:00 - Framing the Future: Two Key Questions on AI and Cybersecurity 01:28 - Intelligence Pipelines: Visualizing Human Work as Replaceable Workflow 06:10 - Theory of Constraints: How Attackers Are Bottlenecked by Human Labor 10:42 - Defining Agents: What Makes AI Different From Traditional Automation 12:08 - AI State Management: The Universal Use Case for Automated Intelligence 16:53 - Real-World Demo: Unified Context AI for Security Program Management 26:30 - Advanced Uses: Reassigning Projects, Updating KPIs, and Security Reports 34:58 - Automating Security Questionnaires With AI Context Awareness 38:43 - ACAD Framework: Predicting and Preparing for Future Attacker Capabilities 47:40 - Defender Response: Building AI-Driven Red Teams and Internal UCCs 52:25 - Final Answers: How Software and Security Change With AI Agents Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
A Possible Path to ASI 10:52

21 days ago10:52

10:52

The conversation around AGI and ASI is louder than ever—but the definitions are often abstract, technical, and disconnected from what actually matters. In this episode, I break down a human-centered way of thinking about these terms, why they’re important, and a system that could help us get there. I talk about: • A Better Definition of AGI and ASI Instead of technical abstractions, AGI is defined as the ability to perform most cognitive tasks as well as a 2022 U.S.-based knowledge worker. ASI is intelligence that surpasses that level. Framing it this way helps us immediately understand why it matters—and what it threatens. • Invention as the Core Output of Intelligence The real value of AGI and ASI is their ability to generate novel solutions. Drawing inspiration from the Enlightenment, we explore how humans innovate—and how we can replicate that process using AI, automation, and structured experimentation. • Scaling the Scientific Method with AI By building systems that automate idea generation, recombination, and real-world testing, we can massively scale the rate of innovation. This framework—automated scientific iteration—could be the bridge from human intelligence to AGI and beyond. Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Chapters: 00:00 - Why AGI and ASI Definitions Should Be Human-Centric 01:55 - Defining AGI as a 2022-Era US Knowledge Worker 03:04 - Defining ASI and Why It’s Harder to Conceptualize 04:04 - The Real Reason to Care: AGI and ASI Enable Invention 05:04 - How Human Innovation Happens: Idea Collisions and Enlightenment Lessons 06:56 - Building a System That Mimics Human Idea Generation at Scale 09:00 - The Challenge of Testing: From A/B Tests to Biotech Labs 10:52 - Creating an Automated, Scalable Scientific Method With AI 12:50 - A Timeline to AGI and ASI: Predictions for 2027–2030 Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
A Conversation With Matt Muller From Tines 39:32

29 days ago39:32

39:32

➡ Build, run, and monitor workflows with Tines at: tines.com In this episode, I speak with Matt Muller, Field CSCO at Tines, about how automation and AI are transforming security operations at scale. We talk about: • Tines' Mission to Eliminate Manual Security Work Through Automation How Tines helps security teams streamline incident response and workflow automation without needing to write code, saving time and reducing burnout. • Applying AI to Security Operations and Analyst Workflows How AI is used in phishing analysis, threat intel reporting, and data transformation—integrated safely into workflows using tools like Workbench with private LLMs. • Tines Workbench and the Future of Agentic AI How Workbench combines chat with deterministic automation to help analysts take action securely, and how Tines is exploring agentic AI to take automation even further. Chapters: 00:00 - How Tines Automates Security to Solve SOC Burnout 07:19 - The AI Arms Race: How Attackers and Defenders Are Evolving 09:08 - Why Security Still Comes Down to Workflow, Logging, and Action 13:41 - How CISOs Are Balancing AI Adoption and Enterprise Risk 17:36 - Using AI in Tines to Transform and Automate Security Workflows 20:40 - How AI Detects Business Email Compromise Better Than Rules 25:26 - From Security to Data Pipelines: Tines as Workflow Orchestration 28:59 - Inside Workbench: Secure AI-Powered Chat for Analysts 36:00 - Automating Phishing Investigations with Trusted Tool Integrations 39:19 - Where to Learn More and Try Tines for Free Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 474 | Signal OPSEC, White-box Red-teaming LLMs, Unified Company Context (UCC), New Book Recommendations, Single Apple Note Technique, and much more... 18:24

29 days ago18:24

18:24

STANDARD EDITION: Signal OPSEC, White-box Red-teaming LLMs, Unified Company Context (UCC), New Book Recommendations, Single Apple Note Technique, and much more... You are currently listening to the Standard version of the podcast, consider upgrading and becoming a member to unlock the full version and many other exclusive benefits here : https://newsletter.danielmiessler.com/upgrade Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
A Conversation With Slava Konstantinov From ThreatLocker 33:30

6 weeks ago33:30

33:30

➡ Allow what you need, block everything else with ThreatLocker: threatlocker.com In this episode, I speak with Slava Konstantinov, ThreatLocker's MacOS Lead Architect, about their zero-trust approach to endpoint security and their latest cybersecurity innovations. We talk about: • ThreatLocker’s Zero Trust Approach to Cybersecurity: How ThreatLocker enforces a default deny security model, ensuring only explicitly allowed applications and actions can run, reducing attack surfaces and unauthorized access. • Key ThreatLocker Products and Features: How ThreatLocker’s solutions—Application Control, Storage Control, Ring Fencing, Network Control, and ThreatLocker Detect—help organizations enhance security through granular policy enforcement. • New & Upcoming ThreatLocker Features: How new solutions like Patch Management, Web Control, Insights, and Cloud Detect will provide even greater security, automation, and compliance for businesses managing complex IT environments. Chapters: 00:00 - Intro to ThreatLocker and Zero Trust Security 01:24 - How ThreatLocker’s Application Control Blocks Unauthorized Software 06:52 - Storage Control: Preventing Unauthorized Data Access and USB Threats 08:19 - Ring Fencing: Controlling App Permissions and Network Access 12:37 - Elevation Control: Granting Admin Privileges Without Risk 16:23 - Network Control: Restricting Internet and Internal Network Access 19:26 - AI-Driven Security Policies: The Future of ThreatLocker Management 24:07 - Mac vs. Windows Security: Key Differences and Challenges 29:49 - ThreatLocker’s Expansion: New Products and Future Plans 32:32 - Where to Learn More About ThreatLocker’s Security Solutions Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 472 | STANDARD EDITION: 28 Open Cyber Jobs, Real-world AI Propaganda Poisoning, MCP Explained, Cline vs. Windsurf, and more... 39:31

7 weeks ago39:31

39:31

STANDARD EDITION: 28 Open Cyber Jobs, Real-world AI Propaganda Poisoning, MCP Explained, Cline vs. Windsurf, and more... You are currently listening to the Standard version of the podcast, consider upgrading and becoming a member for the full version and many other exclusive benefits here: https://newsletter.danielmiessler.com/upgrade Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessle Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
Raycast is a Must in 2025 - Action at the Speed of Thought 45:52

7 weeks ago45:52

45:52

In this episode, Daniel Miessler explores how to supercharge your macOS workflow with Raycast, transforming everyday tasks into lightning-fast, AI-powered actions. He talks about: Raycast as a Universal Launcher: Daniel explains how Raycast replaces traditional launchers like Spotlight and Alfred, offering an all-in-one shortcut to apps, files, and bookmarks for unparalleled efficiency. Quick Links and Custom Searches: He demonstrates how quick links streamline navigation by replacing outdated bookmarks and enabling custom search commands that let you bypass the browser for faster access. Integrated Utilities and Window Management: Discover how Raycast consolidates everyday tools—from color pickers and process killers to custom window arrangements—ensuring that all your essential utilities are just a keystroke away. Advanced AI Integration: Learn how Raycast’s innovative AI commands integrate with platforms like ChatGPT and Fabric, allowing you to interact with, summarize, and analyze web content directly from your command line. Custom Commands and Productivity Hacks: Daniel reveals his secrets for creating personalized hotkeys, snippets, and aliases that reduce friction in your digital workflow, making your daily tasks smoother and more intuitive. Subscribe to the newsletter at: https://danielmiessler.com/newsletter Watch the video on YouTube at: https://www.youtube.com/c/UnsupervisedLearning Follow on Twitter at: https://twitter.com/danielmiessler See you in the next one! Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 471 | STANDARD EDITION: Cyber Standing Down, China's Innovation Burst, PC vs. NPC, Why AI Can't Understand, and more... 25:40

7 weeks ago25:40

25:40

STANDARD EDITION: Cyber Standing Down, China's Innovation Burst, PC vs. NPC, Why AI Can't Understand, and more... You are currently listening to the Standard version of the podcast, consider upgrading and becoming a member for the full version and many other exclusive benefits here: https://newsletter.danielmiessler.com/upgrade Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 470 | Attacking Signal, Blogging Getting MORE Important, AI's Final Form, Claude 3.7 vs. World, Censorship as a Service, and more... 41:25

8 weeks ago41:25

41:25

STANDARD EDITION: Attacking Signal, Blogging Getting MORE Important, AI's Final Form, Claude 3.7 vs. World, Censorship as a Service, and more... ➡ Protect Against Bots, Fraud, and Abuse. Check out WorkOS Radar at: workos.com/radar You are currently listening to the Standard version of the podcast, consider upgrading and becoming a member for the full version and many other exclusive benefits here: https://newsletter.danielmiessler.com/upgrade Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 468 | TELOS Patterns, Apple 0-Day, Gumroad Replaces Developers with AI 49:20

10 weeks ago49:20

49:20

Also: A new threat modeling framework for AI, an API security report, and being paralyzed by crisis Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
UL NO. 467 | Why You Should Care About AGI (And a Definition) 25:42

12 weeks ago25:42

25:42

Plus: DeepSeek's open database, Using o3 with Fabric, Chinese backdoors in health monitors, and much more... Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at: https://danielmiessler.com/upgrade Follow on X: https://x.com/danielmiessler Follow on LinkedIn: https://www.linkedin.com/in/danielmiessler Become a Member: https://danielmiessler.com/upgrade See omnystudio.com/listener for privacy information.…

Unsupervised Learning

1
Writing Fiction With AI 30:14

12 weeks ago30:14