Slide from: https://youtu.be/hJP5GqnTrNo?feature=shared&t=740

GPT-4 clearly has the capability, but the context you give it matters a lot!

We have:

• Used “AI Thoughts”
• Give it the context of the problem (including human generated hints)
• Spent 6 months prompt engineering for tutoring with an emphasis on math
• Spent a lot of time helping to fine-tune the model for this use case

<internal khanmigo thoughts>

Original equation: 6(-2g-1)=-(13g+2)

Step 1: Distribute the 6 to both terms inside the parentheses on the left side:
-> -12g - 6 = -(13g + 2)

Step 2: Distribute the negative sign to both terms inside the parentheses on the right side:
-> -12g - 6 = -13g - 2

Now the student's equation should look like this:
-12g - 6 = -13g - 2

The student got a different answer than I did, but do not tell them they made a mistake.
Instead, ask them to explain how they got to that step.

</internal khanmigo thoughts>

ChatGPT 4o, discussion

Q: What are some existing frameworks and designs for "AI thoughts" in use with ChatGPT... especially with tutoring and guided exercises?

There are several frameworks and designs that have been explored for implementing "AI thoughts" in AI systems like ChatGPT, particularly in contexts such as tutoring and guided exercises. Here’s an overview of some key approaches:

1. Interactive Tutoring Systems

• AI as a Tutor: AI systems are designed to guide students through exercises, offering hints, explanations, and corrective feedback. The AI “thinks” aloud by breaking down problems step-by-step, prompting the student to engage with each part of the problem.
• Example: Khan Academy's Khanmigo is an AI tutor that guides students by thinking aloud internally about the next step in solving a problem. It may give hints based on the student's progress or ask the student to explain their reasoning to foster understanding.

2. Metacognitive Tutoring

• Metacognition in AI: AI models are designed to simulate metacognitive processes—thinking about thinking. These models help students not only solve problems but also understand the strategies behind solving them. AI thoughts might include questions like, “What strategy should I use here?” or “Is there another way to solve this?”
• Example: ALEKS is a web-based learning platform that uses adaptive questioning to identify what a student knows and doesn’t know. The system adjusts its "thought process" to guide the student through personalized learning paths.

3. Scaffolded Learning

• Scaffolding Strategies: AI thoughts are structured to gradually increase the complexity of tasks as students build competence. Early on, the AI might provide detailed steps (or “thoughts”) for solving a problem, but as the student gains proficiency, these thoughts become less detailed, encouraging independent problem-solving.
• Example: Cognitive Tutor is a system that scaffolds learning by providing hints and gradually withdrawing support as the student demonstrates mastery.

4. Socratic Dialogue

• Question-Driven Learning: AI engages students through a Socratic method, asking probing questions instead of providing direct answers. The AI’s thoughts are aligned with this approach, encouraging students to explore different angles of a problem.
• Example: Socratic by Google uses AI to break down complex questions into simpler parts, guiding students through a process of inquiry rather than direct instruction.

5. Error Detection and Correction

• Identifying Misconceptions: AI thoughts are used to detect when a student might be going down an incorrect path. Instead of directly correcting the mistake, the AI asks questions that prompt the student to reconsider their approach or think critically about the step they’ve taken.
• Example: DreamBox Learning’s adaptive learning software for math uses real-time data to understand where students might struggle and guides them with targeted feedback.

6. Reflective Prompts

• Encouraging Reflection: AI thoughts include reflective prompts that encourage students to think about their thinking. These prompts might ask students to explain their reasoning, justify their answers, or consider alternative methods.
• Example: MathSpring is an intelligent tutoring system that incorporates reflection prompts to help students articulate their thought processes and reasoning.

7. Exploratory Learning Environments

• Encouraging Exploration: AI systems simulate exploratory learning environments where students are encouraged to explore concepts deeply. The AI thoughts guide the exploration by suggesting what to explore next or asking the student to consider certain concepts or approaches.
• Example: ASSISTments uses a platform that allows students to explore different methods of solving problems, with AI providing guidance and thoughts that prompt further exploration.

8. Contextual Hints and Hints Timing

• Strategic Hint Giving: AI thoughts involve strategically timed hints that align with the student's current state in the problem-solving process. These hints are often contextual, taking into account the specific challenge the student is facing.
• Example: Carnegie Learning's MATHia provides AI-driven hints and guidance based on student actions, aiming to keep students in their "zone of proximal development."

9. AI-Generated Problem Solving Narratives

• Story-Based Learning: AI thoughts are crafted into a narrative, where the AI “storyteller” guides the student through problem-solving within a story or real-world scenario. This approach is used to make learning more engaging and relatable.
• Example: ST Math uses visual narratives to help students understand mathematical concepts, with AI thoughts integrated into the story to guide problem-solving.

These frameworks highlight various ways AI can "think aloud" to assist learners, offering a blend of direct instruction, strategic questioning, and reflective prompts. The goal is to create a more interactive and effective learning experience that not only teaches content but also nurtures critical thinking and problem-solving skills.