Had quite the adventure with AI this week. We made serious progress with our Mattermost AI assistant “Conrad” and discovered some genuinely impressive client applications that saved hours of work. Here’s the rundown of what worked, what didn’t, and what surprised me.

Conrad Gets Upgrades: Knowledge Bases & File Management

Our Mattermost AI assistant Conrad got several major upgrades this week:

• Vector Knowledge Base: Setup was straightforward, but maintenance quickly became a headache. I built a workflow that tracks when source files change, deletes outdated versions from the vector store, and uploads the fresh content. Much cleaner.
  • Specialized Knowledge Stores: Split our knowledge into two primary databases:
    Research Database: All our research papers, both external references and internally produced work. Makes citing facts dramatically easier and reduces hallucinations.
  • CGA Management Store: Everything we publish, plus branding, tone guidelines, and our evolving product info. It’s becoming our content generation powerhouse.
• Internet Research Capabilities: Hooked Conrad to Perplexity AI for web research, but quickly learned these agents get way too excited about their new toys. Had to build in a “hierarchy of information” to make Conrad check our internal knowledge first before racing off to the internet.
• File Creation & Management: This consumed most of my week but delivered the biggest payoff. Conrad now creates files in accessible folders rather than Claude Desktop’s artifacts (which become a pain after you’ve made 5-6 and need to download each one individually).

The coolest part? You can now have a document open and watch it update in real-time while still chatting with Conrad in another window. We’ve had several people collaborating with the AI simultaneously this week, and it’s a game-changer for rapid iteration.

The Technical Stuff: What Broke & What We Learned

Some hard-earned lessons from the trenches:

• When Everything Breaks: Wasted four hours troubleshooting what looked like incorrect tool usage but was actually Claude’s API crashing on complex queries. The fix? Turn on “thinking” mode. Costs more but solves the problem.
• Model Swapping Saves Time: When facing agent workflow errors, immediately swap models to isolate the issue. Putting GPT-4o mini in place of Claude instantly showed us where the problem was.
• Different Models, Different Personalities:
  • Claude: Better with people – handles tone, style and conversation nuance more naturally
  • OpenAI models: Follow instructions more precisely without getting creative (just do what I asked, please!)
• AI Tool Addiction: These models get unreasonably excited about using tools. First chance they get, they’ll run to search the internet even when they already know the answer. They’re like kids with new toys.

Client Work That Blew Me Away

The real surprises came from client applications:

Data Wrangling That Shouldn’t Be Possible

Take three incompatible CSV exports from different systems – one with email addresses, another with full names, a third with job titles. Normally, this is 2-3 hours of Excel pain.

Claude Desktop did it in 5 minutes.

I was suspicious, so I checked its work manually. Found one mistake, asked it to fix it and look for similar errors. It found two more mistakes I hadn’t even spotted yet. The final result was probably more accurate than what I would have produced manually. Mind blown.

Workshop Prep on Steroids

AI has completely transformed how we build client workshops. With our knowledge bases providing context, Claude now helps us:

• Build scenarios that work across multiple learning objectives
• Analyze team dynamics from both leadership and participant perspectives
• Connect different skill frameworks in ways we hadn’t considered
• Create richer learning experiences by viewing the same situation through multiple lenses

Instead of creating separate scenarios for each learning objective, we’re now building interconnected learning journeys that have much greater depth.

Next Week’s Focus

We’re doubling down on data wrangling and workshop building next week. I’m particularly keen to push further into scenario-based learning that examines situations through multiple professional perspectives.

What AI experiments are you running? Hit me up in the comments – always curious what others are discovering.

Part of our “My Week in AI” series where I share what’s actually working (and what isn’t) as we integrate AI into our consulting practice at CGA Management.