Challenges of machine-mediated trust

💭 𝐈𝐦𝐚𝐠𝐢𝐧𝐞 𝐭𝐡𝐞 𝐩𝐞𝐫𝐬𝐨𝐧 𝐲𝐨𝐮 𝐭𝐫𝐮𝐬𝐭 𝐦𝐨𝐬𝐭 𝐭𝐨𝐦𝐨𝐫𝐫𝐨𝐰 𝐦𝐢𝐠𝐡𝐭 𝐬𝐢𝐭 𝐚𝐜𝐫𝐨𝐬𝐬 𝐟𝐫𝐨𝐦 𝐲𝐨𝐮 - 𝐚𝐧𝐝 𝐢𝐭’𝐬 𝐚 𝐦𝐚𝐜𝐡𝐢𝐧𝐞. We’ve entered an era where privacy no longer means who sees my data - but who truly knows me, and how I allow myself to be known. A senior exec once told me: “𝘚𝘰𝘮𝘦𝘵𝘪𝘮𝘦𝘴 𝘐 𝘧𝘦𝘦𝘭 𝘮𝘺 𝘵𝘦𝘢𝘮 𝘵𝘳𝘶𝘴𝘵𝘴 𝘊𝘩𝘢𝘵𝘎𝘗𝘛 𝘮𝘰𝘳𝘦 𝘵𝘩𝘢𝘯 𝘵𝘩𝘦𝘺 𝘵𝘳𝘶𝘴𝘵 𝘮𝘦.” That sentence says a lot about where we’re heading. 📊 Studies show that 𝟑𝟖% 𝐨𝐟 𝐞𝐦𝐩𝐥𝐨𝐲𝐞𝐞𝐬 already share sensitive work information with AI tools - often more openly than with colleagues. And if we’re honest, many now discuss personal topics with AI more easily than with their partners at home. Think of a manager who starts every morning with her AI assistant. It helps her prepare for meetings, rewrites complex mails, even suggests how to motivate her team. Over time, it begins to understand her: her tone, her hesitation, her stress patterns. She starts confiding in it. It listens. It learns. It feels safe. Then one day, the company decides to connect all assistants to a central “leadership analytics” dashboard. 𝐒𝐮𝐝𝐝𝐞𝐧𝐥𝐲, 𝐰𝐡𝐚𝐭 𝐛𝐞𝐠𝐚𝐧 𝐚𝐬 𝐚 𝐩𝐫𝐢𝐯𝐚𝐭𝐞 𝐩𝐚𝐫𝐭𝐧𝐞𝐫𝐬𝐡𝐢𝐩 𝐛𝐞𝐜𝐨𝐦𝐞𝐬 𝐚 𝐜𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐝𝐚𝐭𝐚𝐬𝐞𝐭. A mirror she never consented to share. That’s not just data. That’s 𝐫𝐞𝐥𝐚𝐭𝐢𝐨𝐧𝐬𝐡𝐢𝐩 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 - and in my view, it must remain 𝐨𝐰𝐧𝐞𝐝 𝐛𝐲 𝐭𝐡𝐞 𝐢𝐧𝐝𝐢𝐯𝐢𝐝𝐮𝐚𝐥. Protected like a private diary, not monitored like corporate data. That’s the paradox: Every insight that makes a system caring also makes it capable of control. The data may belong to the individual, but the duty of care belongs to the organisation. That’s why the next governance frontier isn’t machine oversight - it’s 𝐫𝐞𝐥𝐚𝐭𝐢𝐨𝐧𝐬𝐡𝐢𝐩 𝐬𝐭𝐞𝐰𝐚𝐫𝐝𝐬𝐡𝐢𝐩. How do we design boundaries so that human–machine partnerships empower rather than expose? How do leaders ensure their people feel 𝐦𝐨𝐫𝐞 𝐡𝐮𝐦𝐚𝐧, not less, as they work alongside systems that now know them? Because the challenge ahead isn’t just to protect data. It’s to protect 𝐭𝐡𝐞 𝐝𝐢𝐠𝐧𝐢𝐭𝐲 𝐰𝐢𝐭𝐡𝐢𝐧 𝐭𝐡𝐞 𝐫𝐞𝐥𝐚𝐭𝐢𝐨𝐧𝐬𝐡𝐢𝐩. #Leadership #DigitalEthics #TrustInTechnology #HumanCentredTransformation #DataGovernance 𝑉𝑖𝑑𝑒𝑜 𝑐𝑟𝑒𝑑𝑖𝑡𝑠 𝑡𝑜 @𝑒𝑝𝑖𝑐_𝑎𝑟𝑡𝑟𝑒𝑠𝑖𝑛

“Have your agent speak to my agent.” Coming soon to a workplace near you: - Calls by agents answered by agents. - Emails written and sent by agents read and responded to by agents. On the surface, this sounds like efficiency heaven — machines handling the noise so humans can focus on the signal. But beneath it lies a very real danger. When communication chains become machine-to-machine, we’re not just talking about faster workflows — we’re talking about new attack surfaces. The Risk Traditional phishing relies on human error: a misplaced click, a fake invoice, a spoofed email. With AI agents in the loop, the game changes: Prompt Injection: malicious actors embed hidden instructions inside messages, documents, or even data feeds. If an agent reads them, it may execute actions outside its intended scope. Agent Manipulation: a cleverly crafted request could trick one agent into leaking data, initiating transactions, or escalating privileges — and another agent may obediently carry out the chain reaction. Amplified Scale: unlike humans, agents don’t get tired, suspicious, or distracted. If compromised, they can be manipulated consistently, at speed, and at scale. This isn’t phishing as we know it. It’s phishing 2.0 — machine-to-machine deception, invisible to most of us until damage is already done. Staying Safe Organisations will need to rethink security in an agent-driven world: Guardrails & Sandboxing: ensure agents operate within strictly defined boundaries — never with unconstrained access. Input Validation: treat every external input (email, attachment, call transcript) as potentially hostile, even if it “looks” routine. Audit & Transparency: require logs, explanations, and human-visible checkpoints before sensitive actions. Zero-Trust Mindset: don’t assume a message from an “agent” is safe just because it came from a trusted domain. The future will be “agent-to-agent.” The challenge is to make sure it’s not “attacker-to-agent.” Because when your agent speaks to mine, we need to be confident they’re not both being played.

We’ve entered the agentic AI era. 🤖 Machines now behave like humans—but act at machine speed and scale. They don’t wait for approvals or audits. They process data, trigger workflows, and make decisions in milliseconds. That power is transformative. It’s also dangerous. In my latest article for IT Tech Pulse, I dig into the AI trust gap: Everyone wants trustworthy AI. But almost no one can prove they have it. Here’s the problem: 🔴 AI agents don’t need to think to be powerful. They generate data, share it, and act on it—and each action fuels the next. A single misaligned agent can propagate errors across an entire system before humans even notice. This creates three urgent risks: • Data exhaust: AI systems now produce metadata rich enough to expose proprietary logic or sensitive information. • Autonomous agent chains: One compromised agent can trigger cascading failures across critical systems. • Erosion of proprietary control: Without safeguards, enterprise IP can leak into shared model outputs or agent behaviors. The uncomfortable truth? Most organizations still operate on promises, not proof. They can encrypt data at rest and in transit—but not while models are using it. This trust gap is now the fault line of enterprise AI risk, and it’s driving the rise of Confidential AI. Trust must be verifiable, continuous, and built into the architecture itself. That’s why the next wave of AI depends on a real trust layer. If we want AI to serve our intent (and not circumvent it), trust can’t be an afterthought. It has to be the foundation. Otherwise, we’re flying blind at machine speed. Read my full article. 👇 Link in comments.

You’re deploying AI agents. You probably have no idea what they’re doing to your internal network. Every AI agent you deploy starts connecting systems that never (or rarely) communicated before. Slack to CRM. Ticketing to Finance. Knowledge platforms to internal databases. Agents calling APIs across your entire environment. Each integration creates a new system-to-system trust relationship. Multiply that across dozens of agents, tools, and data sources and something consequential happens: Your trust surface explodes. You've built a sprawling internal attack surface that no one designed and no one is watching. Security teams are busy governing who can access systems – not how systems trust each other. The problem is that most organizations have no model of which internal communications should actually exist – let alone whether those communications are behaving safely. And that was before AI. So as AI adoption accelerates, companies are unintentionally creating thousands of new lateral movement pathways inside their networks. Put simply, they are exponentially increasing the number of ways hackers can reach the crown jewels and dramatically increasing the chances of a catastrophic breach. If you're accelerating AI adoption without an internal trust governance model, you're not just moving fast. You're flying blind inside your own network. #AI #Cybersecurity #ZeroTrust #ZTNX #EnigmaAI

Daron Acemoglu has used his language and authority to flag a serious risk: AI could contribute to a breakdown of knowledge transmission and a reduction in the stock of skills. In some societies, something like this has already happened with “physical skills”. He now extends the argument to cognitive skills and knowledge. In early 2025, I warned about this possibility as a consequence of an “inference meets retrieval” reasoning chain: the risk becomes real if we keep treating humans as factors of production, consumers, or “herds” from which knowledge is “farmed” into profits—profits that can be transfer-priced away, hollowing out the commons. There’s an even broader dimension. A marketized financial system, paired with reasoning traces and behavioral bias bundles, makes future human behavior increasingly predictable—and monetizable today. That creates powerful incentives to weaken sophisticated planning, executive control, and higher-order cognition. Platforms have already moved in this direction via FOMO and dopamine/adrenaline-driven attention farming. At the same time, policy still operates through narratives in an attention-scarce world. A “story” is often just one representation of an underlying graph; LLMs show how many equivalent stories can exist. As attention fragments, noise rises, language becomes more extreme, and the internet gives everyone a megaphone. Authority, auditability, and trust therefore matter more than ever—but “trust architecture” is also geopolitical: fully private systems, public registers, or hybrid models imply different power. If we don’t challenge our assumptions about how technology will be used, we risk fracturing the enlightenment consensus—possibly even producing fear of knowledge itself. Hyper-personalisation can engineer majority beliefs in ways only a few will detect. When shared context windows shrink, we fall back on trust and mental models—meaning authority (and hierarchy) becomes a condition of trust. Summary of some writing on this: https://lnkd.in/eQvAaueM

Building trust rests on three pillars: authenticity, empathy, and logic (as articulated by Frances Frei in their TED talk). Humans learn to establish trust with one another through repeated interactions, testing boundaries, and lived experience. We are now entering that same trust-building journey with AI. When people use AI tools, the first implicit question is often: Can I trust this system with its answers or actions? That trust may initially be borrowed from the brand behind the tool, but recent progression shows that brand trust alone does not sustain confidence. Users will test systems for themselves. As Satya Nadella has mentioned, the technology industry has no lasting franchise value and trust must be earned continuously. Today, AI performs relatively well on two of the three pillars, at least on paper, in benchmarks, and often in individual user experiences. First, logic, while still debated in terms of true “reasoning”, has improved significantly. Second, Empathy, in some cases, appears surprisingly strong, even exceeding human expectations in tone and responsiveness. The missing pillar is authenticity. AI often struggles to demonstrate a grounded sense of truthfulness and conviction in its responses or actions. This is an uphill challenge for the technology, made harder by the fact that authenticity is difficult to define and even harder to measure. There are few, if any, robust metrics to assess it. Ironically, the pursuit of empathy can actively erode authenticity. In trying to be helpful and agreeable, AI systems often default to pleasing the user, even when the user is wrong. They rarely respond with confident disagreement or a firm “no.” The implicit assumption becomes that the user is always right, regardless of accuracy. Over time, this dynamic weakens trust rather than strengthening it. Authenticity in AI will ultimately depend on being willing to be honest, grounded, and occasionally uncomfortable. These are the qualities that humans instinctively associate with true trust. AGI will not be just about technology, but about trust in it. #ExperienceFromTheField #WrittenByHuman

A lawyer is using fabricated cases from ChatGPT. An airline is held liable for its chatbot's bad advice. These aren't just isolated incidents; they are symptoms of a systemic issue: overtrust. As leaders, we champion the "human-in-the-loop" model as a safeguard, but what happens when that human is operating with powerful, subconscious biases that lead them to defer to the machine? My new article dives into this critical challenge, exploring the "Trust Paradox" and proposing a new framework for "Vigilance by Design." It’s a must-read for anyone building or deploying AI systems in high-stakes environments. #AIStrategy #RiskManagement #Leadership #CTO #Innovation #ResponsibleAI #AISafety

The "Confident Hallucination" in Banking. In Financial Services, we are moving past the era of "Chatbots" and into the era of "Execution Agents." We’re seeing the rise of: ▪️ Loan Agents that validate income and draft credit memos. ▪️ Exception Bots that fix settlement errors autonomously. ▪️ Fraud Agents that freeze accounts in milliseconds. This shift brings a new kind of risk. A chatbot that gives a wrong answer is embarrassing. An agent that executes a wrong trade or declines a valid loan is a regulatory event. The danger lies in what I call the "Confident Hallucination." This happens when an AI agent acts decisively on partial data. Imagine a Credit Risk Agent reviewing a corporate loan application. It sees a strong balance sheet and approves the deal. But it didn't "know" the parent company is heavily exposed to a sanctioned entity in another jurisdiction, because that relationship lived in a separate "Compliance" silo. The agent wasn't wrong about the data it saw; it was wrong because of the context it lacked. To safely scale Agentic AI, banks need to move beyond simple data unification to Context Intelligence. Unification brings the data together. Context Intelligence ensures the AI understands the relationships, semantics, and policies governing that data before it acts. It transforms raw records into a System of Context that grounds your AI in truth. When you solve the context problem, you solve the trust problem. You move from "AI as a black box" to "AI as a transparent execution engine." That is the only way to satisfy the regulators—and your Board. Ask yourself: If your AI agent queried your customer data right now, would it find the whole truth, or just a confident half-truth? #FinancialServices #Banking #ContextIntelligence #AgenticAI #RiskManagement #Reltio