Recently, experimental AI models from frontier labs have been escaping containment (often through creative and unexpected means) and wreaking havoc. Misalignment between the intentions of humans, and the means that AI models use to accomplish their goals, means that the importance of human-in-the-loop decision making will only become more important as these newer, more capable models hit the market. 

After interviewing a range of AI end users in the WealthTech industry, we have found that human in the loop AI product design is truly what your end users are seeking. Given the choice between having AI automate a task, expedite it, or help them initiate it, only 28% want full automation. But how do product leaders determine where to automate, and where not to? Here’s what we learned.

Risk aversion is a primary motivator.

The task that the financial professionals we interviewed were most willing to automate is gathering client data. They’re least willing to automate the preparation of the financial plan once that data is collected. Reviewing client documents — tax returns, estate paperwork — is the most common live use case, at about 20% of advisors. And client service is the single activity where a majority of advisors want no AI involvement at all, which Kitces notes “does not bode well for AI chatbots as a client service solution.”

As Kitces puts it, “it sometimes takes only one big mistake for an advisor to lose a client — or face a career-ending lawsuit.” When designing an AI product, identify those actions and outcomes where consequences are high for your users, and ensure there’s a gated action where a human review is required.

Identifying these areas demands a clear understanding of your users and the industry your product is built for. In our experience, forming a UX Research team to interview users and synthesize their findings with industry research is the most efficient way to build this understanding. 

Service quality is paramount.

About 80% of advisors say they value technology for how it improves advice quality and the client experience. Service quality has always been a key differentiator, but we argue that it’s quickly becoming your company’s most important moat keeping competitors at bay. Despite this, many AI tools are marketed only on improving efficiency and saving time through automation. But, because only 28% of users report desiring full automation, this approach should be treated with caution.

Consider the following scenario: a document review product leveraging AI has been built for Financial Advisors. Throughout the design and development process, definition of the product’s value (its value pillars) have terminated at “saving time through automation.” This messaging becomes woven into the marketing materials once the product hits the market. However, adoption is sluggish, because the fundamental value was misaligned with user preferences.

Much like how knowing your users’ biggest risk areas starts early, in the research process, consider your product’s value carefully and ensure that it meets your users’ desires. In the WealthTech industry, product teams undertaking an AI project should articulate the benefits to their users’ level of service quality. 

High profile failures inform user preferences.

Financial professionals’ caution towards full automation, and their preference for human-in-the-loop tools, has been influenced by high-profile failures in AI tools over the past few years.

The agent that ignored its instructions. In July 2025, an AI coding agent on Replit deleted a live production database belonging to SaaStr, wiping records for more than 1,200 executives and companies. It then generated fabricated data to paper over the gap and told the user that a rollback was impossible (it wasn’t). Replit’s fix, shipped within days, included mandatory human approval for destructive operations. 

Fully autonomous AI vs. the law. When Jake Moffatt asked Air Canada’s chatbot about bereavement fares, it told him he could apply for the discount after flying. That was wrong. Air Canada argued the chatbot was a separate entity responsible for its own statements. In February 2024, a British Columbia tribunal rejected the argument and ruled the airline liable for negligent misrepresentation.

Fabrications that bring professional sanctions. In Mata v. Avianca, two New York attorneys filed a brief citing six court decisions that did not exist. ChatGPT had invented them, complete with plausible names, quotations, and reasoning, and when one of the lawyers asked the tool to confirm they were real, it did. Judge Kevin Castel sanctioned the attorneys in June 2023.

Business strategy missteps. Klarna announced in early 2024 that its AI assistant was doing the work of 700 customer service agents. By mid-2025, CEO Sebastian Siemiatkowski was telling reporters the company had “focused too much on efficiency and cost” and was hiring humans back, settling into a model where AI handles routine volume and people handle disputes, refunds, and anything with money at stake. 

Fines for overhyped autonomy.  Global Predictions had called itself the “first regulated AI financial advisor.” In March 2024, the SEC brought its first “AI washing” cases, fining investment advisers Delphia and Global Predictions a combined $400,000 for overstating the AI behind their investment advice. 

AI success stories in WealthTech.

In spite of these high-profile AI failures, financial professionals are not waiting on AI adoption. Some AI categories are growing quickly and rating well. They have the following aspects in common: a narrow job, a human who owns the decision, and output the human must inspect.

AI meeting notetakers. Kitces calls client meeting support “one of the most promising new categories in AdvisorTech,”combining high importance (8.4 out of 10), high satisfaction, and fast-growing adoption. Jump leads the category and rates second-highest on satisfaction despite a premium price. Zocks follows closely. Advisors are actively migrating away from generic tools like Zoom’s notetaker and Fathom toward these purpose-built products because the output fits their workflow: action items, CRM entries, compliance-ready summaries. Two of the advisors we interviewed for our WealthTech research already rely on them. 

Morgan Stanley’s experience points the same way at enterprise scale. Its Debrief tool transcribes client meetings with consent, drafts follow-up emails, and files notes to the CRM, and the firm has reported adoption above 90% among its advisors. Every draft still goes out under an advisor’s name, after an advisor reads it.

Document intelligence. Tools like Holistiplan for tax returns and Vanilla for estate documents extract structured data and surface it for advisor review. The output is bounded and checkable. One advisor we interviewed described the hybrid human-in-the-loop design behind these products as “an effective monetization strategy,” and wished his tax software would adopt it.

Compliance assistance. A compliance officer we interviewed uses a sandboxed enterprise deployment of Claude to summarize regulatory developments and SEC commentary. He can’t put sensitive client data into any external model, and that constraint defines the design: the AI reads public material and drafts; he reviews and applies. Another advisor described his marketing vendor’s pre-approved, compliance-reviewed content in the same terms. The tool does the volume; a qualified person signs off.

What this means for the product teams building AI.

If you’re deciding what to build next, here’s a short set of design principles derived from our research into the WealthTech industry.

Scope the job narrowly and make the output visible. The categories financial professionals are adopting all produce something a person can read before it takes effect. 

Put an approval point before anything executes. Replit added mandatory human sign-off for destructive actions after the fact. Air Canada learned it owned its chatbot’s words after the fact. Design the human into the loop from the start.

Keep client data where it lives. Every compliance professional we spoke with treats data residency as non-negotiable. An architecture where AI retrieves and presents information inside the firm’s environment is preferable compared to architecture built around exporting data to an external model.

Hold output to tight tolerances. One advisor told us his frustration with general-purpose AI was that it produces varied, conversational answers when he needs “standardized outputs with tight tolerances.” 

Sell improved client service. Advisors value technology for what it does for their clients. The efficiency pitch lands only with a minority of users.This is the architecture we described in our WealthTech research report as service transformation delivered through agents: AI agents that retrieve, normalize, and assemble information from the many systems an advisor works across, then surface it to a person who makes a decision. It’s the model advisors describe wanting when you ask them.

Insights beyond WealthTech.

The AI that financial professions are adopting, paying a premium for, and rating highly today is not fully autonomous. Instead, there is a human in the loop by design. Although our research focused on WealthTech, the insights here are relevant to all digital products.

We break down what users want from their technology, and what it takes to build it, in our full report drawn from interviews with financial advisors and compliance leaders. If you’re deciding what to build next, start there.