To evaluate an AI vendor, ask what job the product does better than the general tools you already have, which underlying model it uses, where your data goes, and what evidence it has from customers like you. Most AI products sold to businesses in Japan are built on a foundation model from OpenAI, Google, Anthropic or a similar provider. That is fine. What you are paying for is the workflow, the integrations and the support around the model, so judge those, not the claims about “proprietary AI.”

I started paying close attention to this after reading the sales material of several Japanese AI startups promising to clone the skills of a company’s best salesperson or customer service rep into an always-on digital workforce. The pitch is attractive. The public evidence behind it was usually thin. The same pattern shows up in smaller vendors selling chatbots, AI-OCR, meeting summaries and “AI DX” packages to SMEs, so the questions below apply to all of them.

What are you actually buying from an AI vendor?

Very few companies selling AI products to businesses train their own large language model from scratch. It is enormously expensive, and the general-purpose models from the major labs are hard to beat. Most products combine four things:

  • A foundation model accessed through an API, often from OpenAI, Anthropic or Google, sometimes a Japanese model or an open-weight model.
  • Prompts and instructions tuned for a particular task, such as answering customer inquiries in polite Japanese.
  • Your data, retrieved from documents, a CRM or a knowledge base and passed to the model with each request.
  • An interface and integrations with tools such as kintone, Salesforce, LINE, Chatwork or Microsoft Teams, plus admin controls and support.

Every one of those is real engineering work, and a vendor that does them well can be worth paying. The problem arises when the marketing describes this combination as a new form of intelligence, or as “automated human connection.” That language hides what you need to know to judge the product: how good the integration is, how your data is handled, and what happens when the model underneath changes.

Which questions should you ask an AI vendor?

Which model does it run on, and can that change?

Ask directly which model or models the product uses and whether the vendor can switch. A product that runs on a single provider is exposed to that provider’s pricing, outages and policy changes. A vendor that tells you “our own proprietary AI” and will not say more is either hiding a simple setup or does not understand its own stack. Neither is reassuring.

What can it do that ChatGPT, Copilot or Gemini cannot?

Many small businesses already pay for Microsoft 365 or Google Workspace, which now include AI assistants in many plans. Before adding a specialist product, ask the vendor to show the specific task where their tool beats a well-configured general assistant. Good answers are usually about integration: it reads your kintone records, it posts into Chatwork, it follows your approval flow. Weak answers are about the model being “smarter.”

Where does our data go?

Ask where data is stored and processed, whether it is used to train any model, how long it is retained, who at the vendor can see it, and which subprocessors receive it. If you handle personal information, Japan’s Act on the Protection of Personal Information (APPI) applies, and the Personal Information Protection Commission has warned businesses to check how generative AI services use data they input. The Japan SaaS compliance guide covers the wider checks.

What evidence do you have from customers like us?

Logos on a website and a funding announcement are not evidence. Ask for a reference customer of similar size and industry you can actually speak to, and ask what measurable change they saw. A vendor with real results is usually glad to connect you. If the only proof is a press release about a funding round, treat the product as unproven.

Who handles errors?

Every language model produces wrong answers sometimes. Ask what the product does when the AI is unsure, how staff review or correct outputs, and whether there is a log of what the AI said to customers. For customer-facing use in Japanese, test it with your own awkward real inquiries, not the vendor’s demo script.

What does it cost at our real volume, and how do we leave?

Get pricing for your actual number of users or conversations, including overage charges. Then ask how you export your data, prompts and configuration if you cancel. Lock-in with AI vendors looks a lot like lock-in with traditional SIers: the setup lives in a system you do not control.

Why does investor backing not prove a product works?

Many AI startups in Japan have strong founders from major technology and HR companies, respected venture capital firms, and angel investors from well-known SaaS businesses. That is a signal the company is serious. It is not a signal the product works in your business.

Investors are betting on a market and a team, often at an early stage when there is little customer evidence. Their involvement gets reported, the reporting gets repeated, and buyers start treating a funding round as a review. It is worth separating the two. A well-funded vendor can still sell you something that does not fit your process, and a small vendor with no press coverage can be the right choice.

What do human-sounding AI products actually do?

Some of the most ambitious pitches claim to reproduce empathy, wit or the rapport of a top performer. A language model predicts likely text from its training and instructions. It can be polite, follow a sales script and adapt its tone well. That can be genuinely useful for handling routine inquiries in the evening or drafting first replies.

It does not understand the customer in the way a good salesperson does, and it cannot take responsibility for a promise. When the product is sold as a replacement for your best people, judge it on the parts of their job it can actually do, and plan who handles the rest. I go into where to draw that line in which work should stay with people when you adopt AI.

How to run a fair trial

Define the job first. Write down the task, how it is done now, how long it takes and what a good result looks like.

Test on your own material. Use real inquiries, documents and edge cases, in Japanese and English if you work in both.

Compare against the tools you already pay for. Give the same task to Copilot, Gemini or ChatGPT on a business plan. The gap tells you what the specialist product is worth.

Measure for a few weeks. Track time saved, error rate and how often staff had to correct the output.

Check the exit. Confirm you can export your data before you sign an annual contract. If generated images or text will be published, check the Japanese copyright rules on AI output as well.

These steps also expose a common underlying problem: the process or data the AI needs does not exist in usable form yet. In that case no vendor will fix it, and the work starts earlier, as I explain in why AI adoption stalls in Japanese SMEs. Procurement habits matter too; see technology adoption in Japan fails at procurement.

Getting a second opinion

If a vendor is pitching you an AI product and you want an independent view before you commit, I can review the proposal against how your business actually works. That can be part of a Diagnostics review, or ongoing through Ongoing Stewardship if you face vendor decisions regularly. You can also describe the decision you are facing.

Further reading: what to check before connecting business apps to ChatGPT · Japan’s copyright rules for businesses using AI · what work should stay with people when you adopt AI · what belongs in a Japan SME technology stack · TetsuClaw: AI work OS for operators in Japan