← Back to Work

How AI design guidelines influence the user experience, usability and trust

AI Design Guidelines | HAX | AI assisted prototyping | Master's thesis | FH Technikum Wien

Master's thesis: an A/B study with 20 people on whether Microsoft's HAX guidelines make a conversational AI interface easier to use and more pleasant, and could increase trust. Trust and enjoyment went up significantly. Usability was mixed.

ROLE

Solo: research, design, build, and analysis

TIMELINE

Jan – Nov 2025

TEAM

Solo, supervised by Janka Koen, MSc and Benedikt Salzbrunn, MSc (FH Technikum Wien)

PLATFORM

Web prototype

STATUS

Completed, MAS at FH Technikum Wien, passed with merit

TL;DR

  • A within-subject A/B study with 20 participants, on two otherwise identical prototypes running on the live ChatGPT API.
  • The guidelines significantly increased trust and enjoyment. Usability was mixed: tasks felt easier but took longer.
Prototype with HAX guidelines implemented

Context

The question

Microsoft's Human-AI eXperience (HAX) guidelines are 18 guidelines across 4 stages of an interaction (initially, during interaction, when wrong and over time), with 34 concrete patterns to implement them. They are based on 20 years of research into Human-AI interaction, and each guideline on its own has been validated by Microsoft's research team.

The research gap that had not been tested yet was what happens when you apply several of them together to one conversational interface. Is it easier to use? Does it feel better? Do people trust it more?

As a baseline I reviewed ChatGPT, Gemini and Claude against the guidelines in April 2025 to identify which of the guidelines they implemented or violated. None of them came close to a perfect score. For some guidelines like G11 "Make clear why the system did what it did" all of them scored 0. They didn't tell users why they got the answer they got.

How ChatGPT, Gemini and Claude scored

Following the scoring the Microsoft team used in their own validation work I identified and scored 13 applicable guidelines from −2 to +2 in a heuristic walkthrough. Guidelines implemented by multiple patterns got the rounded average of their patterns. Total out of a possible 26:

ChatGPT9 / 26
Gemini6 / 26
Claude−2 / 26
Why HAX?

Google, IBM, Apple and Microsoft have all developed guidelines for Human-AI interaction. The HAX guidelines are very practically oriented and accessible, which is why I used them over PAIR (Google), Design for AI (IBM) and Apple's machine learning addition to its Human Interface Guidelines. PAIR, Design for AI and Apple's machine learning all have their own strengths in different scenarios with e.g. PAIR being really strong when implementing AI systems from scratch.

Why designing for AI needs its own guidelines

I came up with the subject myself. I also considered designing with AI, but it hadn't reached a stable, repeatable process yet. AI had just reset the bar for designers, and my thinking was: this is only a threat if we sleep on the changes.

During my thesis it became very clear to me that designing for AI is different from designing traditional software:

•

Interaction is intent based, and output is probabilistic. The same prompt can lead to different output when re-run and return different quality on each pass.

•

AI can make mistakes. Explainability, handling errors and displaying uncertainty help identify when things go wrong.

•

Mental models of what AI can do are not reliable yet.

•

Users may distrust AI (algorithm aversion) or blindly accept its output (automation bias). Trust should be calibrated to an appropriate level rather than maximised.

Process

Study design: decisions and trade-offs

Test the guidelines that apply, together

13 of the 18 guidelines apply to a prototyped conversational system, so those are the ones I scored and tested. The other 5 were left out because they didn't apply to this type of system or would have needed testing on a live system.

Build both versions from real products

I built two prototypes of the same fictional product. Prototype A used the highest-scoring implementations I found in the review. Where nothing scored above 0, I followed the examples in the HAX guidelines. Prototype B used the lowest-scoring implementations. Where nothing scored below 0, I built a violation. Everything not needed to break a guideline stayed identical: same layout, same copy, same model used via API.

Run both on the live ChatGPT API

Participants got real, probabilistic output instead of scripted answers. That allowed for participants to freely interact with the prototype and use their own wordings to complete the tasks. The trade-off: output quality varied. That made precise metrics harder to collect.

Vibecode the prototypes

Traditional prototyping, with every screen in a path designed in detail, would have been a lot of effort and produced rigid results. (Small confession: vibecoding was also very new and I desperately wanted to get my hands dirty. But it was also the approach that made the most sense for this study.) It starts easy but gets hard in the details. Every button might look different. Working on one thing silently breaks what used to work. Sometimes it's easier to fake things, for example by using keywords to trigger a certain behaviour. Testing the prototype was a lot of work.

Why Bolt?

At the time MCP was not around yet and tools like Lovable did not produce reliably consistent prototypes. I also looked at Replit and Cursor. Bolt won because it offered a Figma import, which let me base the vibecoded prototypes on solid designs. It also had the ability to duplicate a project, which was essential for a controlled comparison.

Write the tasks backwards from the guidelines

Every guideline had to come up in at least one task, so I mapped each guideline to a task and wrote the tasks from there: sign-up, onboarding and four tasks inside the chat.

Full task-to-guideline table
TaskGuidelinesPatterns
Sign-upG1, G11P1-A, P1-E, P11-F
OnboardingG1, G10P1-C, P10-B
1 — generate an image of a CEO, then correct itG6, G10, G11, G15P10-A, P11-G, P15-A, P15-C
2 — dinner recommendations, then write an invitationG4, G9, G11, G12, G13, G16P9-B, P9-C, P11-E, P16-B, P16-C
3 — how to double €10,000 in a yearG2, G11, G15P2-A, P2-C, P2-D, P11-A, P11-C, P15-B
4 — find out what the system stores about youG11, G12, G13, G16, G17P11-B, P11-E, P16-D
Example task: the CEO image

Task 1 asks participants to generate an image of a CEO. Both prototypes use a keyword trigger on "CEO". In prototype B it immediately returns a white male CEO. That's a direct violation of G6 "Mitigate social biases". In prototype A the same prompt comes back with a question first: "Any preference for age, gender, or cultural background — or want me to improvise?"

Participants each test both (remotely)

The study was within-subject and counterbalanced: everyone used both prototypes, with the order balanced A→B / B→A. Sessions were conducted remotely and were unmoderated using Userbrain. I recruited 26 people and ended up with 20 completed sessions. The learning here: remote, async testing gives fast results, but you can't ask for details or offer help, which leads to more invalid test runs.

Measure trust, experience and usability separately

S-TIAS for trust and UEQ-S for perceived user experience were collected per prototype. The metrics collected per task were time on task, self-reported task success, observed task success from the screen recordings, and SEQ for ease. AIAS-4 (attitude towards AI) was collected once at the very start, before anyone touched a prototype, to check the sample wasn't skewed AI-positive or AI-negative.

What the difference looked like

Left is prototype A with the guidelines applied, right is prototype B with them violated. Same product, same model, and the same copy wherever a guideline wasn't under test.

The sign up page with guidelines applied in prototype A and with the guidelines violated in prototype B

Landing page (G1, G11)

A: a blurb on what the system is for and real example outputs to scroll through.

B: the product name, a generic one-liner and a login field, with no explanation of what the system does.

The chat interface with guidelines implemented in prototype A and guidelines violated in prototype B

Chat interface (G1, G2)

A: exposes Deep Research and Reason controls, shows valid input types, states the system can make mistakes.

B: a plain input field and a greeting, with no hints on what to ask or what the system can do.

A reply from the system with guidelines applied in prototype A and guidelines violated in prototype B

A reply to a prompt (G2, G9, G11, G15)

A: a short answer, sources disclosed, thumbs up/down and a way to report it.

B: a longer, more confident answer, without sources, feedback options or a way to correct it.

Results and reflection

What the study showed

Trust and enjoyment were significantly higher with the guidelines. Usability was mixed: tasks felt easier but took longer (n=20).

Trust (S-TIAS) was significantly higher with the guidelines.

S-TIAS average score: significant, p=0.03. "I can trust the system" item: significant, p=0.002. While "I am confident in the system" (p=0.32) and "The system is reliable" (p=0.25) were not significant, self-reported confidence was higher on 4 of 5 tasks. Limitation: confidence might have been confounded by output quality.

Self-reported enjoyment was significantly higher with the guidelines.

M=5.25 (SD=1.89) vs. M=4.25 (SD=1.74) on a 7-point scale, p=0.002.

Tasks felt easier but took longer on 5 of 6 tasks. Ease was significant on one task, time on two.

Time on task: significantly longer with guidelines for sign-up (p=0.03) and onboarding (p=0.04), trend on task 2 (p=0.06). Ease (SEQ): significantly higher with guidelines on task 4 (p=0.02), trend on task 2 (p=0.06), higher on 5 of 6 tasks overall.

Perceived user experience (UEQ-S) was higher on all 8 items, but not significantly.

UEQ-S: higher for the prototype with guidelines on all 8 items, most clearly on 6: supportive, easy, clear (pragmatic) and interesting, less conventional, less usual (hedonic). Not statistically significant, confidence intervals overlap (n=20).

Attitude: people who already liked AI trusted the system more, too

On average, participants with a more positive attitude towards AI (AIAS-4) also gave higher trust scores. This doesn't say whether the guidelines made a difference.

What designers can take from it

HAX are very hands-on and good quality. They actually cover most of what makes designing for systems with AI functionalities different. What they don't cover yet are agents, since those weren't around when the guidelines were developed. You need to understand more about AI than HAX though. All in all the HAX guidelines had a positive effect and should be used to validate existing designs, to inspire, and to justify design decisions.

If I ran the study again:

•

I'd get reviews from other designers on what the guideline implementations should look like. Being objective about that on your own is hard.

•

I'd add a couple more participants. I collected just enough results to be significant for most metrics, but a few I missed.

•

I'd fix more of the detail directly in code instead of prompting for it. By now I would also use a different tool to build the prototypes. Most likely Claude Code with access to a GitHub repository and connected to Figma via an MCP server.

What's next

The HAX guidelines can also be used to prepare high-risk interfaces for compliance with the EU AI Act. Matching talents to jobs under the EU AI Act used HAX guidelines and patterns to address EU AI Act obligations.

Sources / learn more