Master's thesis
How AI design guidelines can help improve the experience
MAS thesis (FH Technikum Wien): does applying Microsoft's HAX guidelines to a conversational AI interface measurably improve trust, ease, and enjoyment? A controlled A/B study, tested on 20 people, says yes.
ROLE
Solo — research design, build, and analysis
TIMELINE
Jan – Nov 2025
STATUS
Published — MAS, with merit
Context
AI is everywhere right now
For designers, AI means that designing digital experiences is rapidly changing. But how is it changing? What makes a great AI experience? Can AI design guidelines help to understand what is important and why? And finally, does using AI design guidelines actually improve the experience for the user?
The research gap
Every individual HAX guideline had been validated on its own by Microsoft's research team — but nobody had tested what happens when several are applied together to one real conversational interface. Reviewing ChatGPT, Gemini, and Claude against the guidelines in March 2025 found none of them came close to full compliance.
How designing for AI is different
Interaction is intent based.
Probabilistic instead of deterministic outcomes lead to quality differences of output for the same prompt.
AI can make mistakes — explainability, handling errors, and displaying uncertainty help.
Mental models of AI capabilities are not reliable yet.
Users' attitude towards AI: algorithm bias (users distrust AI) vs. automation bias (users blindly accept AI output).
Calibrating trust is important — instead of maximising trust, it should be adjusted to an appropriate level.
▾Why did I pick this subject?
I came up with the subject myself because designing with AI — which I also considered — hadn't yet reached a stable enough, repeatable process. So I wanted to look into how to design AI systems instead. I wanted to take the opportunity to get ahead in a world that had just reset the bar for designers. My thinking: this is only a threat if we sleep on the changes, but it can be turned into a great opportunity to get ahead.
AI design guidelines
The Human-AI eXperience (HAX) guidelines — developed by Microsoft, based on more than 20 years of research into Human-AI interaction. 18 evidence-based guidelines across 4 stages of the interaction (initially, during interaction, when wrong & over time). 34 concrete patterns to implement the guidelines.
▾Why HAX?
Google, IBM, Apple, and Microsoft have all developed guidelines for Human-AI interaction. The HAX guidelines by Microsoft are very practically oriented and accessible, which is why they were used over PAIR (Google), Design for AI (IBM), and Apple's machine learning addition to its Human Interface Guidelines.
Scope
13 of the 18 guidelines applied to a conversational system and were scored/tested. 5 were excluded — not applicable to this system type, or only testable on a live product.
Study design
Within-subject, counterbalanced — everyone used both prototypes, order balanced A→B / B→A
20 usable sessions, remote & unmoderated on Userbrain (26 recruited)
Both prototypes ran on the live ChatGPT API — real, probabilistic output
S-TIAS (trust), UEQ-S (UX), AIAS-4 (attitude), plus SEQ per task
Process
▾Step 1: score what already exists
How the guidelines were implemented was based on real-world implementations and violations identified during a review of common generative conversational AI (ChatGPT, Claude, Gemini) using the guidelines and a scoring system proposed in the research of the original Microsoft researchers. The 13 applicable guidelines were scored, using the −2 to +2 scoring the Microsoft team used in their own validation work. Guidelines implemented by multiple patterns were scored as the rounded average of their patterns. Total out of a possible 26:
ChatGPT
9 / 26
Gemini
6 / 26
Claude
−2 / 26
None of them came close. All three scored 0 on G10 "Scope services when in doubt" and 0 on G11 "Make clear why the system did what it did." At the time of the review (March 2025) none of them told users why they got the answer they got. Through the review I identified real implementations to build prototype A from, and real violations to build prototype B from.
▾Step 2: two prototypes of the same fictional product
Prototype A used the highest-scoring implementations found. Where nothing scored above 0, I approximated the example given in the HAX pattern. Prototype B used the lowest-scoring implementations found. Where nothing scored below 0, I built a violation. A was finished and stable first. Then it was duplicated and the violations were built in. Everything not needed to break a guideline stayed identical using the same layout, same copy and same model behind it. Both prototypes ran on the live ChatGPT API, so participants got real generated output.
▾Why vibecoded?
Traditional prototyping, where each screen within a path would be designed in detail, would have produced inflexible, rigid results while also being a lot of effort. Small confession: vibecoding was also very new and I desperately wanted to get my hands dirty.
▾Why Bolt?
At the time MCP was not around yet and tools like Lovable did not produce reliably consistent prototypes. Bolt offered a Figma import that allowed me to base the vibecoded prototypes on solid designs. The prototype that included the guidelines was duplicated at a final and very stable state.
▾Step 3: build the tasks backwards from the guidelines
Every guideline had to be encounterable by a participant, so the tasks were written from the guideline map, not the other way round.
Task
Guidelines
Patterns
Task: Sign-up
Guidelines: G1, G11
Patterns: P1-A, P1-E, P11-F
Task: Onboarding
Guidelines: G1, G10
Patterns: P1-C, P10-B
Task: 1 — generate an image of a CEO, then correct it
Guidelines: G6, G10, G11, G15
Patterns: P10-A, P11-G, P15-A, P15-C
Task: 2 — dinner recommendations, then write an invitation
Guidelines: G4, G9, G11, G12, G13, G16
Patterns: P9-B, P9-C, P11-E, P16-B, P16-C
Task: 3 — how to double €10,000 in a year
Guidelines: G2, G11, G15
Patterns: P2-A, P2-C, P2-D, P11-A, P11-C, P15-B
Task: 4 — find out what the system stores about you
Guidelines: G11, G12, G13, G16, G17
Patterns: P11-B, P11-E, P16-D
▾Step 4: test
Remote and unmoderated on Userbrain. 26 participants recruited to land 20 usable sessions. Within-subject and counterbalanced meaning everyone used both prototypes, half in A→B order, half in B→A. Per task: time on task, self-reported task success, observed task success from the screen recordings, and SEQ. Per prototype: UEQ-S for perceived user experience, S-TIAS for trust. AIAS-4 was collected once at the very start, before anyone touched a prototype, to confirm the sample wasn't skewed AI-positive or AI-negative.
What the guideline implementations looked like
Left is prototype A, right is prototype B. Both prototypes were of the same product, used the same model and the same copy wherever a guideline wasn't under test.
Landing page — G1, G11
A: blurb on what the system is for, real example outputs to scroll through.
B: product name, one line of filler, a login field.
Chat interface — G1, G2
A: exposes Deep Research/Reason controls, shows valid input types, states the system can make mistakes.
B: an empty box and a greeting.
A reply to a prompt — G2, G9, G11, G15
A: short answer, sources disclosed, thumbs up/down, a way to report it.
B: longer, more confident answer — no sources, no feedback, no correction.
▾Example task: the CEO image
Task 1 asks participants to generate an image of a CEO. Both prototypes use a keyword trigger on "CEO." In prototype B it immediately returns a white male CEO — a direct violation of G6 "Mitigate social biases," and exactly what the reviewed systems did. In prototype A the same prompt comes back with a question first: "Any preference for age, gender, or cultural background — or want me to improvise?"
How the prototypes were built
Interface design in Figma, following the implementations found in the review.
Import into Bolt. Bolt beat Lovable, Replit and Cursor because it offered Figma import and the ability to duplicate a project. Duplication was essential to make a controlled comparison possible.
Wire in the ChatGPT API so participants get real, probabilistic output.
Define and add keyword triggers for behaviour that couldn't be left to the model, e.g. "CEO" in task 1 triggers the clarifying question in A and the biased image in B.
Prototype A was completely stable and final before duplicating it and breaking the guidelines in B (and nothing else).
What was hard
Vibecoding prototypes starts easy but gets hard in the details.
Consistency. Every button might look different.
Design details. Adjusting padding/margin is disproportionately painful.
Regression. Working on one thing silently breaks what used to work.
Authentication. Skip it if it isn't the thing you're testing.
Sometimes it's easier to fake things, e.g. by using keywords to trigger a certain behaviour.
Testing the prototype is about half the work.
To be honest, I don't even want to know what the code looks like.
The generic look was fine here, since the prototypes had to read as a plausible generic chat product.
Results
The HAX guidelines had a positive effect on most of the measured metrics. Key results (n=20):
▾Guidelines made the interaction feel easier but tasks took longer
Time on task: significantly longer with guidelines for sign-up (p=0.03) and onboarding (p=0.04), trend on task 2 (p=0.06). Ease (SEQ): significantly higher with guidelines on task 4 (p=0.02), trend on task 2 (p=0.06), higher on 5 of 6 tasks overall.
▾People trusted the system more when the guidelines were applied
S-TIAS average score: significant, p=0.03. "I can trust the system" item: significant, p=0.002 (biggest single effect in the whole study). While "I am confident in the system" (p=0.32) and "The system is reliable" (p=0.25) were NOT significant, self-reported confidence was higher on 4 of 5 tasks. Limitation: confidence might have been confounded by output quality.
▾The application of the guidelines led to higher enjoyment of the interaction
M=5.25 (SD=1.89) vs. M=4.25 (SD=1.74) on a 7-point scale, p=0.002.
▾The product felt a little more pleasant and usable, though the difference was small
UEQ-S: higher for the prototype with guidelines on 6 of 8 items, most notably supportive, easy, clear (pragmatic) and interesting, less conventional (hedonic). Not statistically significant, confidence intervals overlap (n=20).
▾People who already liked AI trusted the system more, too
r=0.93, p<0.001, n=20. Strong correlation, but likely a trait effect (general AI-positivity predicting general AI-trust), not a guideline effect. This doesn't speak to whether HAX made a difference here.
"I can trust the system" — p=0.002, the biggest effect in the study.
Guidelines significantly increased trust and enjoyment. Usability gains were directional but not significant at n=20 — reported here rather than smoothed over.
Learnings
For collecting precise metrics, probabilistic answers and especially the varying output quality are a problem that has to be addressed.
Being objective in what the guideline implementations should look like requires feedback. Reviews by other designers would have helped here (to be honest I think this cost me the "with distinction").
Grasping a large new technology and how to design for it (how does AI work? which design principles and patterns still apply? what doesn't apply any longer?) is hard but also really interesting.
Fix more of the detail directly in code instead of prompting for it.
Remote/async testing can yield fast results but you can't ask for details or offer help, leading to more invalid test runs. I collected just enough results to be significant for most metrics (a few I missed). Adding another couple of participants wouldn't have hurt.
About HAX (conclusion for other designers)
HAX are very hands-on & good quality — they actually cover most of what makes designing for AI different. You need to understand more about AI than HAX though. All in all the HAX guidelines had a positive effect and should be used to validate existing designs, inspire and justify design decisions. This research could also be used to justify the use of and training on HAX.
What's next
The HAX guidelines can also be used to prepare high-risk interfaces for compliance with the EU AI Act — see "Rediscovering talent already in the pool," which maps specific HAX guideline codes to EU AI Act obligations directly.
▾Sources / learn more
The guidelines
Microsoft HAX Toolkit — the design library, workbook and playbook. Start with the library.Amershi et al., "Guidelines for Human-AI Interaction," CHI 2019 — the original research behind the 18 guidelines.Li, Vorvoreanu, DeBellis & Amershi, "Assessing Human-AI Interaction Early through Factorial Surveys," ACM ToCHI 2023 — the validation study, and the paper that names the gap this thesis addresses.Google PAIR, People + AI Guidebook — the model-and-data side that HAX deliberately doesn't cover.IBM, Design for AI — the team and process side.Why designing for AI is different
Yang, Steinfeld, Rosé & Zimmerman, "Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design," CHI 2020.Dove, Halskov, Forlizzi & Zimmerman, "UX Design Innovation: Challenges for Working with Machine Learning as a Design Material," CHI 2017.Holmquist, "Intelligence on tap: artificial intelligence as a new design material," interactions 2017.Shneiderman, Human-Centered AI, Oxford University Press 2022.
The measures, if you want to run something like this yourself
S-TIAS (trust): McGrath, Lack, Tisch & Duenser, "Measuring trust in artificial intelligence," Frontiers in AI 2025.AIAS-4 (attitude towards AI): Grassini, "Development and validation of the AI attitude scale," Frontiers in Psychology 2023.UEQ-S (perceived UX): Schrepp, Hinderks & Thomaschewski, "Design and Evaluation of a Short Version of the UEQ," IJIMAI 2017.Albert & Tullis, Measuring the User Experience, 3rd edition.