Synthesizing Design Methods for Probabilistic AI
THE 5 NO’s
Prologue
NO clear method to design for AI.
NO predictions that capture evolving nature of AI.
NO humans in the loop. Mismatch in agency and automation.
NO human-centered AI. Focus is on accuracy of outcomes.
NO emphasis on contextual values (trust, human agency).
Evolving AI has been elusive to design for. We propose a contextual model finding an overlap between human expectations with machine’s capabilities to build trust.
Summary
PROBLEM
How do you design for AI considering its probabilistic and adaptive behavior, finding a sync between human expectations and evolving AI capabilities?
SOLUTION
We took a human-centered approach, framing design methods on transparency and trust. Measuring the effectiveness through task load.
IMPACT
After conducting a seminar study, we observed a 11% decrease in workload among our participants, measured via NASA-task load indexing protocol.
KEY DECISIONS
Narrowing down focus on the right problem, brainstorming on what values are key for human-AI interactions. Identifying the nuances of probabilistic AI.
MY ROLE
Lead UX Researcher, synthesized research questions through literature review. Designed research study for concept validation, along with identifying research protocols.
MY TEAM
1 Lead Researcher (Me), 3 Researcher, 1 Designer, 1 Data Scientist, 1 Supervisor
with ai vs. for ai
1 Scoping the project
Human Expectations and Evolving Capabilities
Finding the right balance between human expectations and machine capabilities enables designers to find optimal solutions when designing human-AI interactions (for AI). This project focuses on designing for AI.
Design FOR AI
Designing for AI focuses conveying AI capabilities and finding a fit with human expectations.
Design with AI
Designing with AI focuses on using AI’s generative capabilities to expedite the design process.
the four levels
2 Breaking down the problem
After breaking down AI complexity, it becomes imperative to understand human expectations of AI. For this study, I focus on level 3 and 4 systems since they are the most complex to design for.
1
Probabilistic Systems
Simplest system among the 4 levels of complexity. Probabilistic systems have fixed number of outputs.
2
Adaptive Systems
Adaptive systems, although a bit complex compared than previous, identifying the right demographics enable the designers to prototype effectively.
3
Evolving Probabilistic Systems
Evolving probabilistic systems is where the problem arises. The capability of the system changes due to the inclusion of a dynamic dataset.
4
Evolving Adaptive Systems
Evolving probabilistic systems is where the problem arises. The capability of the system changes due to the inclusion of the dynamic dataset, making it difficult to identify the right scenarios.
scoping domain
3 Conducting the Research
Primary Research
I conducted primary research which included interviewing people from academia, primarily professors along with industry professionals actively working with AI.
12
Academia
We interviewed many professors and people working in academia to understand their priorities when it comes to understanding human-AI interactions.
04
Industry professionals
In addition to academicians we also interviewed industry professionals to understand their challenges when designing for AI.
Secondary Research
In order to add richness to our findings, I along with my team also conducted literature review of several research papers spanning across AI, perception, and neural networks.
23
Research papers
We interviewed many professors and people working in academia to understand their priorities when it comes to understanding human-AI interactions.
data analysis
4 Thematic Diagramming
The results of this study are based on qualitative coding shedding light on principles of human-AI interactions. Transcripts were generated from 6 hours of recorded data.
Emerged Themes
60
Trainability
43
Deterministic UX
44
Improvements
31
Explainability
I identified four emerging themes along with their frequency of occurrence through interview transcripts and secondary research. This helped us in identifying the key factors for AI design.
User Quotes
Furthermore, I have illustrated few user quotes which reflect the sentiment of our participants and their connections with the identified themes.
Trainability
“Its been good for most tasks, but AI still isn't good at some tasks and with heavy usage the AI agent tends to hallucinate.”-P16
Explainability
“The information provided by AI is not always accurate and requires me to go back and review it.”-P23
Deterministic UX Methods
“AI has helped with my workflow by improving and speeding out my research and design flow.”-P8
Improvements
“AI enhances efficiency pushing people's boundaries, but AI being trained with copyrighted content makes it unfair for creators.”-P4
ai seminar
5 Validating the findings
Pre and Post Seminar Survey
We received a good response on our post and pre seminar survey. The surveys were designed to understand the change in workload pre and post the seminar, to understand the effectiveness of the qualitative values (transparency, autonomy, and ethics).
Pre-seminar survey
32
Post seminar survey
12
Conducting the Seminar
I hosted the seminar along with my team to capture the participants’ reaction, and understand how the qualitative values affects the participants trust in AI.
Rule based Simulators: Wekinator
The last phase of the seminar was used to explore the parallels between rule based simulators and AI prototyping, as it highlights the increasing complexity/unpredictability of models post training.
nasa-tlx
6 Calculating Task load
Understanding Task load
Task load focuses on parameters like cognitive engagement, latency, and physical load which directly links to the user engagement with the concerned application.
Increased Engagement
Mean workload changed from 5.4 to 6.4, indicating that that the average user was more engaged when using AI due to the exploration of qualitative factors like transparency, autonomy and AI ethics.
5.4
Pre-study
6.4
Post study
‘The’ Answer does not exist
7 Results
Higher Workload
Higher Workload = Higher Engagement
Users who reported higher workload also reported higher engagement (Correlation Matrix, r = 0.944). This suggests a higher engagement effect or that deeper cognitive engagement.
High Use
β = -0.69
Frequent AI users tended to trust AI less. This implies that repeated exposure may increase critical awareness or highlight system limitations.
Design vs Development
p<0.001
Academic program significantly affected perceived workload. Software developers reported the higher NASA-TLX scores, followed by designers.
Ease of Understanding
SD=3.22
Ease of Understanding showed the highest variability (SD = 3.22). This suggests that while some users find AI interactions intuitive, others may experience significant confusion, a priority area for design improvement.
Trust in AI
Trust in AI had a relatively moderate average (4.75) with broad spread (SD = 1.93), indicating mixed perceptions even within a technically literate sample, trust-building mechanisms should not be assumed and may need contextual adaptation.
Synthesis
8 Conclusion
Emphasis on Contextual Values
In addition to rule-based simulation, contextual values provide richness to human experiences in the design of human-AI interactions, as evident through thematic diagramming. The excerpt frequency is provided below.
agency
51
TRANSPARENCY
18
task load
41
accountability
37
Emphasis on Qualitative Research
The qualitative factors like accountability or transparency are unique to user demographic and characteristics of the task, therefore, qualitative research is imperative, and provides holistic and rich insights in-tandem with quantitative research.
Quantitative
9 Future Work
Measuring Trustworthiness
After identifying the change in task load post seminar. It is vital to conduct further study on how qualitative factors like agency and transparency can affect trustworthiness through task load.
Task Load and Trust
Furthermore, as it was identified that a higher task load is associated with an increased trust, but an extensive study is required to validate these exploratory findings.
learnings
Epilogue
from data to PATTERNS
I learned how meaningful patterns can be drawn from a large pool of data through qualitative and quantitative methods.
insight synthesis
Patterns helped me identify emerging themes and led to the synthesis of insights, highlighting the role of trust in AI use.
Conducting seminar
Conducting the seminar was an overwhelming experience but it taught me how convey esoteric ideas to a diverse audience.
