Chat GPT- UX Case Study

Generation AI, or Gen AI, represents the generation growing up in a world where artificial intelligence is deeply integrated into daily life, shaping education, work, and social interactions, influencing how they learn, create, and adapt in an AI-driven society. Understanding their relationship with and adaptation to AI is crucial for shaping future technology, ethics, and societal norms.

PHASE 1 | Understanding the Landscape

ChatGPT is a Large Language model developed by OpenAI, based, based on the GPT 3.5 architecture. It can generate human-like text based on the input prompts, answer questions and perform a variety of natural language processing tasks. It was trained on a massive dataset of internet, text to learn patterns and relationships in language

Contextual Understanding

Language genaration Capabilities

Task Adaptability

Multilingual Proficiency

Scalability

Zero Shot & Few Shot Learning

Fine-Tuning Potential

ChatGPT

57M

Active Users in the first month of release

570GB

Close to 5.7 Billion pages of text

text

95%

Prone to erros-

repetition, Lack of orginality, Factuality

accuracy

th percentile

GPT-4 has capabilities in Top competitve exams

How does it operate?

This humongous dataset was used to form a deep learning neural network—a complex, many-layered, weighted algorithm modeled after the human brain—which allowed ChatGPT to learn patterns and relationships in the text data and tap into the ability to create human-like responses by predicting what text should come next in any given sentence.

Competitor comparison

76%

AI models should be required to be trained on datasets that have been fact-checked

Using ChatGPT generated content is not plagiarism

20%

14%

11%

19%

11%

10%

Searching for Information

Novelty

Academic Purposes

Creating Content

Recommendations

Writing code

What do people use ChatGPT for?

53%

can't tell if the content is generated by ChatGPT

71%

would lose trust if they know content has been created by ChatGPT

Source Tidio

Global use-cases and fears about ChatGPT

Why does ChatGPT need to evolve?

There are increasing concerns and conversations over the functioning and processes adhered to by ChatGPT.Deep-diving into why and what the users are feeling was crucial to this.

Phase 1 | Understanding the landscape of Gen AI

Project Pipeline

Phase 2 | Primary Research

Usage patterns and Competitor mapping
Product Intent vs Product Perception
Exploring existing behaviours displayed in Human-machine-AI Interactions

Research tool creation
Key Insights- Auto Netnography, Focus group.
Ux heuristics study
Intendended and vulnerable audience mapping
Hypothesis creation and validation-Stimulation

Behavioural mapping and triggers.
Insight generation
problem statement

Phase 3 | Sense Making

Mapping and UI prototyping

Phase 4 | UX Strategies & Intervention

Questionnaire

/Pre-Task

Objective:Self-Perception Analysis

Personal Self concept (PSQ)

Academic Self-concept Scale (ASCS)

Personal Self Concept

Self fulfillment
Autonomy
Honesty
Emotional self-concept

Self-fulfillment (SF): how each person sees themselves in relation to achieving the objectives they have set themselves in their life, feeling fulfilled, meeting their targets, rising to challenges and their general achievements.

Honesty (HON): how each person sees themselves in the sense of being honest, upright and trustworthy in their behavior. It includes aspects such as being a valuable, honorable and consistent person who tries not to harm others; a man or woman of their word.

Autonomy (AU): how each person sees themselves as an individual equal to, but different from others. This includes aspects such as: the perception of oneself as someone who is independent and different from others; the feeling of not being dominated by others; being able to function without depending on others.

Emotional Self-concept (ESC): how each person sees themselves in the emotional dimension, in relation to the more impulsive and reactive aspects of their personality. This includes the perception of the following components: emotional balance, sensitivity, recognition and control of one’s emotions.

Academic Self-concept

Academic Self-Confidence
Academic Effort

/During-Task

Maintaining the Authenticity of Users' Original Intent: Empowering Interactions with ChatGPT (AU)
Confidence in - ChatGPT's Responses: Trust , your capabilities of articulation.(CON)
Recognition of your current knowledge level about the topic of enquiry (KL)
Humane-ness of the Interaction (HMN)
Sense of Learning and achievement ( LRN)

This part of the questionnaire was filled by the user after 2 prompts given to ChatGPT.

/Post-Task

Sense of Learning and achievement ( LRN)
Perception of ChatGPT
User needs, motivators and drivers

Seinse_Quest

/ Research_Tool_1

Interactive Netnographic Survey Experience

Analysing

Self Efficacy

Primary Research

/ Exploring internal factors of user motivation

Seinse Quest is a tool designed to investigate the internal dependency factors leading to confirmation bias and dependency in ChatGPT

As we have to uncover the internal perceptions of self that an individual held we made a combination of experiences tapping different aspects

/ OBJECTIVE

Create an Experience to gather data and uncover hidden internal factors in play while the user interacts with ChatGPT. The following attributes were analysed:-

Relationship between Personal self-concept and Academic self-concept with dependency on the platform
Awareness about maintaining the originality of your initial thought/intent to visit in the platform- Empowering Interactions with ChatGPT
User needs, perceptions and drivers

Particpants overview:

Number:25

Background:

Students of NID-1st year and final year

/Tools used

AI-Assisted Task completion
Structured Questionnaire- Pre, During and Post Activity
Netnography of the task

Supervised task with netnography

Outcome

Analysing the course of interaction of users with ChatGPT
Originality of the user's thought in the end write-up of the task and awareness regarding it.

Task: Creating a new brand (no limitation on any domain)

Imagine that you are part of a team tasked with creating a new brand that resonates with your consumers. You have access to ChatGPT, an AI-powered assistant that can provide insights and ideas to shape your brand. Your task is to collaboratively use ChatGPT to develop and define this new brand.

Brand Vision and Values: brainstorming and articulation of their brand’s vision and values
Audience Type: brainstorming and articulation of the type of audience you want to cater to
Brand Identity and story: make a quirky tagline and a background story about how you started this
Offerings: products, sku’s, skills, services etc

To view detailed data

analysis. Click here

/Strengths of the Tool

The combination of the tool is such a way that qualitative inputs can be quantified to get clearer insights
Capturing both virtual, self and combination of both behaviours
Interactive task gave awareness about the user about the capabilities of ChatGPT

/Limitations of the Tool

As the attempt is made to capture the inner unconscious beliefs, the whole activity itself might bring the need to answer subjectively
The Data of a completely unaware user was difficult to process and analyse

Simulation

/ Research_Tool_2

Behaviours exhibited in Interaction with ChatGPT were simulated under context and conditions to prove/disprove hypothesis created.

/ HYPOTHESIS

We believe that

Users have a low priority of trust while consuming information which leads to retention of user in platform

priority of trust in user reduces as users receive validation for their outputs

We believe that

Unawareness of capabilities of ChatGPT makes the user retain in the Platform

We believe that

/ACTIVITY

Participants were provided with information cards in an area of their liking in the span of 2 rounds- info in cards in both the rounds generated by ChatGPT (User unaware about the source_- one right & one factually wrong)

Information card provided

Respondents asked to rank these cards in priority with 1- being highest priority and 6- having least priority

ROUND 1

ROUND 2

2nd Information card provided

Respondents were asked to rank these cards in priority with 1- being the highest priority and 6- having the least priority

3rd Information card provided

RespondThis card reveals that one among the both card were factually wrong and a minimum validation scenario is presented in the card. They are asked if they would like to rearrange their priority cards.

/ FINDINGS

All the three hypotheses were proven. Even though the ranking of trust went higher after knowing that information may be false, all the respondents said that they would go back to the source. The Unaware user was fascinated by the concept of ChatGPT and would want to go back again to explore more.

/ Research tool

INSIGHTS

Social comparison: Human vs AI generated output

With the usage of AI in vast use cases, the expectations and demands from users have evolved across different social settings. AI-generated output displayed more 'Signs of Intelligence' in comparison to human output and a sense of lack arose in the user. This comparison led to a decrease in self-efficacy in the user which led to validation-seeking behaviour.

Human-Human Interaction

High/Low Trust, High/Low Confirmity

Seeking Information vs Seeking Approval

As the user's initial Intent is to seek information arising from curiosity/purpose/validation, a cascading effect is triggered which fosters a approval seeking behaviour further leading to dependency.

Sense of lack

Validation

Approval

Dependency

Addiction

Empowering

External and internal insecurities

Low Trust, High Conformity- Human - AI Interaction

Human-Machine

Interaction

High/Low Trust,

No Confirmity

(Machine Heuristics)

Human-AI

Interaction

Low Trust,

High Confirmity

Established behavioural patterns regarding trust and conformity are given below. Even after awareness of the limitations of ChatGPT, the users confirm to it.

Need for Empowered user experience:

Higher Self-concept, Lower confirmation bias

Users with high self-concept value were in more control of the end output generated. They preserved the authenticity of thought and intent with which they had come in the beginning

Autonomy

Perception of Priority of trust

Conformity to the platform

Awareness of capabilities

of platform

User Retention

Dissonance Action taken

High

Medium

Low

Low/Mid

Selective

Full

Low/High

Mid/High

High

Low/High

Targeted User

Intended User

Vulnerable User

Factors affecting User Interaction

An assistant to seek knowledge and produce useful responses for its user
Follow complex instructions and generate answers seeked by the user
Exhibits human-level performance on various professional and academic benchmarks.
Reliable, creative, and able to handle much more nuanced instructions

Frustration due to lack of emotional intelligence by the platform
Privacy and data concerns
Dependency on AI for decision making
Loss of critical thinking
Limited learning for the users
Over reliance on AI answers
Inaccurate expectations from the platform
Ethical dilemmas
Struggle to effectively communicate with the platform

Missalignment

Shift From

Shift to

ChatGPT 's Intent

USERS right now

AI-centric universe

AI-Empowering universe

ChatGPT's Value Funnel shows a misalignment in ChatGPT's Intent w.r.t. to the User need.

Users vs ChatGPT

Problem Statement

How might we re-imagine ChatGPT to eliminate the validation-seeking behaviour of the user and create an empowered experience for the user by preserving the user's intent of visit and originality of thought?

UX Strategies

Preserving authenticity of thought

It is key to enhance that ChatGPT is a 'medium' of knowledge and not a source. This comes in the interaction nudges the user to be in control of the output generated

3.Establishing purpose of Intent

With many Tools and videos in the market to make users write better prompt, its very essential to understand the realm from the pov of user's ambiguity on what to ask/do in ChatGPT to get what they seek.

2.Seek Data to give better data

The existing feedback model of ChatGPT is capable of refining its prompts as per the data set given by the user. However, the user is unaware of what was captured in his/her prompts. Bringing this feedback loop before and after the prompt is key to the user able to understand what went wrong

Primary Prompt Engineering

Conversational Prompt Injection

Organised-History and Main view tab

/1 Preserving Authenticity of Thought

The clutter of previous conversation wont be visible while you are interacting with ChatGPT

History view tab- features

There is a new folder feature added where conversations cane be organised and grouped by the user.

Main view tab- features

Note-taking- This space can be used to copy and paste the content you want. This gives oppurtunity for the user to edit and modify the content from GPT.

Heart it!-Users familiarity with Likes and hearts can be leveraged to access favourites/highlighted notes for ease of access

/2 Establishing purpose of Intent

First question intiated by Platform

With multitudes of context setting available (Research, Content etc.), if wanted user can define the context of the visit. This helps in giving contextual responses and removes ambiguity from users mind

Role setting

Assigning a role to GPT is an existing usage, but bring a 'call-to action' for the usage would benefit user to have a better experience

Improving the' feedback'

The top feedbacks of specific contexts can be given as call-to-action buttons.

This includes- Make test longer,shorter, change tone and translate. This provides a nudge for the user on what they are seeking next.

Download editable file

This gives power in the user's hand on what to do with the content. This also helps in keeping creativity and authenticity alive in user.

Context boundary creation

/3 'Seek' Data to get Data

Display Active Users online

Number of users of online will give credibility tothe user and can act as a touchpoint to gain trust

Click on the header to view and interact with the prototype

UI Interventions

gouri_j@nid.edu

An opportunity? A piece of advice?

How about a jamming session? Reach out to me !

/gouri-jayasurya

All good things end.

Cognitive Heuristics triggered by ChatGPT

Lack of CIALDINI's SIX WEAPONS OF INFLUENCE

RECIPROCITY

SCARCITY

CONSISTENCY

LIKING

CONSENSUS

AUTHORITY

Decision making is effortful, so individuals use a lot of rules of thumb and decision-making shortcuts (heuristics) when deciding what to do, how to behave or what action to take in any situation.

Cialdini detected six principles of influence based on the psychology of persuasion that condition our behavior: Reciprocity, Commitment, Social Proof, Authority, Liking, Scarcity.

The programming of CHATGPT is heavily driven by the requests/tokens we give to it.It fails to take into account the subconscious desires and needs.

Triggering Sleeper Effect

Normal decoy

No Message

Sleeper effect

Immediate

delayed test

Persuasion

Immediate

delayed test

No Message

If you learn after receiving a message that the source has low credibility, at that time you will discount the message, but over time you will tend to increasingly support it. Sources that are similar to the recipient are more persuasive than those that are different.

In the case of Chat-GPT, the complex use of words and vast information architecture, influences the user to support it further.

The presence of the media interface of ChatGPT will cue users to apply the mental shortcut that ChatGPT is more accurate and objective compared with humans, thereby shaping their perceptions and evaluations of the communications attributed to that source.

Triggering Machine Heuristics

Final Output not justifying the Initial thought of user

BEHAVIOUR

AFFECT

TRIGGER

Initial perceptions-casual

and quick responses

Mediocrity

Change in confidence of prompt writing skills after the first prompt

BEHAVIOUR

AFFECT

TRIGGER

Post initial prompt - lack of awareness of effective interaction with GPT

Sense of self is getting depreciated- Autonomy,

Honesty, etc.

Irregularity in Interaction tone and language as the interaction progresses

BEHAVIOUR

AFFECT

TRIGGER

Pre-conceived notions

of self-capabilities

Frustration build up

Low self-confidence

Unable to capture the full essence of the user’s intent

Unable to find out or recall a previous conversation

BEHAVIOUR

AFFECT

TRIGGER

Visible History panel on opening, expectations of user from virtal Interfaces

Frustration build up

BEHAVIOUR

AFFECT

TRIGGER

lack of highlighting the sources of Information, Curated responses

Usage of Information from ChatGPT as the ultimate truth/ Higher power for ChatGPT over your thoughts

Loss of self worth, autonomy, critical thinking

Failure to highlight the source of Information more effectively

Unable to capture easy recall interface elements known by the user

copy pasting the response as it is

BEHAVIOUR

TRIGGER

Urgency, Trust on the platform

AFFECT

Mediocrity,

Lack of creativity

Lacking to highlight GPT as medium of information rather than a source

Failure to capture holistic feedbacks from user

/ Platform Analysis

Behavioural Mapping

Netnography

Netnography has its roots in ethnography. It's a qualitative way of analysing social interactions in the contemporary digital communication context. We analysed previous chat history and live interactions of users with ChatGPT.As ChatGPT is a User-need driven platform- this tool helped uncover preliminary research points to further explore

/ DIP STICK RESEARCH

User Interaction Expectation- 'Creepiness'

/ INSIGHTS

People tend to feel eerily disturbed when they encounter unknown people, situations, and technologies. We define “creepiness” here as a feeling of discomfort & ambiguity that is experienced when a user is not sure how to interact with AI.

Perception of ChatGPT to be an ultimate 'source' of information

Seeking validation from a non-human entity

Believing they are eliminating subjectivity

Seeking belongingness in an intellectually driven society

Clueless about what went wrong in their prompts

Awareness of limitation vs Usage with awareness

Often, users who do possess

awareness about the credibility of the content generated from ChatGPT, seem to continue using ChatGPT due to multiple internal and external factors.External factors being matching upto social stakes by making the content sound more intelligent. Internal factor leading to confirmation bias is further explored in the study

Urge to display signs of Intelligence

Quicker/earlier outputs

Being vocal

Jargon filled/ complex words in sentences

Amount of Information conveyed

Precision, accuracy and flexibility

With the arrival of ChatGPT, a time before its existence seemed to be forgotten. Users often rely on it on a multitude of levels to adhere to match up to the intelligence they believe others showcase. With machine heuristics, adding to this,users start to develop a dependency with the platform

G.JAY

In the dynamic landscape of AI interaction, there exists a notable sense of uncertainty and ambiguity. This evolving relationship between machines and humans is giving rise to many untapped behavioural patterns arising from the our existing mental models and expectations from both our real life and virtual interactions.

This study aims to realign the ways to look at user interaction with ChatGPT through the lens of social behaviour psychology. This design research project focuses to demystify the nuances of Human-AI Interactions and focus on promoting user autonomy and empowering Gen-AI experiences

**Please note the research is done on the GPT-3.5 version and done in Sept,2023. All the strategies suggested are in accordance of the capabilities of ChatGPT existed during the time.

Academic Project with 2 other team mates

Introduction:

Team & Role:

2 weeks

Stimulation

Semi-structured Interview

Timeline:

Focus Group

(25 Partcipants)

Netnography

Research Tool Kit

Key findings

Users perceive ChatGPT as a ‘source of knowledge’ rather than a ‘medium of knowledge’ transfer.

A user's reliance on ChatGPT is linked to their confidence in their abilities and awareness of their own intelligence.

Even after having low trust in the platform, users conform to the platform due to the external validation users get for the outputs generated by AI

A sense of lack arises in the user due to the social comparison of AI-Generated output in our current social structure