Principle 3: Develop student’s GenAI literacy and skills

This principle means integrating GenAI within teaching, learning and assessment across a programme of study and within specific modules. There are several necessary steps to achieve Gen AI literacy and skills:

Understanding the fundamental structures of GenAI and their practical implications.
Increasing student’s critical awareness of the GenAI application you plan to use and the limitations of the technology more generally.
Developing necessary skills for students to make the most effective and ethical use of GenAI applications.

Before these can be considered it is important to take a strategic approach to addressing Gen AI Literacy across a programme of study. This is not just an agenda for individual staff – it is an important development for all programme and course teams. Students can rightly expect that GenAI will be used consistently and fairly by all their teaching staff. We can expect legitimate complaints if staff are inconsistent in their approaches – e.g. if different staff apply different restrictions to assessment tasks which make similar demands on student skills and workload. This is conversant with ‘consistency’ and ‘coherence’ elements of the Brookes’ Digitally Enabled Programmes’ guidance.

Course teams might like to develop a ‘GenAI map’ which demonstrates where and how GenAI is incorporated into the course/programme. This map could be shared with students as an opportunity for consultation and possible co-creation. As well as informing ongoing practice and dialogue between staff and students, such maps could be invaluable assets during course revalidation.

Understanding GenAI’s fundamental structures and their practical implications

It is important for students to understand what GenAI is and how it works.

A growing number of texts and online resources introduce the technology without delving too deeply into the technical complexities. A good place to start is Wolfram’s (2023) ‘What is ChatGPT doing and why does it work?’

The following practices will grow students’ understanding of GenAI whilst building disciplinary knowledge and academic competency.

Pedagogic Practice 4: Test the academic limits of GenAI to further disciplinary knowledge

Contrasting a response from GenAI with a response from another source is just one of the ways we can use GenAI with students to evaluate the accuracy and usefulness of particular GenAI applications whilst exercising their disciplinary knowledge. With carefully chosen examples, this can be a powerful method to illustrate some of the limitations of large language models (LLMs) such as inherent bias and distortions.

There are some tools such as h2oGPT that allows the user to enter a prompt as they would in ChatGPT. It then displays responses to that prompt from several different Large Language Models. There often is a fair degree of agreement in the responses and the agreement and differences can be a great topic for discussion.

Pedagogic Practice 5: Review the implications of GenAI development for future employment and career development

The improved quality of text and images produced by GenAI has obvious implications for graduate-level work and further study. For example, do we need specialist graphic design expertise to produce commercial graphics if we have tools like DALL-E 3?

There are different interpretations of the likely long-term impact of GenAI and how society should respond (CLA, 2023; OECD, 2023) and this offers opportunities for student projects and assignments which review these developments, both in terms of the general impact and specific repercussions on individual career choices.

Pedagogic Practice 6: Identify and analyse the legal implications of GenAI in the context of your discipline

If the LLM has assembled its dataset of text from the internet then this will likely be a collection of texts written mainly in English and representing a limited set of perspectives.

If the dataset includes text which has been ‘scraped’ from the internet, then it is also likely to include materials which are covered by copyright law. In 2023, we saw a string of lawsuits targeting GenAI producers for compensation for their use of copyrighted materials. One ongoing example which is likely to have significant implications for the future of LLM datasets is the first action from a major media organisation – the New York Times – who have filed an action against the OpenAI company, the developers of ChatGPT and DALL-E.

Analysis and discussion of the arguments underpinning these actions (and their outcomes) will be especially significant for a wide range of students, including those expecting to work in areas which rely on income from copyrighted work.

Pedagogic Practice 7: Critique GenAI stereotyping through the lens of your discipline

Stereotyping is a particular distortion which can affect all GenAIs. It can be especially evident in GenAI-produced images, another reflection of the possible limitations of the dataset used in training.

As an example of a useful classroom discussion, we can invite students to generate particular images using GenAI which might be susceptible to common stereotypes and discuss their implications. Students can then experiment with the prompt to see how they can change the image to include less stereotypical, more diverse content. For example, Birmingham City University has produced clear GenAI Diversity guidelines to support such activity.

Increase critical awareness of the GenAI application you plan to use and the limitations of the technology more generally

We cannot (and do not) expect every member of staff or student to know the detailed technological underpinnings of GenAI but we do need to be familiar with the fundamental notions of how the technology works. This allows staff and students to have an informed debate on its uses and limitations, including ones we have already identified such as dataset limitations, establishing the validity of the responses (Bendor-Samuel, 2023), and serious concerns about the impact of GenAI upon the environment. These discussions can be used to further disciplinary knowledge and understanding at the same time as digital and AI literacy.

Errors and hallucinations

Critical awareness should include an understanding that the technology generates incorrect or fabricated responses (often described as ‘hallucinations’) and other limitations, e.g. in calculations and in logical reasoning.

Pedagogic Practice 8: Test for errors and limitations and explore why these are a challenge to disciplinary or academic rigour

Ask students (individually and/or in groups) to test the software to discover errors or limitations. You can also ask the software directly as the following example illustrates:

Prompt to GPT-4 (28/12/23):

In no more than 50 words and using bullet points, please explain in non-technical language the main types of error and misunderstanding we can find in GPT-4 responses

GPT-4’s response

Factual Errors: Incorrect or outdated information.

Context Misinterpretation: Misunderstanding the question or context.

Language Limitations: Struggles with complex language or slang.

Overgeneralization: Making broad assumptions based on limited data.

Repetition: Unnecessarily repeating information.

Lack of Real-Time Updates: Not aware of events after April 2023.

This example can Illustrate another important property of LLMs like ChatGPT, where you can initiate a dialogue. For example, we followed up this last answer as follows:

Prompt to GPT-4 (28/12/23):

In less than 50 words, which sorts of context do you find most difficult to deal with?

GPT-4’s response

As a language model, I find ambiguity, complex technical topics, localized slang, subtext, lack of real-time updates, absence of personal experiences, creative or abstract concepts, and cultural sensitivities particularly challenging to interpret and respond to accurately.

We could have continued this dialogue by asking GPT-4 for further clarifications, e.g. what does it mean by ‘subtext’?

Pedagogic Practice 9: Identify hallucinations and discuss the academic, ethical and legal issues they raise

You can also ask students to find examples of different types of errors in both academic and popular publications (print and online), starting from examples which have already been identified.

One common criticism of ChatGPT is its tendency to ‘invent’ references. The software is set up to ‘predict the next words’. If you ask ChatGPT to produce references to support the text it has produced then it will provide ‘suggestions’ for references which will appear plausible - they will clearly relate to areas of the text and will be formatted correctly. But they probably do not exist.

Asking students to generate a short text and then asking ChatGPT to provide references for it is a quick way to demonstrate this issue to students, especially if they use a text with subject-specific content.

Pedagogic Practice 10: Introduce GenAI as a ‘study buddy’

Introducing students to the ways we can have a dialogue with software like ChatGPT is a useful way of demonstrating its capabilities and also introducing ways that this software can become a ‘study buddy’, helping students with both study and revision. This should be done with the Traffic Light system in mind.

There are now examples online of students demonstrating how they have used GenAI to support their studies, e.g. Bri Does AI, 2023. These may or may not give a realistic view of GenAI’s capabilities. Asking your students to critically comment on these and find examples from their own subject discipline is a useful classroom or online discussion task.

Develop necessary skills for students to make the most effective and ethical use of GenAI applications

The World Economic Foundation has long anticipated the need for a workforce able to effectively use AI. More recently QAA mooted the use of GenAI as a crucial graduate attribute (2023), and UNESCO has made a compelling plea for our graduates to be able to use GenAI ethically, sustainably and for the common good.

Pedagogic Practice 11: Develop and refine skills in prompt engineering whilst exploring disciplinary knowledge

The quality of output you get from GenAI is dependent upon the quality of the prompt. If you provide more detail about the context and aims in your prompts then you will receive fuller and better answers. There are also suggestions that the ‘tone of voice’ you use when constructing prompts can have an impact on the output.

There are now many guides offering advice and examples of prompts both in print format (Hart-Davis, 2023) and online.

The following list outlines potential requirements which you can prompt the LLM to take account of in constructing its response (adapted from Beckingham and Hartley, 2023). For example, do you wish the LLM to respond as if it was a particular expert or practitioner (role), and should it respond in a way that could be understood by a particular age group or social group (audience)? The more specific you can be with these requirements, the more likely you are to receive a suitable response.

Students can experiment with this model to refine prompts and see what difference any changes make, e.g. start with a very general prompt and then introduce elements of the context to see how that affects the output.

A ‘prompt workshop’ at or near the beginning of a course or module is a useful way of ensuring that all the students can become reasonably fluent in ‘good’ prompt design in a particular subject area, and learn from each other.

Some students will be familiar with other models – again it is worth sharing experiences and expertise. For example, a number of schools are using the PREPARE model (Attwell, 2023).

Many of these models cover much the same ground with slight differences in terminology. Students should be encouraged to find the model which they find easiest to use and remember.

Another useful suggestion is to engage in a dialogue with the GenAI in order to refine the response (as in the example given in Pedagogic Practice 8).

Pedagogic Practice 12: Use a consistent and systematic approach to GenAI in classroom activities

A growing number of examples demonstrate that GenAI can be used effectively throughout a module to support a particular educational rationale:

Hazel Farrell describes her “AI Integrated Model” based on a ‘flipped’ approach where classroom sessions are used to apply the knowledge, concepts and techniques which students have acquired through personal study supported by GenAI acting as a ‘personal tutor’.
A number of university tutors now use the PAIR approach where students work with GenAI in four steps. This framework was proposed by Professor Oguz Acar in a paper where he discusses using it with his marketing students. The 4 steps are:
1. Defining the problem or challenge that they want to resolve.
2. AI tool selection.
3. Interaction – students use the tools to help them with the task in hand.
4. Reflection – students reflect upon and evaluate their experience.

Pedagogic Practice 13: Support students to develop ethical and effective research approaches and practices

The proliferation of AI functionality now available to us means that students (and staff) could potentially As illustrated by the list below (adapted from Beckingham and Hartley, 2023), students could use GenAI applications at every stage in the research process.

AI applications can support with:

In short, each stage of the process is supported by at least one major GenAI app. For example, students need to know about tools like elicit.com which can search and summarise research papers. Your students may well be aware of other similar apps.

There is also the important issue of data security – confidential and/or personal data should not be entered into any app which is not fully secure as defined in University guidelines.

Remember, if you use a free application the data you push through the application is the ‘fee’. Your data will not be secure.

You can use a list like the one above to discuss with students alternative and ethical approaches to their research and/or project development approach. This would be another way of extending their awareness of the Traffic Lights used at Brookes.

Pedagogic Practice 14: Support students to develop more effective project plans and collaboration

We should expect all our students to become reasonably fluent in the use of collaboration and presentation technologies. These technologies are increasingly embedding GenAI in their feature set. Examples which tutors could incorporate in their practice include:

new facilities in presentation technology such as the presentation coach in Microsoft PowerPoint
meeting support for both live and online collaboration. For example, the latest versions of Google Meet includes such features as automatic transcription, the generation of minutes, and the ability to summarise what is going on if you join a meeting late (these kinds of features are also being introduced in many other major software applications, e.g. Copilot in Microsoft 365).

Student groups could be invited to present a plan for their group project which includes their suggestions for the effective use of GenAI-collaboration technologies. Group suggestions could be discussed and compared.

Example of a new skill: improving prompts

What is the ‘anatomy’ of a good prompt?

Prompts should include:
1. Request
plus your specific requirements re
2. Context

Request can be:

1. Task (e.g. summarise X or evaluate X or elaborate on X)
2. Question (e.g. what are the limitations of chatbots like ChatGPT?)

Context should cover:
1. Chatbot role/perspective (e.g. Expert? Practitioner?)
2. Audience (e.g. Age?, Location?, Social context?, Experience/expertise?)
3. Social/historical context (e.g. Date?/Timeframe?)
4. Quality level (e.g. Sources? Reference?)
5. Output format (e.g. Tone?, Word length?, Structure and style?)

General issues and concerns about Artificial General Intelligence

There are also debates which students should be familiar with about the long-term future of GenAI and the search for AGI (Artificial General Intelligence) – machines which can act autonomously and learn for themselves. This idea has been popular in science fiction and fantasy for decades (if not centuries), usually associated with robots and often linked to a dystopian science fiction. But organisations such as OpenAI (who produced ChatGPT) are now suggesting that this will become a reality in the fairly near future (as in recent extracts from interviews with Sam Altman, CEO of OpenAI).

Pedagogic Practice 15: Analyse and evaluate what is meant by AGI and identify implications for current and future professional practice

Given that there are very diverse views on the nature and long-term impact of what might emerge as AGI then a useful classroom or online exercise would be to organise a debate or other format which requires students to research and outline the different positions and evaluate them in terms of the potential impact on their professional futures and that of their discipline.

Pedagogic Practice 16: Identify and evaluate issues concerning the ownership and control of GenAI

Students on courses/programmes which examine economic, social and political trends associated with technology will need to develop their understanding of specific factors which influence AI developments.

A very recent example of events with economic, social and political implications and long-term impact on GenAI development is the ‘battle’ for control of the OpenAI company, the company behind ChatGPT. OpenAI managed to fire and then reinstate its CEO within a few days, leading to a fundamental restructuring of the board which has important implications for the future direction of the company and its products.

This example offers a case study which is appropriate for all students who are interested in the interplay between technology and its context of production. It raises fundamental questions of organisational structure, ownership and legislative control. It also raises interesting questions for communication and media students thanks to the important role played by social media as the conflict unfolded.

Students can be invited to explore this case in various ways, using both independent analyses and considering the opinions and messages from the main protagonists.

Students can be invited to compare the different organisational and economic structures which have emerged in the GenAI industry (e.g. OpenAI cf. Elon Musk’s recent initiative) and comment on their implications for the technology which is emerging, e.g. for access and accessibility.

Pedagogic Practice 17: Identify and evaluate how changes in legislative structures affect future developments in GenAI

New legislation and regulations appeared in the latter part of 2023 with profound implications for the ways that social media and GenAI enterprises can operate in future, e.g. the UK Online Safety Bill and the EU Digital Services Act.

Students can be invited to explore and evaluate the different patterns of legislative control which have emerged, e.g. UK cf. USA cf. Europe cf. China. Student groups could take one of the contexts and present their justification for the approach which has been taken. This would be an ideal opportunity to create interdisciplinary groups with say students from computing/AI, economics and organisational studies working together. It also offers opportunities for international students to share their contextual knowledge.