Michael Twidale
Graduate School of Library and Information Science
University of Illinois, Urbana-Champaign, Champaign IL 61820, USA
Email: twidale@uiuc.edu
Abstract
Over The Shoulder Learning is the informal, spontaneous workplace help-giving
interaction that is often used by people to learn from their colleagues
how to use part of a computer application. The concept is analysed in the
light of a study of office activity, revealing both the strengths and weaknesses
of the method. One consequence is that the approach requires a different
perspective on interface design; a consideration of the development of
features to support human-human interaction, using the computer screen
as a shared resource to support this. It also requires us to consider all
computer systems as being (at least temporarily) collaborative applications
used to support learning. The calculus of learning is proposed as a means
of accounting for people’s failure to learn how to improve their productivity
with an application. Various design implications for functionalities to
improve the efficiency of Over The Shoulder Learning are proposed, along
with a research agenda.
Keywords
Interface Design, Computer Supported Collaborative Learning, Informal
learning, Workplace Learning, Help-Giving
1. Introduction
This paper aims to explore the nature of a form of workplace learning
of computer applications that has received little attention in terms of
its implications for systems design. The paper attempts to justify the
importance of this form of learning. The nature of the learning is explored
using a small scale study of an office to inspire an analysis of the problem.
This analysis raises various design implications as well as an agenda for
further research.
How do people learn how to use a particular piece of software? Clearly there are several possible ways:
2. Significance of OTSL
It is almost a truism in interface design that many users rarely read
manuals or use online help (e.g. Nielsen 1993). Often a user is not able
to attend a formal course of instruction, either because it costs too much,
or they need to know now and the next training session is not until next
month. Although some people may prefer to learn a system on their own,
many have a preference for learning in a more social context. Indeed there
is evidence from educational theory that this preference is pedagogically
appropriate, particularly when the interaction is rooted in tasks that
are relevant to the needs and interests of the learner (Vygotsky, 1986,
Lave & Wenger, 1990, Brown & Duguid, 1991). Thus it may turn out
that OTSL is not merely one of several possible learning activities, but
that it plays a significant part in the learning of many users. This paper
does not provide evidence of this relative significance, but rather attempts
to show why the issue is worthy of greater attention and to explore ways
in which it may be facilitated.
Although sometimes the help needed can be obtained from a straightforward verbal interaction, which thus may take place anywhere, including as a result of casual social encounters in a hallways, around a coffee machine or over a meal, there are occasions where talking about the problem is not enough. In the case of computer systems use, it can be far more effective to show the source of the problem and its solution, talking around the screen. It is cases such as these that distinguish OTSL from the broader topic of informal learning and help-giving.
3. Related work
Various studies of the behaviour of computer programmers indicate how
they informally learn from their peers (e.g. Alty & Coombs, 1980, Lang
et al, 1981, 1982). An early classic is the work of Weinberg (1971) which
includes the story of vending machines in a common room that became the
focus of much social chat. This was perceived as time-wasting, but their
removal led to a noticeable decline in productivity leading to the discovery
that interspersed within the social chat was significant problem-solving.
Variations on the coffee-machine story are used as anecdotal evidence for
the importance of informal collaboration and awareness of the activities
of others in the software development process.
Informal communication is an important part of effective work activity (e.g. Kraut et al. 1990, Orr, 1996, Whittaker et al. 1994). The importance of informal learning in workplaces (irrespective of any technological focus), although not a novel discovery, is increasingly being acknowledged. Several earlier studies noting the phenomenon as it applies to end user computing include those by Scharer (1983), Cole (1984) and Lee (1986). It is difficult to get a sense of the relative importance of informal learning. A recent study claimed that up to 70% of learning needs are met by this approach (Center for Workforce Development, 1998). Rieman’s diary study of computer-oriented exploratory learning found that in one week ten participants reported a total of 69 learning episodes of which 18 involved help from others (Rieman 1993).
Bannon (1986) raised the issue of the importance of informal user help as an area of significance for interface design. He makes the distinction between formal and informal sources of help, and lays out a number of ways in which help-giving may be usefully analysed. He notes that the "acceptance of the ‘one terminal – one person’ perspective as the focus of study in human-computer interaction may be overly limiting". Indeed Bannon also uses the term "over-the-shoulder" in the description of a case of a new employee commenting on the advantages of sharing an office with a more experienced person because of its advantages for hints on computer use (Bannon 1986 p403).
Mackay (1990) notes the collaborative nature of the customising of software. Although this is help-giving, it is not necessarily OTSL in that the help-giver may just pass on the customisation files to the recipient without the latter learning how to do it herself. Mackay’s study revealed users’ preferred methods to obtain information about customisation were "ask a person" and "copy and experiment" (which involved getting a file from a colleague). She notes the existence of translators, people who actively share files and talk to others. Translators are not necessarily the most technically adept in a group, but are willing to spend time with others to help them. They are likely to create on simplified and more task-specific customisations to share. The chief disadvantage is that their relative lack of technical skills mean that they may propagate errors through the organisation. Other studies of the process of tailoring also note the importance of informal learning (Maclean et al., 1990, and Trigg & Bodker, 1994).
Blomberg (1987) notes how costs (both effort and social such as embarrassment or reluctance to disturb others) are factored into the decision to seek for help and uses this to account for the differences in satisfaction with the same technology between two seemingly similar workplace situations. Clement (1990) studied the use of word processors by secretaries in a context where they were given computers and minimal training. A preliminary survey noted the importance of mutual support in solving problems over written manuals and outside help. A two week problem-incident diary completed by 4 secretaries reported 40 distinct incidents that interrupted work. 25 of these incidents led to conferring with someone else.
Eveland et al. (1994) examined the nature of help networks and the importance of a small number of help providers, who are similar to those they help in their work and position, and who link users to central help resources. As with other studies they found that users ask for help from people nearby and from people with similar work tasks in preference to more remote but much more expert users/help staff, even when these are available. As they note, this emphasises the importance of contextual knowledge of the work problem in resolving an ostensibly technological problem.
Berlin & Jeffries (1992) examined informal consulting interactions between apprentices and expert computer scientists. They note how apprentices try to minimise their use of consultants’ scarce time by various strategies. They also note how "within minutes, conversations jump around multiple domains and levels as questions are triggered by the code itself, the mentor’s explanations and the physical actions of either participant".
George et al. (1995) emphasise the group-level nature of learning that is an essential part of daily work practices. This shows the importance of communities-of-practice (Lave & Wenger, 1990, Brown & Duguid, 1991) for enabling continuing incremental learning. They contrast two studies, one of professionals and the other of clerical workers, showing how it was much more difficult for the latter to engage in informal learning interactions, above the minimum necessary to do the basic work, because of time pressures, lack of management support (or even understanding of what their work really entailed) and lack of control of their work environment. By contrast, the professional group had considerable freedom and used it to effect substantial informal learning.
Sometimes it is not possible to obtain help from nearby colleagues. Numerous studies have shown how electronic networks, including mailing lists, bulletin boards and Usenet news are used to obtain information from others, across departmental, organisation and even national boundaries (Constant, Sproul & Kiesler, 1996). Help asking and giving is usually constrained to brief textual messages, with an evolving netiquette of when and how to ask for help (Agre 1994a).
Nardi & Miller’s (1991) study of the use of spreadsheets notes how the spreadsheet software is at times a CSCW application in that it supports collaborative work and is used collaboratively to do work. Local developers emerge who are work domain experts but who choose to spend more time ‘tinkering’; learning about the technology and interacting with programmers (Nardi 1993). Users share spreadsheet templates, and local developers and programmers contribute code to the spreadsheets of less experienced users. However, informal teaching also occurs, often precipitated by a user observing a feature in a colleague’s spreadsheet. Thus spreadsheets support the sharing of both programming and domain expertise, using the spreadsheet as a medium of communication. Gantt & Nardi’s (1992) study of the use of CAD systems led to the concept of a ‘gardener’ who helps co-workers to use the tool more efficiently. Gardeners are help-givers whose role has been somewhat more formalised as a result of managerial support for that work. They can also act as intermediaries between users and managers, MIS and systems staff. The idea is extended in Nardi & O’Day (1999). The work by Nardi and O’Day (1996) looking at the work of reference librarians provides an insight from a more formalised kind of help-giving. Interestingly, they also identify the importance of context – the more a librarian works with a given client the easier it is to understand the client’s needs and predict the usefulness of a given item.
Trauth & Cole (1992) propose a framework they call the organisational interface as a way of integrating human computer interaction, management information systems and end user computing approaches to user support. The framework is used to show that problems can be solved by approaches that use organisational forms of support to enhance the existing user interface, enabling the consideration of tradeoffs between candidate technological and organisational solutions. The framework explicitly acknowledges informal peer support as a key component of the overall support mix, as well as the importance of managerial encouragement for different support activities. Eales and Welsh (1995) explore the idea of designing for collaborative learnability, emphasising the potential of informal help from colleagues in learning computer based tools, both initially and for subsequent improvement. They develop design criteria for technologies to support this, focussing on the importance of making tool use visible, by capturing storing and sharing activity.
Various studies (including Mackay, 1990, Clement 1990, Eveland et al. 1994) have considered the patterns of communication; who talks to whom. In this paper, the main focus is on what happens after that decision has been made; once the interaction has begun, what is said and done, and by implication, how to make it better.
4. The study
In order to investigate the concept of OTSL in more detail, a small-scale
study of an administrative office was undertaken. The office was open plan,
with 5 secretarial / administrative staff working on a large number of
different tasks and using a range of software including word processors,
email systems, spreadsheets, databases, and a complex bulletin board system
for communicating with the rest of the department. Activity was observed
in fifteen one hour periods at different times of the day and different
days of the week over a period of 10 weeks. Five of the sessions were videotaped
for further analysis. Participants were also interviewed.
The study should be treated more as a pilot than as offering definitive insights. Its purpose was to gather information in order to begin to explore the implications for design for usability. Thus it raises more questions than it answers. The anonymity of the participants has been preserved by the use of assumed names chosen by the participants.
The workplace was relaxed and informal, with strong managerial encouragement for initiative-taking. Although each staff member had clearly identified responsibilities, they covered for each other when necessary. Their work involved substantial interaction with each other, affording opportunities for learning. It was also a very public space, with other departmental members and visitors constantly entering and interacting with the staff members. Consequently, the context is perhaps nearer to the ideal of a setting where mutual learning is likely to take place than an average clerical setting. Thus the examples of learning observed and analysed in no way contradict the findings of George et al. (1995) of the difficulties that clerical work groups often have in making time for learning.
4.1 Methodological implications of the study
The office members were aware of the nature and aims of the study.
They even volunteered to ‘save up’ some learning episodes for when the
researcher would next be present. Thus these were not exactly the purely
spontaneous learning episodes that we would like to have observed. Nevertheless,
they were authentic in that they were learning tasks that the participant
was intending to undertake anyway at some time, and the participants claimed
that they were similar to normal unobserved learning episodes. They did
not appear to be substantially different from the truly spontaneous episodes
observed, except that the latter seemed much shorter (seconds to a couple
of minutes rather than 20-40 minutes). However the study was too small
for conclusive findings. This raises an important issue of the problems
of undertaking an observational study where a particular focus of interest
is a relatively rare, brief and spontaneous event. Unless you have a lot
of time to spend in observation, you may not see much of it.
It is possible to ask about recollections of previous instances of the event, but the descriptions received about the process may not be sufficiently detailed to help the analysis, particularly that needed to inform systems design. (Actually, as the section on capture of context in design implications notes, that is the same problem that applies to novices seeking post hoc help). Post hoc interviewing can however be a powerful method for accumulating aggregate information about remembered episodes, who was involved and their perceived frequency. The studies mentioned in the Related Work section above mostly used that technique (as well as ethnographic observations and the use of diaries) and they illustrate the strength of the approach in giving a rich view of the nature and patterns of interaction.
4.2 Findings from the study
Extracts 1, 2 and 3 are taken from a transcript of a single session.
The session involved Gretchen explaining to Amanda how to use the Eudora
email system. Amanda was familiar with Pine, accessed from her PC via Telnet.
Extract 1
Thus, just because the aim of the study was to collect insights into learning episodes to inform design, we must remember that these may not be regarded by the participants as unique, isolated learning episodes that just happen to be more tightly temporally bundled into everyday work than formal courses. They may be regarded as being part of work, or not regarded at all.
Just as work episodes lead into learning episodes, so a learning episode may be ‘interrupted’ by a work one, either internally, such as a thought triggering a side discussion, or externally by the arrival of someone into the office needing help and attention. It was noticeable that even in the case of scheduled OTSL activities for the benefit of videotaping, despite all the distractions of this kind of rather intrusive observation, the participants were also monitoring the activity of others in the office, ready to offer help as needed, and especially to address the needs of any visitor to the office. Participants would glance around, shout out helpful nuggets of information to colleagues not involved in the OTSL and break this off if someone approached the desk to ask for help.
4.2.2 A work task focus means a cross-application focus
For the participants, the great advantage of receiving explanations
from other members of their team (rather than say from systems engineers
or formal trainers) was that the help was available as it was needed, that
the explainer would understand what it was that the learner was trying
to do in terms of the office’s actual work, and that the explanations and
examples would be understandable. These are all sound pedagogic reasons
that the advocates of constructivism and situated learning would support.
However in addition is the value of explanations that are geared to the
needs of the office rather than the capabilities of any given system. In
order to solve a work problem, the help offered may involve the use of
more than one system. For example when Anne was explaining to Gretchen
aspects of bulletin board use, the conversation moved from discussion of
the interface to the bulletin board, the menu-based telnet interface to
the departmental information system of which the bulletin boards are a
part, accessing a Unix prompt, and using the pico editor, as well as passing
mention of the use of the pine email system that Gretchen was familiar
with. No one piece of software could do what Gretchen wanted, nor in the
case observed was the new knowledge she needed confined to a single system
(it was not the case that she just needed to know how to integrate a new
system into her armoury of familiar technologies, rather she needed to
be told about relatively minor aspects of several pieces of familiar software
that only in coordination would help her to do what she wanted). Similarly,
the interaction between Gretchen and Amanda, although mainly about Eudora
included detours of explanation of Windows 95, WordPerfect and comparisons
with the Pine email system that Amanda was familiar with.
4.2.3 The advantages of shared context
One of the chief advantages of getting help from colleagues is that
there is so much shared context. The helper can choose examples related
to the work that the system will be used for, and pitch the explanation
at the appropriate level based on what is known about the recipients prior
experience and expertise. Thus Anne’s explanation to Gretchen about bulletin
board options was as much about exactly why Gretchen might want to use
the features under discussion as about how to use them. Discussions are
inclined to swoop over many levels of detail, from exactly which keys to
press, to how to choose between options, to examples of office practice
in which the technique has proved useful. The language used reflects the
shared experience. Jargon words from the work context are used frequently
(Gretchen and Amanda talk about putting things in order "by alpha", and
purging files), but computer jargon may not be used, with participants
struggling to use everyday terms to describe what is being done, or even
using computing terminology in non-standard ways.
Furthermore as with any kind of one-to-one tuition, it is much easier to diagnose misconceptions or even for the learner to ask for remedial help. Just because there might be a logical ordering to learning concepts (including important pre-requisites) does not mean that each learner has travelled that path. Thus in the middle of a discussion of how to create and use mailboxes with respect to the placing of a set of themed mail messages, Gretchen realised that Amanda did not fully understand the convention of inserted right angle brackets in text included in a replied message, although she had encountered it many times. Gretchen detoured from here explanation of mailboxes in Eudora to explain this general email convention. This is a concept that Amanda, with some considerable experience of email use, might have been just assumed to have, but didn’t.
4.2.4 The employment of resources at hand in help-giving
The interleaving of work and learning has additional impacts on the
learning process. The work context means that there are often elements
to hand to serve as part of an exercise or practice that are themselves
meaningful and useful. Anyone who has tried to teach abstractions in a
more conventional classroom context is aware of the importance of creating
relevant examples for the students, and the difficulty of doing this. With
OTSL these examples are often to hand, or even suggested by the learner
themselves. For example, the Gretchen-Amanda interaction was interrupted
by a delivery for another staff member. Although dealing with the delivery
person instantly diverted attention away from the learning (see above for
the interleaving of work and learning), after the delivery was accepted,
Amanda realised that she could now practice what she had learned by sending
a message to the person for whom the delivery was intended, while Gretchen
stood by to monitor progress and help out in case things went wrong.
Other people are also resources. As has been noted, staff in the office are continually monitoring each other’s work activity and asking and offering work related information that may lead to an episode of OTSL. In the same way, an episode of OTSL may precipitate a request for help. Thus Gretchen and Amanda called on Suzanne to send, receive and reply to a number of practice email messages that would help Amanda understand certain kinds of usage. The help-giver may themselves need help and call others into the interaction. A question by Gretchen made Anne (when acting as the help-giver) realise that she did not know the precise composition of certain bulletin board groups used in the department, and she called over to Amanda for advice on the point. The examples serve to illustrate that roles of help-giver and help-receiver are flexible, and can switch direction – Suzanne, Anne, Gretchen and Amanda both gave and received help from each other.
4.2.5 Problems observed
The above description may give the impression that either OTSL is operating
perfectly well and that there is no need to consider additional technological
support, or, more likely, that the group chosen for study has certain special
features that means that it can achieve OTSL more effectively than might
usually be expected and so the findings will be unrepresentative. Certainly
the congenial problem-solving atmosphere, the help-giving and problem-solving
work ethos and the very public nature of work in the office all add greatly
to the ease of achieving successful OTSL. Nevertheless, there were problems,
and perhaps these deserve greater scrutiny than the successes, in that
they may be more likely to be widespread in other contexts.
The most visible problem with OTSL is the difficulty for a learner to remember all the information being provided. Gretchen resorted to taking laborious detailed written notes, slowing down the interaction with Anne considerably. The different rates at which one can attend to an overview as opposed to following and practising a complex sequence of actions so that one is confident that one can undertake them unaided is substantial. The problem is compounded by the free-form nature of the help given, ranging as noted between the general and the particular (Berlin & Jeffries 1992). Serving to mitigate the difficulty, the fact that the help is from someone in the same office means that it is less essential to completely understand every part of the concepts described – it is easy to ask for additional help in the future.
Extract 3
Although obvious after the fact, it was surprising to note how much work and learning interleaved. This has been stated this as a positive good, but that is not always the case. Learning episodes may be interrupted with work problems that take higher priority and must be addressed. As noted, this may lead to a learning opportunity or practice task, but it may also serve as just an interruption and distraction in the middle of a complex task. As such the learning activity is just like other computer based work activities in a busy environment where designers need to be aware that they should be designing systems to support people working with constant interruption (Rouncefield et al. 1994) rather than the simplified assumptions of single focus that may influence systems designers and laboratory testing.
The spontaneous nature of the help giving means that errors will occur. Thus in practising sending files, Amanda accidentally sent one in an uninterpretable format. Although this might be a learning opportunity, it can just serve as an additional complication, a further distraction from the core learning. The advantage of OTSL is that when such errors occur, they are much easier to detect and remedy. As well as errors by the learner, the teacher may make pedagogical errors. Sometimes the choice of topics to explain, the ordering and the way things were explained were not those that a computing expert would have chosen. For example, an explanation of Eudora’s features included detailed description of how to include coloured text. Although an amusing feature, this does not seem crucial, and the time perhaps could have been better spent on teaching something more useful, clarifying the understanding of a topic already covered, or even just cutting short the whole interaction and reducing the number of things the learner has to absorb. It is difficult to resist the temptation to consider how the interaction could be improved if either the help giver or receiver were more knowledgeable about computing (or indeed educational theory). However that is to miss the point. It is precisely because the help giver and receiver feel an affinity in that they are not computing experts but rather are experts in the administrative tasks of the office that makes the interaction as a whole work so well despite this flaw. If representative, these extracts are the interactions that people do, and wishing that they all had advanced training in computer science and education is pointless.
Finally, it is necessary to acknowledge the most blatant cost of OTSL – that it takes time and effort away from the work in hand. We need to consider the opportunity cost of OTSL – the other things that could be done instead. These costs fall on the help seeker and also the help giver. Even if one believes that OTSL is a desirable activity and one that should be actively encouraged, it is still important to consider ways in which the costs can be minimised and the benefits maximised. The following analysis and then the design implications are based on this perspective.
5. Analysis
Taking the observations of the study and adding them to those of our
other studies (Crabtree et al. 1997; Twidale et al. 1998) and informal
observations of practice and my own experiences of help giving and receiving,
we can begin to outline a ‘folk’ analysis of OTSL (rather as ‘folk psychology’
and introspection was an imperfect precursor of systematic scientific research).
The aim here is to inform both design and future, more principled studies.
5.1. Whose shoulder? Whose desk?
So far in the paper it has been implied that in OTSL it is the expert
who comes to the learner’s desk and leans over his shoulder. But of course,
the learner may move to the expert’s desk and indeed may be the one doing
the leaning over. It may be that the learner was visiting the expert’s
desk on a related or otherwise context and notes an activity he doesn’t
understand. On being asked how she did that, the expert explains, including
details about the work related context of why and when it is useful. Alternatively
the expert may take control at the novice’s desk. Normally this is regarded
as poor pedagogy, with the danger that it is good if the expert is doing
something for the novice, but bad if she was aiming to teach him how to
do it himself. However there are cases where it can be a useful temporary
expedient:
5.3. One-to-one tutoring
Clearly OTSL is a form of one-to-one tutoring and so can benefit from
analysis of this highly effective but extremely expensive form of pedagogy.
The significant features of OTSL are its extreme situatedness, spontaneity,
informality and relative brevity.
The advantage of all forms of one-to-one tutoring is that explanations can be tailored to the needs of the learner both in terms of what the learner needs to know, what she already knows and prior experience that may help or hinder learning (Bloom 1984). Furthermore, if it becomes clear that the learner needs more detail, or can manage with less, it is easy to adjust accordingly. This saves the time of both learner and teacher, and can be contrasted with the difficulty of achieving equivalent flexibility in help manuals and online texts. The key to such effective interactions is to rapidly determine and continually monitor the learning context. The more that is known about the learner’s prior experience and current understanding, the easier it is to tailor the interaction. Again, this is another contributing factor to explain the advantages of receiving help from people who know you well, even in preference to people who know the topic under discussion better (Lang et al. 1982).
5.4. The expertise of the help giver and recipient
OTSL need not be an interaction between an expert in the topic and
a novice. If both participants are fairly expert (but happening to know
different things), the interaction can be more abstract and so shorter.
An action sequence can be described in more general terms, with the assumption
that the listener can fill in the gaps, reconstructing the precise set
of actions to be undertaken. With a novice of course, such assumptions
can not be made and the process must be much slower and laborious. Equally,
neither participant need be an expert in the topic (although they may still
be experts in their work). The help-giving can still proceed, using the
screen as a substitute for the jargon and abstractions that in this case,
neither participant may possess.
5.5. Language, abstraction, linguistic and literal pointing
An extreme case of expert-expert help giving is where the interaction
is entirely about making the recipient aware of the existence of a functionality,
with no attempt to explain how to achieve it. Clearly such an explanation
need take no more than a few seconds, saving the time of the help-giver
and encouraging the altruistic volunteering of information and the introduction
of the topic into informal social chat (the coffee machine phenomenon).
Existence information places the onus on the recipient to investigate how
to apply the feature, using other techniques such as exploration, manuals,
online help, etc., but they have been given the clue that the feature is
there somewhere. Thus effort expended in the search for the desired feature
is not wasted, unlike the case where the searcher does not even know if
what is desired is possible, and thus at some point needs to decide when
and whether to abandon the search. Perhaps surprisingly, some of the interactions
observed in the office study also fit the model of existence information.
As the recipients were sometimes relative novices in the topic, it is unlikely
that they would be able to find out how to use the feature unaided. Rather
the information seems to be volunteered as an option for future OTSL –
should the occasion arise that the recipient wants to use the feature,
they would call over and ask for an elaboration on how to use it.
Do experts even need to do OTSL? As interactions between experts are possible at higher levels of abstraction than between novices, perhaps it can all be done linguistically, and even away from anyone’s desk, (such as in the hallway) or at a distance, by phone or email. This certainly matches the patterns of help giving that have been observed amongst certain groups of professionals noted in the related work section. This is a question that merits further study. We might predict that experts don’t need OTSL as much for successful interactions, but that when it is a possibility, it still supports more effective interactions.
In many cases the interaction will be between individuals where there is a substantial gap in relative expertise. This poses extra problems in language selection. With two experts, abstraction is possible and extremely effective. With pairs where neither would consider themselves experts, abstraction isn’t even a possibility and they will naturally choose a lower level, more verbose means of communication that although not efficient is mutually intelligible. In cases of a marked disparity in expertise the difficulty for the expert is in choosing the appropriate level of abstraction in order to maximise understanding. It is in such contexts that OTSL can help in keeping conversations grounded in concrete examples. Indeed in cases where the novice does not know the name for an abstraction, either party can instead point to an instance of it on the screen and talk about "that" or "when you have to do things like that". Although less clear, it is also possible to talk about abstracted action sequences by referring to action sequences just done as exemplars. That is not to say that experts do not or should not introduce new vocabulary. However sometimes the number of new things that would need to be introduced are just too overwhelming when what is needed and expected is a brief help-giving interaction so that a given work task can be accomplished.
5.6. The difficulty of talking about computer use
One of the problems noted in the study was the difficulty that can
arise in talking about computer use. This applies for novices with any
kind of computer application. However it seems that there are problems
that arise in particular with graphical user interfaces. These are often
touted as more intuitive than command line interfaces, and certainly it
is easier to recognise commands than to have to recall them. It would seem
reasonable to assume that the interfaces were designed to help a new user
learn how to use the system on their own. There does seem to be a problem
when trying to explain to a colleague in words what to do with the mouse,
the pointer on the screen, the mouse buttons and the keyboard, particularly
when these aspects have to be co-ordinated in a time critical way (consider
the problems that some mouse novices have with the process of double clicking).
Why should this explanation be harder with a graphical user interface that
by most measures is easier to use than with a textual interface? Once articulated
the answer is clear – the medium is different. Talking about a textual
user interface is easy, you just say the command words. However even there
conventions have to be established for the names of certain characters
(~ is pronounced tilde, \ is pronounced backslash, etc.) as well as for
formatting (such as when to insert spaces into a complex command). Likewise,
written text describing what to do with a textual interface is relatively
easier to understand than text trying to describe use of a graphical user
interface. Indeed some of the bulk of current manuals can be ascribed to
the need for screendumps to show what must be clicked on – a means of explanation
that can consume more space than textual descriptions of textual commands
Thus the success of textually-mediated technical help-giving on bulletin boards etc. may in part be ascribed to the nature of the interface being discussed (say conventional Unix) as much as to the relative expertise of the participants in such discussions. One way to test this hypothesis would be to analyse the content of help forums and attempt to measure the usefulness of the help received (and the efficiency of the help request – final response cycle) in the light of the medium under discussion. We would expect high level concepts of any technical topic to work well, but that low level topics of textually based interfaces to be much more useful (and so perhaps much more used) than for graphically based interfaces.
Unfortunately, there are two effects that may get conflated. Text-based interfaces are often used by expert users (e.g. Unix) and so the success of a forum may be due to relatively high expertise, or the textual medium. An interesting way to separate out the two would be to look at text-based MOOs, which often have far greater ranges of expertise amongst their participants, and which still have been noted as involving significant help-giving (O’Day et al. 1998). Even in MOOs, if there is the option, it can be easier to give help if both participants are physically co-present because talking is faster than writing, and sitting together round a shared screen reduces the possibility of confusion about exactly what the other participant is seeing (Bruckman 1998).
5.7. Human indexing
An extension of the existence help considered above is where the help
giver aids in the translation from the work task that the recipient wants
to do to the operations and especially the names of operations available
in the software. One of the noted difficulties with writing usable user
manuals is the problem that people call the same thing by many different
names (Furnas et al 1987), so much so that even choosing the most popular
terminology may still mean that as few as 20% of potential readers would
think of calling the action by that name. Thus finding and pointing to
a page in a manual, or merely telling someone what an operation is called
in a given application so that the online help or user manual can be productively
used, can be a powerful (and conveniently brief) intervention. The help
giver here acts as a context-specific indexer into other help resources.
O’Malley (1986) explores this issue of a colleague acting as an intermediary
to help someone find the answer to their technical problem in other resources
such as manuals and online help.
6. The calculus of learning; the cognitive economy of learning costs
and benefits
It is useful to try to explain why people fail to learn how to use
the software around them optimally. In particular, why do people seem to
improve so slowly, if at all, failing to exploit the more powerful features
of the system? The difficulty of learning more optimal use of computer
systems may be one of the contributing factors to the poor productivity
performance of computing investment in the service sector (Landauer 1995,
King 1996). The following analysis, should it prove convincing, may help
in informing the development of systems and infrastructures to improve
learning. Rather than just blaming computer users’ intransigence or incompetence
for poor levels of improvement in skills, it may help to consider the barriers
that lie in the way of this improvement.
A common complaint is the lack of management support. If there are no resources provided, then workers will be unable to obtain appropriate training. Even if training is available and funded, if the work pace is so intense, workers may not be able to justify (either to their supervisors or themselves) taking time away from their work in order to learn how to do it better. Such short-termism, although unfortunate, is entirely understandable. Cases of workers being given technology to use with little warning, training and support do occur (e.g. Clement, 1990), and the workers may compensate for this unfortunate situation by doing more OTSL. Although this paper has focussed on informal, spontaneous learning interleaved with work tasks, that does not mean that it should be taken as a justification for the abolition of formal training. OTSL is a valuable component of a range of learning options, rather than a panacea. Indeed a management that used this and similar arguments to cut back on training is also likely to be unwilling to permit or support the free, informal and casual atmosphere that OTSL needs to thrive. Unfortunately OTSL is sometimes identified as a problem (or a symptom) to be rectified, rather than being acknowledged as a powerful tool to address a larger problem.
Like Eason’s (1988 p178) analysis of training implications for occasional users of software, but elaborating and extending to all users, the approach employs a form of cost-benefit analysis. Although not necessarily articulated, or even conscious, the lack of learning and improvement may be accounted for by rational decisions based on the perceived costs and benefits of learning about features of an application. This draws from an approach of economic theory, where models are constructed based on rational economic behaviour by groups of customers, even if most people do not, or are not aware that they think along these lines. Thus the following is more in the flavour of a micro-economic model to aid future analysis, than a claim for a faithful psychological model.
6.1. The claim
For the sake of this argument, let us assume a well-motivated worker,
in an environment where there are people able and willing to help, and
the management encourages initiative-taking. Our aim is to account for
why such a person might fail to learn, even under seemingly ideal conditions.
A learner in a work context has multiple claims on her time. Even if she
only has a single work task (an extremely naïve assumption), she must
decide between doing that task and investing time in learning in order
to do that task more efficiently. As with all investments, this involves
a calculation of risk. In this context the risks are the unknowns of how
long it will take to learn enough to be more productive, and what the actual
productivity gain will be. In the light of such uncertainty, various clues
are exploited to inform the calculation, including prior experience of
the results of similar cost-benefit analyses and the recommendations of
others. Unfortunately, even the collection of additional information to
inform the decision adds to the cost of the overall learning investment.
Thus it can be that under certain conditions of estimates of cost-benefit
values (note it is the inferred values, not the actual values that influence
the decision) it is a rational (albeit undesirable) decision not to bother
trying to learn to be more effective. There are two main factors that play
into the calculation:
The issue of initial learning, or switch to a new technology (either paper to computer-based or one program to another) is the most visible aspect of learning decisions and so the one most liable to external coercion. The calculus of learning applies equally to the process of incremental improvement. It is entirely reasonable to learn a minimal subset of a new system in order to be able to do a few commonly occurring aspects of one’s job. Indeed this approach has been shown to be a highly effective learning strategy (Carroll 1990). What this analysis aims to show is why it is that so many users seem to fail to improve by subsequently learning more advanced functions, or improve extremely slowly. The analysis provides reasons that have nothing to do with the intelligence of the user, their ability to learn in general, their overall facility with computers or their motivation to learn (although all of these will be sub-factors in the cost benefit analysis). One could be an autonomous college professor with substantial control of one’s work time and tasks, highly intelligent, curious, adept and confident with computers, and still use a word processor sub-optimally for hours every day.
The above framework now allows us to organise other factors into the calculus by how they can tip the calculation.
6.2. Degree of freedom of choice
Where the decision is about whether or not to adopt a new technology,
there may be external factors that either tip the cost-benefit calculation,
or even render it moot. For example, one may simply be required to adopt
the new technology by one’s employers. This can be a matter of power and
autonomy, as noted by others (George et al. 1995), so that clerks may have
far less control over choice of software to use than, say, college professors.
Nevertheless, as noted, there remains the danger of workers just learning
enough to get by and failing to improve. This is particularly likely where
workers are actively discouraged from taking time out to reflect on their
practice and discuss and help each other. Similarly a more enlightened
management can not only remove barriers to learning, but actually tip the
cost-benefit calculation, by for example, including incentives for improved
productivity (increasing benefits) or creating an expectation of time-outs
for learning (reducing costs).
Other means of coercion are more subtle. If one’s work is closely linked with others, the costs of not moving to the new technology may increase because of the incompatibility of formats or the difficulty of conversion. Hence a professor using a Macintosh computer who discovers that she now mostly shares files with PC users, may switch operating systems purely to avoid the complexity of endless conversions. Similarly, having many people around who also use the technology may lower the learning costs (as argued throughout this paper).
6.3. Prior experience
Memories of positive learning experiences are likely to adjust the
perception of the probabilistically-weighted costs and benefits. Likewise
a bad experience of an afternoon wasted floundering around and achieving
nothing will tip the scales against future learning decisions.
6.4. Affect, stress and time pressures
These are ambiguous because they often serve to change several variables
simultaneously. For example, assuming a rapidly approaching deadline to
write a paper and a minimal experience with a tool to correctly format
references, does it make sense to invest time in learning how to do it
better, or the proper way, as opposed to a familiar, but highly labour
intensive way? The benefits now focus on immediate usefulness (up to the
deadline). However, because of the overall stressfulness of the situation,
and presuming that the time saved (if any) could be very usefully spend
on other important activities relating to the paper, both cost and benefits
may increase. It then becomes necessary to factor in the uncertainty of
both measures. If only one knew that it would take an hour to learn, and
then save two hours of work, it is still worth doing. But if one isn’t
sure, is it worth the risk?
6.5. Meta-learning skills
If one has experience of learning new features of computer systems,
not only is one’s confidence boosted, lowering expectations of costs, but
one has accumulated a set of skills of how to go about learning using computers,
including how to recover from complex and confusing error states. These
reduce the actual learning costs as well as perceptions.
6.6. Playfulness and permission
Much talk of the development of human-computer interaction as a discipline
notes the important transition between design of early computer applications
which were intended for use by computer scientists, often in an academic
environment, and design for use by people without formal training in computer
science. It is claimed that the former applications did not require sophisticated
user interfaces because the users would be prepared to invest time an effort
in learning them because they found computers intrinsically interesting.
Furthermore, the designers were designing for a relatively homogeneous
group of users, most importantly, users like themselves. The difficulty
of designing easy to use systems for non computer scientists has led some
researchers to try and compare the experience of learning computer games
(Neal 1990) (where the learning of the interface seems very successful)
with learning conventional applications (where the learning of the interface
often seems problematic).
In the light of the above analysis, these issues link up. A contributory factor in the ease of learning of applications for computer scientists in an academic setting may be the playful approach to learning. That is, new applications may be treated as a game, with the objective of uncovering what it does and what it can be made to do. This certainly fits the stereotype of computer science graduate students fooling round with software just for the fun of it. If learning is a game, the calculus of learning may be rendered moot, or apply but with the relative costs and benefits radically altered. If one is exploring for fun, the costs reduce substantially, and the benefits become greater by including additional items (not just what this particular piece of software will do for the user, an almost irrelevant issue in a game, but also a sense of mastery of having figured out how to use the system). Indeed in a game, often the aim is not just competence, but total mastery, having explored every aspect (or level) of the system.
It can seem a long way from the curiosity and playfulness of such exploration to the high-pressure world of most work. For many workers, such playing with a system in order to build confidence and explore is just not an option. It is not considered part of real work and not permitted. Even those with greater autonomy may not consider it appropriate behaviour. Even when playful learning is individual, it often has a social side, as participants share clues, hints and tips with each other and delight in discovering obscure details to share with their friends.
Such playfulness has been observed even in commercial settings (Nardi 1993, p105) with users tinkering with spreadsheets to learn more about the system. It should be noted that these users were professionals with the autonomy that enabled them to be able to explore and learn in a playful manner. On the other hand, this spontaneity may not be allowed. Bruckman (1997) in her study of the use of MOOs in education notes: "when a student at Massachusetts Public stood for a few minutes watching over one of her classmate’s shoulders, she was admonished not to waste her computer time, and ushered over to an empty workstation."
6.7. Calculus of learning summary
In summary, the above analysis aims to show why learning may fail to
occur, even when the classic barriers identified in other studies have
been removed. Clearly these major barriers of managerial support and access
to many different kinds of help, including people, must also be removed,
but it may not be sufficient. To reiterate, what is being proposed in the
calculus of learning is a useful fiction, rather than a psychological model.
If some people do reason this way, it is likely to be in an extremely qualitative
way, with a weighing up of pros and cons, rather than anything resembling
a numerical calculation. Its chief purpose is to help us consider how we
can develop mechanisms to adjust such calculations in the desired direction
by changing certain relative costs.
7. Implications for design
Having undertaken this small-scale study, we can use its findings and
the resulting analysis to consider the design of systems, functionalities
and interfaces that could improve the process of OTSL. Even if we are unable
to specify exactly what should be built, we can at least try to set an
agenda for the construction of systems that have the potential for exploring
the issues outlined through their evaluation. This will involve the novel
combination of existing technologies, as well as the development of new
functionalities and interfaces. The following design implications are based
on the assumption that OTSL is indeed worth supporting, but that it is
costly and that technologies should be developed to minimise those costs.
The implications are speculative; design intuitions informed by the preceding
analysis. The proof of their usefulness and usability will come from their
subsequent iterative development, evaluation and refinement.
As well as support during the help-giving interaction, it is important to consider designs that affect activity before and after it. Support before the interaction includes helping the help seeker to decide who to contact. In some circumstances (not the case in the study reported), this is a major problem. The help seeker may not know the right person to ask, or who they are allowed to ask. Ackerman’s work on Answer garden (Ackerman & Malone 1990) directly addresses this crucial problem. In this analysis, we are assuming that the user does know who to ask, in which case, other kinds of support before the interaction include features to adjust the costs and benefits in the calculus of learning so that the help-seeker is more inclined to ask for help, and capturing the context of the problem that necessitated the request, so simplifying the diagnosis part of OTSL. Support after the interaction includes helping the recipient to remember what has been learned so that she can still do the desired actions in the absence of the help-giver.
One key motivation in any design must be to reduce the cost of OTSL. Even its strongest supporters must acknowledge that it is very labour intensive, taking up the time of both the help-giver and the help receiver. A technology which can reduce the length of that interaction from say half an hour to five minutes would clearly be beneficial. Likewise, one can be in favour of OTSL in general and still propose technologies to support individual learning, using OTSL as the fall-back for the more difficult cases, rather like the concept of escalation in software support help-lines. In the absence of such design changes, OTSL is in danger of being regarded solely as an expensive invisible overhead of computer use (Nolan et al. 1992).
7.1. Implications from the calculus of learning
The options are to try to lower both perceived and actual costs of
learning, to try to raise both perceived and actual benefits, and to lower
the search costs for the user of obtaining that information. Lowering the
learning costs is a key aim of conventional interface design, help and
manuals, training courses etc. OTSL is an additional approach, but with
the same aim. Increasing the benefits from learning more advanced functionalities
is the aim of conventional systems development in providing these functionalities
in the first place. It will not be considered further here. Both processes
are also strongly affected by managerial support. Making information about
the costs and benefits more accessible is rarely considered and is outlined
below.
For any user, the sort of questions that would help inform a decision to learn include:
It will often be the case that those whose work and experience most closely match those of the learner can most effectively provide that information. Thus, the questions can be phrased not just in terms of abstract functionalities but in terms of particular tasks relating to the work context. Where the person giving the information has herself both done the task, and learned and put the new functionality into practice, her assessment of relative costs and benefits will be far more convincing than a generic estimate intended for all users of the software. However, it may help the process to include in user guides and online help, various estimates of time costs and effort savings. Van der Meij & Carroll (1998) have advocated that chapters in minimal manuals should be of a length so that they can be completed in about 20 minutes. Nardi and O’Day’s gardeners (1999) as well as giving the help, can also play a role in providing tailored cost-benefit information.
Clearly there are many possibilities ranging from generic estimates for standard learning needs, to tailored estimates based on information about a certain type of user or a particular individual. The tailoring activities themselves may be undertaken by individuals (we could look for evidence of the offering of such cost information in current help-giving interactions), or at least in part by systems themselves.
7.2. Capture and Visualisation of Context
When someone asks for help, it is very useful to know about their background;
how much they know about computes in general and this application in particular.
It also helps to know what they have done so far and what they wanted to
do. When receiving help from a close colleague, much contextual information
is available for free. However even here it is useful to have a record
of what has been done before help was asked for. Asking the user what they
did is problematic; they may not remember (or even have noticed) any errors
they made – people naturally autocorrect, they may find it difficult to
remember (often the reason that they asked for help is because they are
confused), they may lack a vocabulary for describing what they did at any
manageable level of abstraction, consequently, they may attempt to reconstruct
what to them was a meaningless arbitrary sequence of micro-steps which
is prone to substantial error, they may have progressed through many increasingly
complex sequences as they make a mistake, try to correct it, make a mistake
in the correction, etc.
It would be useful to be able to record what has been tried and be able to provide this information in an easy to understand visualisation. A mere transcript is not sufficient because they can become very large, particularly in the case of a doggedly determined user who then will claim in response to pretty much any suggestion: "But I already tried that and it didn't work". Not only is that not useful in the help-giving interaction but it is demoralising for the student who may start to doubt her own competence, or that she just does not have the magic ability to get the computer to work. Being able to show such a student that she very nearly did the right thing (from the context report) but just missed one step that led to her being unable to figure it all out on her own is a much more motivating and likely effective educational interaction.
The design implications for this kind of visualisation are intriguing. We need a visualisation of the process of using a computer system. That often means a visualisation of the process of using a graphical user interface. One could just record the whole interaction and play it back like a movie. Indeed systems such as Lotus ScreenCam (Lotus Development Corporation 1999) already provide such a functionality. However, as noted above, that is likely to be overly cumbersome. It is a start, but we should be looking for options for abstraction so that an expert can quickly skim the interaction to get a broad impression of activity, before deciding which bits she needs to examine in detail in order to offer the most efficient help.
The Unix history command is a good example from the world of command line interfaces. It lists the commands a user has recently issued. It is relatively concise, partly because of the terseness of Unix as a command line system, and partly because history lists what the user did, but may require considerable mental processing to determine the consequences of those actions. Thus history is an example of a text-based representation of the process of text based interaction. It is relatively straightforward and as a result is a simple but intriguingly powerful help-giving tool. We have been exploring the development of graphical representations of the process of interaction with text-based interfaces, specifically telnet based connections to library catalogues and bibliographic databases (Twidale & Nichols 1998). The graphical representation allows for interesting ways of visualising activity over time, both actions and their results, and providing the abstraction features that history lacks.
The next stage is to consider graphical representations of the process of using graphical interfaces. Even after building a system such as this that proves effective, it is still necessary to consider issues of privacy and ownership. The system is recording a user’s activity. To be really effective, it should be on all the time, constantly recording all activity and destroying the record after a suitable interval, since the user may not be able to predict that a problem will occur and it would have helped the future help-giver to know what had occurred up to the point that the problem was identified. Clearly such a system and its use will need to be both trusted and shown to be useful before users will be willing to sacrifice their privacy (even if it only records the details onto their own hard drive).
As an interface to support human-human interaction, the desired context capture tool can involve asymmetric use. It does not matter if the help receiver does not understand it all, so long as it is usable by the help-giver in giving information about what the user has done. However, if it is usable to a limited extent by the recipient, the helper can show her what she did, and contrast it with the correct steps. In such a context, the helper would ‘drive’ the application, needing to know how to operate it effectively, while it is only necessary that the learner can understand what is on the screen as a representation of what she has done in the past. One powerful diagnostic and pedagogical feature of this approach is to contrast what the learner wanted to do (from her verbal reports) with what she managed to do (from the tool). This can then lead to a discussion of how to match those goals to the capabilities of the system.
7.3. Capture of the help-giving context
A common problem with receiving all forms of verbal help is that the
learner may forget the details as soon as he is away from the help giver.
Current solutions such as taking notes, or the help giver testing the learner
all add to the length (and thus cost) of the interaction. In the same way
that the data capture device described above can record the user’s interaction
with the system prior to the OTSL episode, a related system could record
the OTSL interaction itself. This interaction should ideally include not
just the action steps taken in using the system(s) under consideration,
but also the verbal discussion that can add so much in terms of explaining
why the steps are being done and providing useful concepts and abstractions
to help make sense of multiple action sequences and alternatives. In this
case, the usability priority is that the learner can make sense of the
underlying record on his own, after the OTSL episode is over. Provision
of abstractions, such as visualisations of the OTSL interaction as a whole,
may be useful, but is not essential. Given that OTSL interactions can be
rather rambling, include blind alleys, poor explanations and even errors
by the help giver, it may be useful to provide basic editing features so
that she can excise potentially confusing parts of the interaction. It
may be that existing technologies such as Lotus ScreenCam can serve the
needs of this usage context quite adequately. ScreenCam is intended for
mass-distribution of polished generic help, but the kind of ad-hoc, individualised
help embodied in OTSL may also benefit from the technology. This needs
further investigation.
Records of interactions once made, can then be re-used. A helper, seeing a classic problem, may pass on a pre-recorded tape and tell the recipient to try it first, with the offer of additional help if it does not prove sufficient. In the same way that a user asking for help on a Usenet newsgroup is expected to first read through the FAQ, a suitably indexed set of preserved OTSL episodes may be made available. These are all mechanisms to reduce the time costs of the helper (but not necessarily of the recipient).
7.4. Shared input devices
As noted in the study, it can be exceptionally difficult to describe
verbally how to use a graphical user interface, especially compared to
a textual user interface where one can just dictate verbally the textual
inputs. In the case of a graphical user interface, it may be easier to
just show the learner, or to point at appropriate parts of the screen.
In such circumstances, having more than one input device might be very
useful, enabling the learner to keep control of the main interface but
allowing the help giver to hint, point, or intervene as necessary. The
work of Stewart et al (1999) and Bricker et al. (1998), although aimed
at supporting collaborative learning by young children, may have much to
offer. Myers et al. (1998) have investigated the use of personal digital
assistants as input devices. Although their example scenarios and focus
is on meeting support, such devices could be of great use in OTSL, facilitating
the mobility of participants while still being able to transport part of
their context.
7.5. Adjusting the output device
The shared screen of OTSL is a wonderful resource. However, depending
on the context of use, leaning over to share a screen can be awkward. Screen
layouts in interfaces are usually designed with a single user in mind.
When sharing sound output, it is relatively easier to accommodate additional
users, as when doing a demonstration, by turning up the volume. Doing the
equivalent for the screen, is more problematic. An easy to adjust screen
resolution control may help in making things bigger for clarity in the
context of help-giving, reverting to regular size for sole use. It would
be necessary to experiment if the benefits outweighed the costs of learning
to use an interface at one size and then using it at another. Zooming interfaces
such as Pad++ (Bederson et al. 1996) may be another approach to this problem.
Other options possible in the near future include plugging in an additional
screen, so that for example, the main screen can remain focussed on the
interface to be learned, and the other screen used for learning and teaching
support, such as visualisations serving as process representations. Projection
devices, including work on the integration of familiar whiteboards and
interactive computer screens (Mynatt et al. 1999) also open up the space
options for multi-user interaction around an interface. All the possible
additional functionalities afforded by these advanced hardware technologies
must be weighed against the advantages of the basic flexibility of interacting
around the conventional work screen of a user.
7.6. Portable context
In addition to obtaining the local context of the request for help
(what the learner did immediately before, and what has already been tried),
it can be useful to have a wider context. Thus in the case of the expert
helping out at the novice’s desk, she may realise that if she were at her
own computer, she could create an explanation, or summon up an example
much more efficiently. Likewise, if it is the learner who takes the problem
to the expert’s desk, he may wish to show exactly what his interface and
system configuration looks like. In both examples, it is useful if participants
are able to move their computational context to an arbitrary machine, and
to switch easily between different contexts.
7.7. Cross-application support
The office study revealed the cross-application nature of the secretaries’
work and learning needs. Once pointed out it seems entirely plausible that
this would be an aspect of much office work. However it does impose an
additional complication for effective design. A system to support OTSL
should ideally be able to support help giving activities that jump between
applications. This adds to the complexity of the design, and for many functionality
options requires substantial access to the operating system. With most
current systems in use involving proprietary operating systems, this causes
numerous problems. However at the early stages of this research, we can
focus on systems that are a proof of concept that only work with one, or
perhaps a limited number of applications. The existence of open source
operating systems (notably Linux) and applications affords another route
to addressing the problem. In the context of supporting information search,
the growing prominence of the web both as an information resource, and
of web-based interfaces to library catalogues, bibliographic databases
etc. means that it is possible to focus on an OTSL tool to support work
solely with web browsers. This simplification will still allow us to support
the switching between applications (albeit only between web based applications),
and consequently to analyse the use of the tool in that context.
7.8. Sophisticated undo
When helping a colleague, it may first be necessary to help her recover
from the numerous activities that she has undertaken to try and solve the
problem herself. Coupled with the contextual capture features described
above, a powerful, multi-level undo feature (e.g. Prakash & Knister
1992) would be a very effective means of minimising the costs of this preparatory
phase of an OTSL interaction. Thimbleby (1990) and Dix & Mancini (1997)
consider the implications of sophisticated undo features.
7.9. Microworlds
The practice environment of a microworld can be engineered to include
features such as non-crucial files that the learner can manipulate as if
they were real ones, without the consequences of doing it for real. Many
training packages provide such features. Often the help-giver has to invent
practice activities. As they are time consuming, and often reusable, it
would be useful if she could bring along with her, her set of useful practice
materials, files etc. This may be a simple matter of networking and pulling
over exercises, or using various options for supporting portable context.
In the study, there was a case of the creation of new email messages to
enable Amanda to practice various new concepts. This involved bringing
colleagues in to form a social microworld. Although very desirable, in
cases where this is not feasible, we can envisage the creation of a practice
mailing agent that can be asked to send different kinds of messages, do
replies and other activities to enable practice of the interactive aspects
of the system’s features. To fit within the situated ethos, the files and
the activities performed on them should be as realistic as possible.
7.10. On-screen annotations
Keeping a record of the entire OTSL episode is useful for the recipient,
but it can be laborious to have to subsequently search it, merely to remember
which menu sequence produces a desired effect. In the physical world, we
have various mechanisms for indexing into complex information resources.
We attach bookmarks, labels and sticky notes to books, but also to items
of technology such as VCRs. In using an interface, it may be useful for
a learner to be able to attach a note to a menu or button, reminding her
of something that the help giver said. The design challenge here is the
very limited space available for such an attachment. Interfaces are complicated
anyway (after all that is why we have to ask others for help) without adding
a host of additional annotations. Clement (1990) includes a design suggestion
reminiscent of Hill and Hollan’s (1992) read wear, where menu choices commonly
used by others are emphasised. The resulting usage patterns can then serve
as information for a recommender system (Resnick & Varian 1997) to
suggest who to ask for help.
7.11. Remote help: over the virtual shoulder
The chief focus of this paper is on co-located synchronous interactions.
However there are many potentials for alternative kinds of OTSL, although
this also dilutes the concept as it merges with other existing kinds of
formal and informal help. Remote synchronous and asynchronous support already
exists in such forms as telephone hotlines (Pentland 1992), technical support
(Paepcke 1996), remote reference (Sloan 1998) and mailing lists, bulletin
boards etc. The research challenge would be to improve their effectiveness,
by incorporating some of the affordances of OTSL. This would make use of
various CSCW technologies to support informal communication (Isaacs et
al. 1997) by use of video, shared screens, telepointing and attempts at
establishing a richer context of use. Remote synchronous support increases
the options in terms of potential help givers, at the expense of loss of
context. Remote asynchronous support offers cost savings for the help-giver
at the expense of the convenience of the help seeker. Nevertheless both
can be useful, particularly in the case of complex problems where no-one
at hand can help. Again, the challenge is to try and build back in context
in order to speed up the interaction.
7.12. Trivial interfaces
The speed and flexibility of OTSL episodes that necessarily are ad
hoc and hopefully are brief imposes another kind of intriguing design challenge.
If one is attempting to do something difficult (preferably something that
would be extremely difficult without pen and paper), it is worth investing
time and effort in learning how to use the system. If one is using a resource
for just a few minutes or even seconds in order to help someone use a sophisticated
application, the helper resource must be simple to use, even at the cost
of limited functionality. That is especially the case where there are a
set of resources to choose from. The more powerful, more complex resources
may just be ignored in preference to the simpler ones that enable the user
to concentrate on the task at hand. Thus explainers find it easy to make
use of simple physical objects (pens, highlighters, sticky notes, pointing
at books, even using colleagues) or simple technological features (the
standard cut and paste feature across applications). If we are to try and
improve on these features, the systems we produce must compete on the cost-benefit
trade-off. This is skewed because of the brevity of the interactions. Thus,
using a video camera might be useful in teaching (it is used extensively
in music education and many sports), but in a ten minute informal interaction,
it is not worth the bother of fiddling with complex equipment. Likewise,
just setting the computer to record the verbal interaction along with the
screen interaction may be especially effective, but if it requires more
than a couple of button clicks, it just may not be used. This can be contrasted
with using the identical technology to produce help demos to be made available
to many users. In that context, the work is expected to take longer, and
there are expectations of a higher quality product than is possible in
an informal interaction. Thus it is worth investing the time and effort
in using a sophisticated tool.
The case of design to support remote help-giving provides a scenario that would be familiar to CSCW researchers - although not one that has been much investigated to date. One reason may be that the superficial resemblance hides a more fundamental difference. If one is a member of an international team working on a given project, the overhead of learning how to use the remote meeting support software can be discounted over the length of the total project activity (amounting to many hours of software use). If one is struggling with a new piece of software and calls up the remote technical support collaborative software (equivalent to the meeting support software), one hopes to be only using it for a couple of minutes. There is a distinct danger that the solution to the initial bewilderment itself involves even more bewilderment in how to operate the technical support software. This argument can be countered by claims for initial training in using the tech support software, and how its subsequent usefulness pays back for the cost of learning. However it is likely that individual’s personal cost-benefit calculations will lead to it never being adopted enough for its benefits to be manifest.
This leads to the need for what might be called ‘trivial interfaces’. The name tries to capture the problem of the design task – that usage should be as trivial as grabbing a pen and sticky note. Unfortunately it also captures the problem with working on this task – that it may be regarded as beneath the attention of skilled designers. Most of the kudos in software design goes in developing powerful and sophisticated functionality. Interface design work comes a poor second in trying to mitigate the complicating effects of this functionality. Design of trivial interfaces comes even lower, involving conscious decisions to limit functionality in the interests of simplicity. Such an approach is in many ways the computational equivalent of part of Carroll’s (1990) work on minimal manuals. A related issue is the fact that learning episodes may be interrupted. Again, this is not unique to learning, but applies to many kinds of computer use in busy offices (Rouncefield et al. 1994). Our designs must be sensitive to this fact and make it straightforward for users to resume their learning activity once the interruption is over.
Clearly such trivial interfaces are not unique to those to support OTSL. They apply to any system where it is not really worth the effort of learning how to use a powerful functionality. An example would be a walk-up information system. Nevertheless OTSL design may serve as an interesting focus for work in this area since it contrasts the flexibility of the OTSL support interface with the complexity of the application interface that is the subject of the OTSL interaction.
Finally, it should be acknowledged that the most significant changes are likely to be managerial and organisational rather than technical. If workers feel that they do not have permission to invest time in informal learning and teaching, they will at best only do it surreptitiously. Helping colleagues needs to be recognised as valuable work, rather than undesirable overhead, ‘indulging’ in activity that should be left to professional trainers, in the same way that in manufacturing, participation in quality circles is viewed positively rather than as time wasted by amateur quality experts who should spend all their time on the production line, and leave quality improvements to professionals. The skills of help-giving and asking for help can be explicitly taught (Agre 1994a&b) as a way of improving the efficiency of the interactions, just as important as the technical ways outlined.
7.13. Broader design implications
If OTSL is acknowledged as a significant aspect of how people learn
to use computers, it leads to a reconsideration of various aspects of systems
design, both functionality and interface. There are different approaches.
One involves the design of application-independent OTSL devices that can
be plugged onto any unfamiliar application in order to help the user learn
or improve by asking and receiving help. Another is to make changes to
the design of all applications so that they have a sensitivity to the existence
and needs of OTSL. This may include supporting greater compatibility with
the generic OTSL plug-on, and/or support OTSL in the absence of such a
plug-on.
OTSL may require a different approach to interface design. In conventional interface design, designers ask themselves questions such as "How can we change this interface so that it is easier for the user to work out how to do what she wants". In design with a sensitivity to OTSL there is an additional question: "How can we change this interface so that it is easier for the user to ask and receive help on how do what she wants from someone else?". It remains to be seen the degree to which these two kinds of questions yield different kinds of solutions, and how we might make trade-off decisions between them. It is likely that many of the technologies developed to support OTSL, including those outlined above, will also support individual learning. However this will not always be true, nor will the reverse; that technologies to support individual learning will also adequately support OTSL.
Most research on HCI has focussed on the interaction between an individual user and a computer system via the interface. The advent of computer supported cooperative work and computer supported collaborative learning has broadened this concept to include the use of computer systems as a medium via which people interact with other people. This can involve aspects of conventional interface design (interacting with the system in order to be able to interact with people) as well as new kinds of media and interfaces to support human-human interaction. This latter draws on research in communications, social psychology and business practice amongst other areas for insights into how to improve the quality or efficiency of these interactions. Traditionally, CSCW focussed on remote interaction, where networked computers are used in attempt to enable dispersed individuals to work as if they were co-located. Recently there has been a resurgence of interest in co-located collaboration, including the use of shared whiteboards and other kinds of hardware (Mynatt et al. 1999), as well as a conventional computer plus display. This last is sometime called single display groupware (Stewart et al. 1999). In these cases the focus is on how people already co-located either around a desktop, or moving to appropriate technologies available in an advanced meeting room (Streitz et al. 1998), can have their conventional interactions augmented by the provision of advanced technologies. OTSL can be seen to fit within that viewpoint, provided certain clarifications are made. Firstly, to be effective OTSL must focus on supporting the learning of other systems – it is strictly a technology that is a means to an end. Secondly, it is not a mode of working that we must invent – it happens anyway, regardless of what we as designers do to help or hinder it. The design task is to seek ways to enable it to work more efficiently, and where appropriate, more frequently. A useful slogan to remember is that in the case of OTSL, every application is (temporarily) a collaborative application. Unfortunately, these interactions were never designed with an awareness that they would be used collaboratively (even if only very briefly, perhaps in occasional five minute bursts).
Unlike research in intelligent user interfaces (Maybury & Wahlster 1998), or intelligent tutoring systems (Wenger 1987) the aim is not for the system to diagnose and remediate the difficulties of the user. Such intelligent activity is left to the help giver. The design task is to develop mechanisms to aid that interaction. As such it has strong parallels with research in Computer-Supported Collaborative Argumentation (Buckingham Shum 1998). To emphasise the point, the interface design task is less to support the interaction between the user and the system than to support the interaction between the learner and the help giver. The technologies and especially the interface serves as a notation, or a reification that is used as a resource in the clarifying dialogue between the human participants (Lajoie & Derry 1993). This leads to a different kind of metaphor – instead of the screen as a window onto the data and its manipulation, or a window through which to collaborate with others, we should (also) consider it as a kind of dynamic, interactive whiteboard – a resource for discussion with another person.
Hutchins (1990) shows how technological devices are better seen as media for representation than as amplifiers or surrogates for cognitive abilities. He notes how different representations using different technologies can transform a problem so that it is easier for people to solve. We can look to innovative interface design to tackle the same kind of problem. Note that this is different from saying that it is possible to automate a problem (so that a user just has to input a minimal amount of data and receive the answer). In re-representation, the person is still involved in the problem solving activity, but makes use of different representations (potentially including those that we could implement on a computer screen) to make the problem easier, by changing what is required of the problem-solver to something that is easier to do. Hutchins also notes that the nature of tools affects their suitability for supporting interaction and hence learning. An open tool makes much of the work visible and so open for the detection of errors (or for raising questions by learners) and provides a forum for discussion.
The reconsideration of interfaces proposed here, especially in its focus on context and the use of interfaces as conversational resources, has strong links with activity theory (Nardi 1996). In particular, Kaptelinin (1996) explores how collaborative systems mean that computer interfaces can be both individual artifacts and group artifacts, viewed as part of a wider context of human interaction with computers.
OTSL can also be considered an example of ‘Designing for Failure’. Users resort to OTSL when they are unable to discover the solution themselves. Thus OTSL is a symptom of the failure of a supposedly easy to use interface. If we were confident that we could design an interface that was so well constructed that end users could easily figure out how to use it on their own, it would make sense to allocate resources to that design and not to bother with OTSL. If we are not that confident, then it can be justifiable to spend time on developing features to support OTSL. But we must acknowledge that this is a design trade-off. More resources for OTSL means less for improving the usability of the underlying design.
8. Research questions and future work
This paper seeks to raise problems more than to propose solutions.
Clearly we need more studies of OTSL in a wide variety of settings in order
to inform our understanding of the process and hence inform our design
of technologies that may facilitate it. Ethnographic techniques will be
a powerful tool in this approach. A significant concern is that in the
absence of an understanding of a subtle aspect of OTSL, we will develop
tools that are quite useless when tried out in authentic settings. The
study of OTSL is rendered more difficult by some of the features that make
for its success – its fluidity, speed, adaptability to the circumstances
and resources at hand and its strong interweaving with the more dominant
work activity.
Complementing the qualitative insights of ethnography, it would be useful to obtain some quantitative information about OTSL. How much does it happen? How many people use it? What proportion of learning does it entail? What are its (quantifiable) costs? What are its (quantifiable) benefits? Whether it is feasible to obtain such information in a form that is not hopelessly compromised remains to be seen, but it would clearly be useful in the political negotiations of allocating resources to encouraging OTSL. The clearest analogy is with the work devoted to cost-justifying better interface design (Nielsen, 1993). In cases where a manager perceives an activity as a mere overhead (OTSL, usability engineering), having numerical estimates, even if only as unreliable as figures for potential sales, can help in the negotiation over the allocation of resources.
The kinds of systems described in the design implications section need to be developed and tested. As with all iterative design, this can yield valuable insights from what did and did not work in authentic contexts. The information obtained can be a valuable complement to the observational studies of existing practice derived from ethnographic study.
More detailed study may reveal answers to questions such as:
References
Ackerman, M. S. and T. W. Malone (1990). Answer Garden: a tool for
growing organizational memory. Proceedings of ACM Conference on Office
Information Systems, 31-9.
Agre, P. (1994a). The art of getting help. The Network Observer 1(2)
http://dlis.gseis.ucla.edu/people/pagre/tno/february-1994.html#help
Agre, P. (1994b). How to help someone use a computer. The Network Observer
1(5)
http://dlis.gseis.ucla.edu/people/pagre/tno/may-1994.html#how
Alty, J. L. and Coombs, M. J. (1980). Face-to-face guidance of university computer users-I: A study of advisory services. International Journal of Man-Machine Studies, 12, 389-405.
Bannon, L. J. (1986). Helping users help each other. User centered system design: new perspectives on human-computer interaction. D. A. Norman and S. W. Draper. Hillsdale, NJ; London, Lawrence Erlbaum Associates: 399-410.
Bederson, B. B., J. D. Hollan, et al. (1996). Pad++: a zoomable graphical sketchpad for exploring alternate interface physics. Journal of visual languages and computing 7: 3-31.
Berlin, L. M. and R. Jeffries (1992). Consultants and Apprentices: Observations about Learning and Collaborative Problem Solving. Proceedings of ACM CSCW'92 Conference on Computer-Supported Cooperative Work: 130-137.
Blomberg, J. L. (1987). Social interaction and office communication: effects on user evaluation of new technologies. Technology and the transformation of white-collar work. R. E. Kraut. Hillsdale, NJ; London, Lawrence Erlbaum Associates: 195-210.
Bloom, B. S. (1984). The 2 sigma problem: the search for mathods of group instruction as effective as one-to-one tutoring. Educational Researcher 13(6): 4-16.
Bricker, L., Baker, M., Fujioka, E., Tanimoto, S. (1998) Colt: A System for Developing Software that Supports Synchronous Collaborative Activities, Proceedings of EdMedia '99, Seattle, WA.
Brown, J. S. and P. Duguid (1991). "Organizational learning and communities-of-practice: toward a unified view of working, learning, and innovation." Organization Science 2(1): 40-57.
Bruckman, A. (1997). MOOSE Goes to School: A Comparison of Three Classrooms Using a CSCL Environment. Proceedings of CSCL 97; Toronto, Canada, December 1997.
Bruckman, A. (1998). Community support for constructionist learning. Computer Supported Cooperative Work: The Journal of Collaborative Computing 7(1-2): 47-86.
Buckingham Shum, S. (1998). Negotiating the Construction of Organisational Memories. In U.M. Borghoff and R. Pareschi (Eds.), Information Technology for Knowledge Management (pp. 55-78). Springer: Berlin.
Bulkeley, William M. (1992). Study Finds Hidden Costs of Computing, Wall Street Journal, November 2.
Carroll, J. M. (1990). The nurnberg funnel: designing minimalist instruction for practical computer skill. Cambridge, MA, The MIT Press.
Center for Workforce Development. (1998). The teaching firm: where productive work and learning converge: report on research findings and implications. Newton, MA: Center for Workforce Development, Education Development Center.
Clement, A. (1990). Cooperative support for computer work: a social perspective on the empowering of end users. Conference on Computer-Supported Cooperative Work, Los Angeles, CA, ACM Press, 223-236.
Cole, E. (1984). Sources of Problem Solving for Computing End-Users: Human Computer Interface and Social Networks in Organizations. In Human Factors in Organizational Design and Management, H.W. Kendrick and O. Brown, Jr. eds., New York: North Holland, 1984, pp. 213-218.
Constant, D., S. B. Kiesler, et al. (1996). The kindness of strangers: The usefulness of electronic weak ties for technical advice. Organization Science 7(2): 119-135.
Crabtree, A., Twidale, M.B. O'Brien, J. & Nichols, D.M. (1997). Talking in the Library: Implications for the Design of Digital Libraries. Proceedings of ACM Digital Libraries '97, (Eds.) Allen, R.B. & Rasmussen, E., Philadelphia, PA, 221-228.
Dix, A and Mancini, R (1997). Specifying history and backtracking mechanisms. In Formal Methods in Human-Computer Interaction, Eds. P. Palanque and F. Paterno. London, Springer-Verlag. pp. 1-23.
Eales, R. T. J. and J. Welsh (1995). Design for collaborative learnability. The First International Conference on Computer Support for Collaborative Learning, Indiana University Bloomington, Indiana, USA, Lawrence Erlbaum Associates, Inc., 99-106.
Eveland, J. D., A. Blanchard, et al. (1994). The role of "help networks" in facilitating use of CSCW tools. ACM 1994 Conference on Computer Supported Cooperative Work, Chapel Hill, NC, USA, ACM SIGCHI & SIGOIS, 265-274.
Furnas, G., T. K. Landauer, et al. (1987). The vocabulary problem in human system communication. Communications of the ACM 30(11): 964-971.
Gantt, M. and B. A. Nardi (1992). Gardeners and Gurus: Patterns of Cooperation among CAD Users. Proceedings of ACM CHI'92 Conference on Human Factors in Computing Systems: 107-117.
George, J. F., S. Iacono, et al. (1995). "Learning in context: extensively computerized work groups as communities-of-practice." Accounting, Management and Information Technology. 5(3/4): 185-202.
Hill, W. C. and J. D. Hollan (1992). Edit wear and read wear. Proceedings of the Conference on Human Factors in Computing Systems. CHI'92, Monterey, CA, ACM, 3-9.
Hutchins, E. (1990). The technology of team navigation. Intellectual Teamwork: Social Foundations of Cooperative Work. J. Galegher, R. E. Kraut and C. Egido. Hillsdale, New Jersey, Lawrence Erlbaum Associates: 191-220.
Isaacs, E.A., Whittaker, S., Frohlich, D., and O'Conaill, B. (1997) Informal communication re-examined: New functions for video in supporting opportunistic encounters. In Finn, K.E., Sellen, A.J., and Wilbur, S.B. (Eds.), Video-Mediated Communication, New Jersey: Lawrence Erlbaum, pp. 459-485.
King, J. L. (1996). Where are the payoffs from computerization? Technology, learning and organizational change. Computerization and Controversy value changes and social choices. R. Kling. San Diego, CA, Academic Press: 239-260.
Kraut, R. R., Fish, R. S., Root, R. W., & Chalfonte, B. L. (1990). Informal Communication in Organizations: Form, Function, and Technology. In S. Oskamp & S. Spacapan (Eds.), People's Reactions to Technology in Factories, Offices, and Aerospace (Proceedings of The Claremont Symposium on Applied Social Psychology) (pp. 145-199 Newbury Park, CA: SAGE Publications.
Lajoie, S. P. and S. J. Derry, Eds. (1993). Computers as cognitive tools. Hillsdale, NJ, Lawrence Erlbaum Associates.
Landauer, T. K. (1995). The trouble with computers: usefulness, usability, and productivity, MIT Press.
Lang, K.N., Lang, T. & Auld, R. (1981). Support for Users of Operating Systems and Applications Software. International Journal of Man-Machine Studies v.14 n.3 p.269-282.
Lang, K.N., Auld, R. & Lang, T. (1982). The goals and methods of computer users. International Journal of Man-Machine Studies, 17:375-399.
Lave, J. and E. Wenger (1991). Situated learning : legitimate peripheral participation. Cambridge, UK, Cambridge University Press.
Lee, D. (1986). Usage Pattern and Sources of Assistance for Personal Computer Users," MIS Quarterly, 10(4) pp. 313-325.
Lotus Development Corporation. (1999). ScreenCam http://www.lotus.com/home.nsf/tabs/screencam (Accessed June 21 1999).
MacLean, A., K. Carter, et al. (1990). User-Tailorable Systems: Pressing the Issues with Buttons. CHI '90, Seattle, Washington. April, ACM Press, 175-182.
Mackay, W. E. (1990). Patterns of sharing customizable software. CSCW'90 Conference on Computer-Supported Cooperative Work, Los Angeles, CA, ACM SIGCHI & SIGOIS, 209-222.
Maybury, M. T. and W. Wahlster, Eds. (1998). Readings in intelligent user interfaces. San Francisco, CA, Morgan Kaufmann.
Myers, B. A., H. Stiel, et al. (1998). Collaboration using multople PDAs connected to a PC. CSCW '98, Seattle, WA, ACM Press, 285-294.
Mynatt, E. D., T. Igrashi, et al. (1999). Flatland: new dimensions in office whiteboards. CHI '99, Pittsburgh, PA, ACM Press, 346-353.
Nardi, B. A. and J. R. Miller (1991). Twinkling lights and nested loops: Distributed problem solving and spreadsheet development. International Journal of Man Machine Studies 34(2): 161-184.
Nardi, B. (1993). A small matter of programming: perspectives on end user computing. Cambridge, MA, MIT Press.
Nardi, B. A. and V. O'Day (1996). Intelligent agents: what we learned at the library. Libri 46(2): 59-88.
Nardi, B. A. and V. L. O'Day (1999). Information ecologies: using technology with heart. Cambridge, MA, MIT Press.
Neal, L. (1990). Implications of computer games for system design. Human Computer Interaction- INTERACT 1990, North Holland, Elsevier Science Publishers, 93-99.
Nielsen, J. (1993). Usability Engineering. London, Academic Press.
Nolan, Norton & Company (1992). Managing end-user computing. Boston: Nolan, Norton & Company
O'Day, V., D. Bobrow, et al. (1998). Moving practice: from classrooms to MOO rooms. Computer Supported Cooperative Work: The Journal of Collaborative Computing 7(1-2): 9-45.
O'Malley, C. E. (1986). Helping users help themselves. User centered system design: new perspectives on human-computer interaction. D. A. Norman and S. W. Draper. Hillsdale, NJ, Lawrence Erlbaum Associates: 377-398.
Orr, J. E. (1996). Talking about machines: an ethnography of a modern job. Ithaca, Cornell University Press.
Paepcke, A. (1996). Information Needs in Technical Work Settings and their Implications for the Design of Computer Tools. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5:63-92.
Papert, S. (1987). Microworlds: transforming education. AI and Education: learning environments and tutoring systems, Vol. 1. R. W. Lawler and M. Yazdani. Norwood, NJ, Ablex. 1: 79-94.
Pentland, B. T. (1992). Organizing moves in software support hot lines. Administrative Science Quarterly 37(4): 527-48.
Prakash, A. & Knister, M.J. (1992). Undoing actions in collaborative work. In CSCW'92 - Proceedings of the Conference on Computer-Supported Cooperative Work, pp. 273-280. ACM Press.
Resnick, P. and H. R. Varian (1997). Recommender Systems. Communications of the ACM 40(3): 56-58.
Rieman, J. (1993). The Diary Study: a workplace-oriented research tool to guide laboratory efforts. Proceedings INTERCHI'93 Conference (Amsterdam), ACM Press, New York, pp. 321-326.
Rogoff, B. (1990) Apprenticeship in thinking. New York: Oxford University Press.
Rouncefield, M., J. A. Hughes, et al. (1994). Working with "constant interruption" CSCW and the small office. Computer Supported Cooperative Work, Chapel Hill, NC, ACM, 275-286.
Scharer, L. L. (1983). "User training: less is more." Datamation(July): 175-182.
Sloan, B. (1998). Service perspectives for the digital library: remote reference services. Library Trends 47(1): 117-143.
Stewart, J., B. B. Bederson, et al. (1999). Single display groupware: a model for co-present collaboration. CHI '99, Pittsburgh, PA, ACM Press, 286-293.
Streitz, N. A., J. Geißler, et al. (1998). i-LAND: An interactive landscape for creativity and innovation. CSCW '98, Seattle, WA, ACM Press, 120-127.
Thimbleby, H. (1990). User Interface Design. New York, NY, ACM Press.
Trauth, E. M. and E. Cole (1992). The organizational interface: a method for supporting end users of packaged software. MIS Quarterly 16(1): 1-17.
Twidale, M.B., Nichols, D.M. & Paice, C.D. (1997). Browsing is a Collaborative Process, Information Processing & Management, 33(6), 761-83.
Twidale, M. B. and D. M. Nichols (1998). Designing Interfaces to Support Collaboration in Information Retrieval. Interacting with Computers 10(2): 177-193.
Van der Meij, H. and J. M. Carroll (1998). Principles and heuristics for designing minimalist instruction. Minimalism beyond the Nurnberg funnel. J. M. Carroll. Cambridge, MA, MIT Press: 19-53.
Vygotsky, L. (1986). Thought and language. Cambridge, MA, MIT Press.
Weinberg, G. M. (1971). The psychology of computer programming. New York, Van Nostrand Reinhold.
Whittaker, S., D. Frolich, et al. (1994). Informal workplace communication:
what is it like and how might we support it? CHI '94, Boston, MA, ACM Press,
131-137.