RSD-TR.19-87 TELE-AUTONOMOUS SYSTEMS: METHODS AND ARCHITECTURES FOR INTERMINGLING AUTONOMOUS AND TELEROBOTIC TECHNOLOGY Lynn Conway Richard Volz Michael Walker October 1987 CENTER FOR RESEARCH ON INTEGRATED MANUFACTURING Robot Systems Division COLLEGE OF ENGINEERING THE UNIVERSITY OF MICHIGAN ANN ARBOR, MICHIGAN 48109-1109

-1 074'S1-.';)l':

RSD-TR- 19-87 TABLE OF CONTENTS 1. INTRODUCTION.............................................................. 2 1.1 Paradigms of Telerobotics.............................................................. 2 1.2 Paradigms of Autonomous Systems.................................................... 3 1.3 Illustrative task examples.............................................................. 4 1.4 Merging the augmented paradigms...................................................... 6 2. BASIC TELE-AUTONOMOUS SYSTEM CONCEPTS................................. 6 2.1 Expanding the functions of telerobotics.............................................. 6 2.2 Expanding the functions of autonomous systems............................... 8 2.3 Integration of tele-autonomous function.............................................. 9 3. TELE-AUTONOMOUS SYSTEM ARCHITECTURE AND CONTROL METHODS....................................................................................................... 9 3.1 Disengaging time-control synchrony using a Time-Clutch................ 10 3.2 Disengaging position synchrony using a Position Clutch................... 11 3.3 Scenario showing use of the Time and Position Clutches................. 12 3.4 Time-Ratio Control.............................................................. 13 3.5 Handling of contingencies by using Time Brakes.............................. 14 3.6 Manipulation and cognition control-transitions and their mirrorings in telerobotics and supervised autonomous systems........ 14 * -

RSD-TR- 19-87 4. INITIAL RESEARCH ISSUES AND HYPOTHESES.................................... 16 5. EXPERIMENTAL ENVIRONMENT AND RESULTS................................... 17 5.1 Experimental Parameters, Trials and Results...................................... 19 5.2 Plans for further experiments and concept demonstrations................ 22 6. FUTURE RESEARCH CHALLENGES........................................................... 24 7. SUMMARY........................................................................................................ 26 8. ACKNOW LEDGEMENTS.......................................................................... 26 9. REFERENCES................................................................................................... 27 iii

[Paper presented at the 1987 IEEE International Conference on Robotics and Automation] TELE-AUTONOMOUS SYSTEMS: METHODS AND ARCHITECTURES FOR INTERMINGLING AUTONOMOUS AND TELEROBOTIC TECHNOLOGY Lynn Conway, Richard Volz and Michael Walker Robotics Research Laboratory Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor, Michigan 48109 ABSTRACT As a result of recent advances in artificial intelligence, human cognitive modelling, autonomous systems and telerobotics, there is an opportunity to broaden our concepts of technology for projecting action at a distance. We draw on these advances to develop a conceptual and architectural framework that enables efficient projection in time and space of intermingled manipulation and cognition tasks. Where AI-based autonomous systems have previously been concerned with human supervisory intervention primarily at a cognitive level, we add methods for rendezvous, capture and rehandoff of embedded manipulation tasks. Where telerobotics has been concerned with the projection of sensory-motor manipulation, we add the projection of cognitive processing. Thus extended, the two technologies mirror one another and merge into one of "tele-autonomous systems". We introduce notions of how the sensory, cognitive and motor functions of teleautonomous systems can be factored and transferred back and forth between human and machine. We illustrate how the times to complete tele-autonomous tasks can be reduced through time and space constraint relaxations effected through simple controls: We employ the concepts of forward simulation and predictor display, augmented by "time and position clutches", "time ratio controls" and "time brakes", to control the resulting manipulation paths and event transitions. We sketch some generic architectural and human interface 1

implications of these methods. Finally, we describe our environment for exploring these methods and the results of some recent experiments. INTRODUCTlION In this paper we draw on recent research advances in autonomous systems and telerobotics, and develop a framework for an integrated technology that enables efficient projection in time and space of intermingled manipulative and cognitive tasks. The technology builds a bridge between telerobotics and intelligent autonomous systems by providing methods for controlling real-time transitions between human and machine control of remote events. In this introductory section we reflect on the paradigms of the telerobotics and autonomous systems research communities. We illustrate gaps in the two paradigms, and opportunities for technology integration, by describing practical tasks humans can do that would be hard to implement with either technology alone. In the later sections of the paper we introduce functional concepts and terminology for "tele-autonomous" or "tele-automation" technology. We present architectural and control methods for implementing these concepts, discuss our experimental environment for exploring these concepts, and finally present the results of some recent experiments. 1. 1. Paradigms of Telerobotics Up till now the concerns of the telerobotics community have been primarily those of the roboticist and control theorist, dealing with sensing the physical environment, measuring positions, forces, and accelerations, and responding with movements and forces to directly manipulate the physical environment. The human provides the cognitive power of the system, with the human's sensory-motor processing intermediated and projected at a distance by the machine.

The coin of the telerobotics realm is manipulation. Support for telerobotics has come primarily from DOE for projection of manipulation capability into hazardous environments. Support has also come from NASA for projection of manipulation into the space environment, and from DoD for undersea applications. When teleroboticists discuss the projection of "autonomous intelligence" to remote mechanisms, the projected capabilities are usually envisioned as programs that can be invoked to independently carry out physical manipulation tasks while the human remains in contact and control at a supervisory level [SHE86]. A common goal of telerobotics research is the production of as realistic a sense of remote telepresence and telecontrol for the human operator as possible, given physical constraints such as communication delay times [NOY84, SHE86]. The goal is to enable operators to do as nearly as well at manipulation tasks as they could do if physically present at the remote location. 1.2 Paradigms of Autonomous Systems Within the past few years, the U. S. Department of Defense has been supporting a rapidly growing autonomous systems research community [DAR83, DAV85]. This community's concerns are those of computer scientists and artificial intelligence researchers working to produce self contained, mobile platforms, such as the Autonomous Land Vehicle (ALV) [MAR86] and various autonomous undersea vehicles, that can maneuver around and employ machine cognition to seek high-level goals in their environments. The focus is on mechanization of sufficient cognitive power to achieve interesting goals, such as complex route planning and replanning to effect reconnaisance or force projection missions, and on providing sufficient perception and maneuvering capability to do things like follow roads, avoid obstacles, and find things in the environment [DAR83, MAR86]. Given present limitations and computational complexity of self-contained machine perception (such as machine vision), the sensory-motor aspects of autonomous systems 3

technology are currently rather crude when compared to telerobotics, which can exploit human perception. The community thus tends to focus on widening the exploitation of machine cognition on tasks that are feasible given the envelope of available perception technology, conducting a parallel effort on incremental enhancement of perception technology performance [MAR86]. The coin of this realm is cognition, and cognitive interaction with the environment at a symbolic level. A common goal of autonomous systems research is the mechanization of cognition and the associated task-dependent knowledge systems so that the remote machine is as smart, robust, knowledgeable and persistent as a human might be in attempting to carry out its mission. Since the focus of the work is on autonomy, human supervision or interaction is seldom stressed. When the notion of supervisory control appears in autonomous systems, it usually is concerned with having the human intervene if the system is "not smart enough" to cognitively handle a given situation [MAR86]. 1.3. Illustrative task examples The following task examples shed light on our problem space, and suggest opportunities and methods for blending telerobotics and autonomous systems. Consider text editing on a workstation. A human operator can often envision and generate the command sequence to achieve a local goal much faster than the workstation can effect the screen manipulation. Thus the human may quickly "type or mouse ahead" (assuming the control stream can be buffered), then shift their cognitive or manipulation attention to the task to be done when the machine catches up [CAR83]. In contrast, in teleoperation systems the operator is often "slaved in real-time" to the local and remote sensory-motor apparatus. Can we imagine a telerobotic analogy to type-ahead? An extension of "type ahead" occurs when the operator has constructed their own, perhaps intelligent, high-level commands such as "sort this list", or "send a message to X to get the address of Y". The operator may then type ahead at a rather high level, with each

command in the sequence performing not just physical manipulations but also elaborate symbolic manipulations of the environment to eventually produce the text. Again, can we imagine a telerobotic analogy? Or, imagine that you are learning to fly in aircraft that has dual controls. On a given flight your cognition may be just fine, but you suddenly fail to manage a manipulation task, and the instructor takes over. By analogy, an autonomous system might be doing fine in its cognitive tasks, but might need occasional human help in its "lower-level" manipulation tasks. For example, an ALV might run off the road and get stuck, and require a skilled "teledriver" to free it; this is quite a different form of intervention than the "mental" supervisory intervention usually envisioned by ALV'ers. Intervention into an ongoing autonomous manipulation task may not be easy, since taking over in mid-maneuver may involve smoothly effecting a multidimensional control rendezvous. You can study a simple form of this situation by interacting with the cruise control of your automobile. The dual-controlled aircraft story yields several scenarios that have interesting autonomous system analogues. The instructor can coach the student on various sensorymotor manipulation tasks, and on various cognitive tasks such as interpreting instrument readings. Visualize the piloting coach as a human supervisor, and the student as a remote autonomous system: The coach can take over either cognition (correcting an instrumentation interpetation) or take over lower-level manipulation (prevent an unwanted stall). There is a matrix of possible division of responsibilities. Sensing, thinking, and acting can be separately assigned and reassigned at any moment to either the supervisor or the system. But what are the embedded protocols that make such human practices feasible? What shared knowledge is involved? How are the transitions performed? How do both student and coach know who's doing what at any moment? To make the picture even more interesting, consider the fact that the overall system has full duality: the role of coach and

student is reversible under some situations. Could such insights have architectural implications for general autonomous systems? 1.4. Merging the augmented paradigms Can we somehow build a solid bridge between these two technologies so as to merge them? We believe the answer is yes, as discussed in following sections. We also suggest that qualitatively new kinds of functions and new opportunities for performance improvement appear as a result. 2. BASIC TELE-AUTONOMOUS SYSTEM CONCEPTS Consider either a teleoperator or a supervised autonomous system consisting of (i) a human, and (ii) a machine that is partly local to the human and partly at the remote site of intended projected activity. Among the concerns of architects of such systems are: In specific situations, what is the human best at? Worst at? What is the machine best at? Worst at? How can we shift control between human and machine to exploit these capabilities? Are there generic architectural principles to draw on? What are the constraints on ultimate performance? What traditional constraints can we find ways around? In this section we will explore these questions to develop step-by-step some basic functional concepts for tele-autonomous systems. 2.1. Expanding the functions of telerobotics Let's first reexamine some of the architect's questions from the "tele-autonomous" point of view. One important constraint on performance in certain key applications is the time delay for communications between the local and remote system. Noyes and Sheridan [NOY84] have innovated and demonstrated a very novel way to cope with such a delay, by using a locally situated forward-in-time telerobot simulator, and a graphical "predictor display" overlay of the forward simulation onto the fixed-delay return video of remote 6

teleoperation (Fig. 2. 1). Such forward simulation enables an operator to move the controls and immediately visualize the effect of control action without waiting for the return video. Their experiments show that the time to perform manipulation tasks in the presence of communication delays can be reduced by exploiting such predictor displays (Fig. 2.2). But instead of just finding ways to better cope with constraints, can we also find ways to relax some constraints? Suppose we had a forward simulator and predictor display, but were not operating through a large time delay. Although we needn't enter commands prior to the observed time of their remote execution, we still might want to do so, and we could use a modified version of forward simulation to do so. For example, we might be able to enter commands a lot faster than the telerobot could carry them out, as in "type ahead". Graphical overlay of accelerated forward simulation enables us to do this, and to then manage the cognitive limitations of short term memory when commanding in advance of observed effects. Such simulation can be augmented by including kinematic and/or dynamic forward simulation of portions of the environment. The result is a sort of coordinated "faster than real-time recording, and then real-time playback" form of manipulation control. There appear to be a number of ways that displays of simulation overlain onto telerobot video can relax telerobotic time-synchrony constraints, yielding possibilities for improvements in manipulation time performance. We also hypothesize that freeing the operator from the "time-slaving" and attention constraining aspects of time-synchronized control may make a qualitative difference in the subjective "feel of the controls" of such systems, making them more like the controlling of one's own limbs. This hypothesis may eventually be made testable by defining new measurements of performance and fatigue in new forms of telerobotic situations. We can also visualize telerobotic manipulation as analogous to text editing, in that it is a series of sensory-motor limited tasks intermixed with cognitively-limited tasks. Thus in some situations we may be able to project intelligent cognitive functions into the

manipulation world (analogous to the "go find the address" command during text editing) while continuing direct editing manipulations. Methods that relax the constraint of command-to-manipulation time-synchrony might enable operators to better intermix such tasks. 2.2. Expanding the functions of autonomous systems The transfer of cognitive tasks between supervising human and remote machine is already a part of the autonomous systems paradigm, being based on past artificial intelligence work on human machine cooperation in areas like diagnostics, design, advising and coaching [HAY83]. Shifts between machine and human manipulation while the machine retains cognitive control have not usually been considered. However, these can now be seen as just a mirror image, role-reversed version of the augmented telerobotics described above. Any forward simulation, time manipulation and control methods that work there will apply here also. In both cases we must deal with control and human interfacing of rendezvous, capture and rehandoff of manipulation tasks between human and machine. As we discover control and human interface methods for such manipulation task transfers, perhaps we can also gain insights into how to better structure the methods for cognitive task transfer between human supervisor and autonomous system. Those methods are presently rather ad-hoc, being based on diverse applications experiences in AI. Finally, there is the human interface challenge of presenting "who has control, of what, and at what time and position?" The human may set goals into the autonomous system, and then later be called on to enter tasks to help the system reach either cognitive or manipulation subgoals. Can we use some sort of task lattice or tree, to represent and interface the distributed tasks underway towards goals and subgoals? New human computer interaction knowledge and technology must be developed to support these new possibilities for autonomous systems. 8

2.3. Integration of tele-autonomous function The distinctions between telerobotics and autonomous systems blur when the technologies are each expanded as discussed above. But we don't just get the sum of the two technologies. We get a technology with some new dimensions for enabling action at a distance. This leads us to wonder if we should revise the goal of telerobotics. Could it be possible to project manipulation capabilities to a distance that are better in time performance than those of the unaided human? Considerable research will be required to generate and test hypotheses to determine feasible perfonrmance improvements and limits of such an extended tele-manipulation technology. In addition to examining new aspects of robotics and control methods, this research will also enter previously unexplored areas in the psychology of human-computer interfacing. 3. TELE-AUTONOMOUS SYSTEM ARCHITECTURE AND CONTROL METHODS In this section we describe methods for relaxing the operator to manipulator timesynchrony constraints usually found in telerobotics. The first key idea is the use of a "time-clutch" to enable disengagement of time synchrony during path planning. We extend this idea by adding a "position clutch" that allows forward simulation manipulation and positioning trials without generating path plans. We include a "time-ratio control", to enable variations in the ratio of simulation time to real time. We introduce the concept of a "time brake" to allow the forward simulation to be "braked" back in time to avoid unforseen contingencies. We then provide scenarios of how these new controls might enable operators of the augmented systems to achieve considerable time improvements in certain manipulation tasks. We also suggest how the augmented architectures enable easy transitions of control of cognition and manipulation tasks between human and machine, thus enabling integrations and mirrorings of telerobotic and intelligent autonomous functions. 9

3.1. Disengaging time-control synchrony using a Time-Clutch We build upon the forward simulation and predictor display concept of Noyes and Sheridan as follows. Suppose we are using a telerobotic system as in Fig. 2.1. We augment the system with a control that we call a "Time Clutch". This control enables us to disengage the "direct gearing" or time-rate (but not absolute time) synchrony of simulated time and real-time, and move the forward simulator ahead as fast as skill and judgement will allow. The predictor display presents a forward path as a goal, that is as a sequence of point positions to be followed by the system as fast as is feasible. Note that the path could be generated subject to some settable mean error parameter, for example as a "tube" of given radius [SUH87]. The time clutch enables an operator to disengage from real-time, and manipulate ahead of the displayed video of the real manipulator by working the overlay wire-frame figure of the manipulator on the predictor display. Example situations where this would have benefit would be during slow movements of large space structures and in slow undersea vehicle manipulations. The operator can thus do the telerobotic equivalent of "type-ahead" and then perhaps slow down and carefully position for some tricky maneuver. We hypothesize that in many manipulation task sequences such time saving accumulations and later exploitations will be possible, thus reducing overall manipulation task times and also the fraction of the task time that requires operator involvement. How can we implement the time clutch control? How is the system to determine the path as a function of time when the clutch is disengaged? The time clutch can be thought of as a simple switch used to make or break the connections within the kinematic/dynamic robot simulation that would normally constrain the rate at which the forward simulator could be slewed around in space. When the clutch is engaged, the position (or rate) joystick control of the simulator is sampled and directly controls the movements of a simulator model which is constrained in its movement rates and accelerations as if it were a real 10

robot. A buffer is inserted between the simulator and the telerobotic manipulator, to hold the stream of sampled position increments as incremental "move to" commands (see Fig. 3.1). With the time clutch is engaged, the command buffer presents a stream of position points at a fixed sample rate, and the telerobot can simply increment its position accordingly. But when the time clutch is disengaged, the distance between successive path positions may be greater than the telerobot can move in a time sample, and an interpolator is used to generate intermediate points along the path. This interpolator can always be active, with the only difference in function upon disengaging the time clutch being the breaking of simulator contraints on simulator velocities and accelerations. In sophisticated systems where telemanipulator touch sensing and force-sensing during interactions with the environment are reflected back to the manipulator operator, disengagement of the timeclutch must also disengage these reflected forces and substitute simulated forces generated by the simulator. 3.2 Disengaging position synchrony using a Position Clutch In some cases, we may want to move the forward simulator in space without actually sampling the path, for example to pre-position for a complex manipulation. Thus we may wish to disengage the simulator from recording any positioning commands. To do this we disengage a'IPosition-Clutch" that allows forward simulation without path planning. This provides a positioning-synchrony constraint relaxation analogous to the earlier time-synchrony relaxation. In this case no position information is entered into the command buffer until the position clutch is reengaged, at which time the reengagement position is entered into the control buffer, and later used by the actual robot in path interpolation from the previous path position. If the real system catches up with a positionclutch disengagement point, it hits an "empty mark" in the command buffer and must wait for further path data to enter the buffer (see Fig. 3.1). 11

Note that the time-clutch can be disengaged while the position clutch is engaged. But disengaging the position clutch overrides any actions of the time clutch. Reengaging the time-clutch after an interval of time-saving places the forward simulator and predictor display in much the same relationship to the remote unit as when operating through a time delay, with the operator directly generating a synchronized time and position trajectory in advance of the return video. In all these cases, use of time and position clutches can be superimposed over time delays in the communications between the local and remote machines (assuming adequate buffer capacity). The command buffer can be constructed to hold more complex commands (in parallel) than just simple moves, enabling the operator to mark certain path positions as places where an embedded task is to be done. For example, suppose a switch must be pushed at some point along a path, and that the manipulation program for switch pushing resides in the remote controller. The operator might just mark the spot on the path when the forward simulator reached the switch (momentarily disengaging the time and position clutches and manipulating a screen menu entry signifying switch pushing). The telerobot manipulator (or remote vehicle, etc.) would then execute the task when it arrived at that point on the real path, i.e., when that path information emerged from the command buffer. 3.3. Scenario showing use of the Time and Position Clutches A short scenario for using the two clutches follows: We perform a complex maneuver with clutches engaged. We then disengage the time-clutch to quickly hop over a series of simple manipulation movements, such as pushing a series of switches. A faint "smoketrail" superimposes the forward simulation path over the return video display, helping us visualize our progress along the chosen path. Having saved some time, we then disengage the position clutch, and by trial and error movements position our manipulator in simulation to be at the right place to begin a complex maneuver. During this phase, the simulation-generated manipulator image moves on the display, but leaves no "smoketrail" 12

of a committed path. Upon reaching the correct position and orientation to begin the next maneuver, we reengage both clutches (the "smoketrail will now display the new interpolated path segment) and wait for the remote system to catch up. We then begin the maneuver. In this way we (i) save some time, (ii) use the time saved to later preposition for another action, (iii) avoid taking the actual remote system through complex, manipulatively unnecessary prepositioning movements, and (iv) do this all in a natural way through simple controls. Note that the following of paths generated during time-clutch disengagements can be done by crude methods such as simple interpolations while keeping movements slow enough to avoid robot rate limits. Or it could be done by sophisticated methods that take into account the full dynamics of the situation and drive the remote telerobot at nearly its maximum feasible rate along the path, given specific actuator limits and desired mean-error limits. This defines a large tradeoff space in the computational complexity of trajectory generation vs the time-performance and robustness of the resulting manipulation. 3.4 Time-Ratio Control So far we have implied a 1:1 ratio of forward simulation time to real time when a tele-autonomous system is operated in time-synchronized mode (time clutch engaged). This needn't be the case. For example, we might be able to operate the simulator much faster than the telerobot can follow, and wish to plan the path sequence in a synchronized, but scaled, time. So instead of using the time-clutch to disengage time synchrony, we might want to establish a low time-ratio between simulated and real time. But there also might be tasks that the telerobot can do far more rapidly than we could prescribe with the simulator. In those situations, if we had "saved up" some time, we could establish a high time-ratio of simulated time to real time and slowly perform a maneuver to be later done very rapidly by the telerobot (when it catches up to that section of the path). 13

These "time-ratio" scalings relating real-time to simulated time can be easily implemented and then controlled by allowing a change of time-ratio while the time-clutch is disengaged (analogous to changing the gear-ratio of a vehicle while the clutch is disengaged). The time-ratio then holds its new value until changed again during a later time-clutch disengagement. Time-ratio scaling should not be confused with operating while the time clutch is disengaged (where no fixed relationship is specified between simulator time to generate a path and telerobot time to follow the path). 3.5. Handling of contingencies by using Time Brakes What are we to do if we are forward simulating way out in front of the telemanipulator and suddenly see (in return video) something intrude into the planned path of the manipulator? To handle such simple contingencies, we introduce a mechanism we call a "Time Brake". Depression of the time brake disengages the clutches and "decelerates simulated time" by incrementally extracting (LIFO) previously generated position commands from the command buffer. The forward simulator is correspondingly moved in reverse back down the path. This allows the operator to move (as quickly as desired) back in time along the forward simulation path until located in space on the earlier side of the obstacle. We also provide an "emergency brake" that "immediately" empties the command buffer and halts the telemanipulator (subject, of course, to overshoots due to manipulator compliance and/or dynamic constraint management, and to races against 1/2 Tc). 3.6. Manipulation and cognition control-transitions and their mirrorings in telerobotic and supervised autonomous systems The control methods described in this paper enable simple and smooth handoffs from local human teleoperation control to and from remote machine manipulation control (using downloaded manipulation commands). But they also provide a base-level protocol that enables easy mechanizations of the other types of transitions from local-control by human or machine of cognition-or-manipulation, to local-or-remote machine control of 14

manipulation-or-cognition. Seen this way, the augmented teleoperation and autonomous systems mirror into one another to become tele-autonomous systems. Human or machine agents on "either side of the mirror" can exploit similar forward simulation and control handoff methods. We hypothesize that human operators of this technology can learn to accomplish graceful and efficient hand-offs, rendezvous and recaptures of real-time thinking and manipulation tasks, and that human-or-machine cognition-or-manipulation operators can also exploit the forward simulation constraint relaxations to improve performance in many situations. Humans could thus supervise, or be dynamically embedded into, complex human-machine task lattices, taking and releasing control'of subtasks at appropriate times and places. For example, the tele-autonomous technology provides a framework that enables us to mimic the aircraft student-instructor scenario, with either student or instructor being machine or human, each undertaking sequences of cognition and manipulation tasks. Consider for example the situation in Figure 3.2., where we see a telerobot, R, following a path specified by a forward simulation, S, that is proceeding with its time clutch disengaged. The operator of simulator S then disengages the position clutch and moves S down and to the right. S is now essentially disengaged from any connection with the telerobot. At the same time, some other operator (human or artificial) is maneuvering another simulator, S', down towards the forward planned path (S' is also operating with its position clutch disengaged). When S' gets "close enough" to the point where S left off path planning, S' can then engage its position clutch, and take over control of the telerobot (subject to acquisition intermediation by arbiter or collision detect mechanism at the telerobot). The interaction controls can be factored from the actual manipulations through a small, but important, time increment. The two "players" can formulate and interact using shared visualizations and rapid cueing methods, much as skilled sports players learn to do. 15

This simple example is suggestive of a number of more elaborate prototcols and scenarios that can be constructed on top of the low-level hand-off and rendezvous protocol. 4. INITIAL RESEARCH ISSUES AND HYPOTHESES Our initial approach to tele-autonomous system research is to form hypotheses concerning the overall human-machine system much as current human-computer interaction work models unassisted human perceptual, cognitive, and motor systems. We then test these ideas by experiment. For example, Fitt's law [CAR83] predicts that the time for the eye-mind-hand task of touching an object of linear size S at a distance D is given by T = Klog2(D/S + 0.5), where K ~100 msec./bit. Therefore, simple tasks based on varying the relative sizes of objects, and the distances between objects, might produce meaningful trials of the various modes of tele-autonomous operation. Could performance operate under some sort of scaled Fitt's law in some modes? Or is it more complex than that? We could find out, and perhaps develop some insights and principles on how to best design such systems. A simple 2-dimensional testbed can accomodate a wide range of such performance trials, such as the manipulation trial sketched in figure 4. 1. In that figure, we see a number of "switches" of linear size "S" located in sequence at known positions in the manipulator workspace. Each switch is distance "D" from its predecessor in the sequence. The objective is to touch each switch in the sequence as rapidly as possible. Increasing the ratio D/S corresponds to increasing task manipulation complexity, possibly requiring more time for manipulation convergence, as abstracted in Fitt's law. We can explore and answer many questions using this simple testbed. What is the functional form of the reduction in manipulation time, over direct teleoperation, that can be obtained when using the different augmentations of control? How are these functions and times affected in the presence of communications delays? How are the times affected by the difficulty of the manipulation targeting task (larger mean values of D/S). What are the 16

effects of other system parameters, such as joystick force constants and robot velocity limits? What determines the percentage of the task execution time that the system operator needn't be in the control loop, so that they can be available for performing other functions? The quantitative results of such trials can yield important early measures of the forms and dimensions of performance improvements possible with the tele-autonomous controls. The results can then help guide planning of further trials and the exploratory evolution of the technology. 5. EXPERIMENTAL ENVIRONMENT AND RESULTS We are building a general experimental environment in which to create and evolve tele-autonomous technology. In this section we describe our initial facility and some early experiments with the different control methods. These early experiments are being done on transitions viewed from the telerobotic point of view, and provide the basis for planning later experiments in which we will study transitions from the various mixed telerobotic/autonomous systems points of view. Our initial facility consists of a Unimation PUMA 560 used as a telerobot, controlled by either a DEC VAX 1 1/750 computer or an Apollo DSP90 computer (both modes are available). A force and moment sensitive joystick is used as an input device to provide a rate input for the telerobot. To simulate a variety of real robots typical of those used in space or undersea operations, appropriate velocity limits are placed on each joint. To simulate remoteness of the telerobot from the operator, a variable delay can be inserted between a forward simulator generated control stream and the PUMA, using a buffer to hold the trajectory sample stream. A high-performance Silicon Graphics IRIS workstation is used to generate and mix the display of the forward simulator and the telerobot, with the telerobot seen either in return video or as a graphics model (the latter can be useful since it is both easier to obtain 17

correspondence between the overlay and the simulation and easier to modify the viewpoint of the system). Simple time and position clutches have been implemented in the system. The logical operation of the clutches is portrayed as a state diagram in Figure 5.1., which shows the allowable combinations of time and/or position synchrony and the transitions between them. There are three allowable control states from the teleoperation point of view: 1) TSyn & PSyn -- time and position synchrony, 2) PSyn -- position synchronism only, and 3) NoSyn -- both time and position synchrony disengaged. Changes between these states are controlled by switches that operators push with their feet -- similar to the clutch in an automobile. The joystick moves the simulator in all three states, but the state determines the effect of simulator movement on path planning and path buffer encoding. A foot operated time brake is implemented which when pressed (i) disengages both clutches, causing an overriding transition to NoSyn, and (ii) begins deleting entries in the command buffer (LIFO), thus running the forward simulator back down the previously generated path. In the time and position synchronized state (TSyn & PSyn), the force and moment outputs of the joystick are sampled at the input rate required by the PUMA. The forces and moments obtained are treated as vectors of desired velocities in Cartesian space. These are integrated to obtain position samples, and these samples are placed in the command buffer. While in time synchronized mode the buffer is emptied at the same rate it is filled, and the values obtained from the buffer are input to the PUMA system, which treats each sample as a goal to reach in its sample period, by slewing any or all of its six joints. When we wish to simulate situations where the actions can be visualized and simulated much faster than they can be manipulated, such as when moving large structures in space [NAS81] or underwater, we place a selected angular velocity limit Wj(i) < Wjmax(i) on each axis, i, of the PUMA. 18

When the time clutch is disengaged, and the position syncronized (PSyn) state is entered, the robot-model physical constraints on simulation distance covered per timesample are removed, and the usual joystick force and torque constants are multiplied by a gain constant, Gs, enabling the operator to rapidly slew the simulator. Path samples may be generated at varyingly wider path intervals than is possible when control is synchronized in time. Values removed from the buffer may thus request incremental moves larger than can be accomplished in one PUMA sample period, given selected constraints on the angular velocities of the PUMA joints. When this occurs, the commanded move is interpolated and spread across more than one PUMA sampling interval, with the actual PUMA rate of motion constrained as above by the selected joint angular velocity constraints. Thus, the simulated telerobot can be moved out along a path well ahead of the real telerobot, and we can "save up some time". In addition, the real robot follows this path at nearly its maximum rate, for a given set of values of Wjmax (i). Thus we predict that the overall manipulation time, Tm, will usually be smaller using this mode than if there were a fixed ratio between simulation time and real-time (as for example in time-ratio control). If a transition is made from PSyn to the unsynchronized (NoSyn) state, the operator becomes free to move the simulated telerobot without values being placed in the command buffer. Then, when a transition is made back to PSyn, the current position of the simulated telerobot is placed in the buffer. When this value is extracted from the buffer, the telerobot makes interpolated incremental moves directly towards that desired position, without going through all of the motions the operator had to use to get to that position. This enables "saved up" time to be used to "edit" out some real time and path motions used, for example, for complex prepositioning. 5.1. Experimental Parameters,Trials and Results In our first trials we used simple, random, 2-dimensional, 5-switch testbeds similar to that in Figure 4.2. We conducted a series of trials varying the following parameters: 19

(i) Three different subjects (X, Y, Z) each performed a series of manipulation tasks using the testbed. Two times were recorded for each trial: The subject's time to specify the manipulation, (Ts), and the system's time to complete the manipulation, (Tm). We also recorded the actual manipulation path length, Lm. The ratio of Lm to the minimum path (- 5D) provides a measure of one dimension of operator skill). (ii) The series of switch-touching tasks varied from simple to difficult by ranging from low values of D/S to high values of D/S (D = 500 mm.; S = 25, 50, 75, 100 mm.). (iii) Communication delays, Tc, of 0, 2 and 4 sec. were used. (iv) Tasks over the range of difficulty and the range of communication delays were performed by each subject using: (a) direct teleoperation (TOP), (b) teleoperation assisted by forward simulation (TOP+FS), and (c) teleoperation assisted by forward simulation and time clutching (TOP+FS+TC). During these first trials, other key system parameters were held constant as follows: (i) Workspace to monitor-screen length-ratio = 8:1. (ii) Joystick sample period = 0.017 sec. (iii) Joystick force constant = 0.01 mm per oz per sample period = 0.6 mm per sec. per OZ. (iv) Joystick torque constant = 0.0012 rad. per oz-in per sec. (v) Joystick gain constant, Gs, in (TOP+FS+TC) = 4.0. (vi) Angular velocities of all 6 PUMA joints were limited to Wj < Wjmax = 0.5 radians per sec. (but see also below). Other comments on our methods: The chosen constant values yield moderately responsive controls when moderate joystick forces and torques are applied. The angular velocity limits yield a moderately fast robot (slower than the PUMA can go at its fastest, but very, very much faster than a scaled shuttle arm). All subjects engaged in preliminary learning trials. All used the joystick "one-handed". Trials began after a period of 20

preliminary learning. Comparable power-law of practice performance levels [CAR83] were recorded for each mode. Results of some of these initial trials for one subject are plotted in Figures 5.2, which shows the specification time (Ts) and manipulation times (Tm) for tasks over the range of D/S difficulty holding D = 500mm. Included are results for communication delays, Tc, of 0.0, 2.0 and 4.0 seconds. The results are displayed for the three relevant modalities of control, (a) TOP, (b) (TOP+FS), and (c) (TOP+FS+TC). We note that a comparison of TOP and (TOP+FS) repeats experiments of Sheridan, et. al. [HAS86, SHE86], confirming the results of that work. We see that (TOP+FS) gives a significant gain in both Ts and Tm over TOP alone. Then we notice that (TOP+FS+TC) gives another significant gain in Ts over (TOP+FS). In the initial trials, we found that Wjmax = 1.0 rad./sec. was high enough for the robot's Tm time to keep up with even the shortest (TOP+FS+TC) Ts times (see Fig. 5.2). We then found that Wjmax = 0.5 rad./sec. constrained Tm so that subjects could easily outpace the robot and save up time (see Fig. 5.2). Many of the initially hypothesized forms of results were demonstrated using these parameter ranges. We then noticed that Ts and Tm grew less rapidly in D/S than anticipated. We hypothesized that D = 500mm was large enough, given the joystick constants and Wj values, to produce dynamic constraints related to D and not just D/S. So we repeated scaled versions of thesetrials at smaller values of D. Figure 5.3 shows the results for D = 250 mm and S = 50, 37.5, 25 and 12.5 mm. (with work to screen scale = 16:1, and Wjmax = 0.5). It also includes results using D = 125mm and S = 25, 18.7, 12.5 and 6.2 mm (with work to screen scale = 32:1, and Wjmax = 0.5). These results are interesting, because for all three modes the data per mode at D = 250mm and D = 125 mm essentially fall on top of one another. The 250mm and 125mm curves for each mode lie well below those for D = 500mm. Refer Fig. 5.2 for the timeclutch mode data for D = 500mm (it would partly overly the Fig. 5.3 time-clutch data). 21

At this scale the system operates in a "Fitt's law-like" region, with Ts and Tm being functions of D/S (but not D), with the values in most cases at D/S = 20 about twice those at D/S = 5. For Wjmax = 0.5, the robot's Tm at this scale could stay up with the subjects Ts. We varied Wjmax and found values of 0.35 (for D = 250) and 0.25 (for D = 125) that yielded demos of significant time differences between Tm and Ts for (TOP+FS+TC) mode on the easier tasks (see Fig. 5.3). On further scaling-down of D, the system enters its "Heisenberg" region on the harder tasks (S < 2 to 3mm). Position sample-sizes, interpolator discretization and operator jitters cause large increases and variances in Tm and Ts (like trying to poke at things with a needle under a microscope). Throughout the trials, subjects noticed striking differences in the "feel" of the different control modes, and developed special tactics for coping with each mode. Most treated TOP in the presence of delays like hitting a series of "successively shorter golf shots", trying to get closer each time. Subjects controlled (TOP+FS) aggressively, firmly driving the simulator to each switch. The (TOP+FS+TC) mode was usually handled with finesse, so as to drive it fast, but not so fast as to yield a wild path and thus large Tm and large Lm/5D. In addition to these preliminary quantitative results, we have demonstrated the use of the position clutch to enable graceful handoffs of control by one agent and rendevous of control by another agent. This is done by simply having two human operators swap use of the controls following disengagement of the position clutch once the forward simulation is out well ahead of the telerobot. We have compiled a video report showing the above experiments, demonstrations and control effects [CON87]. 5.2. Plans for further experiments and concept demonstrations We are continuing the above series of trials, varying additional system parameters. We are also preparing additional types of experiments and demos. The time brake and 22

position clutch will be used to determine their effects on specification and manipulation times, and manipulation path lengths. (Skilled operators can use the time brake to "erase" poor path sections, and the position clutch to make and "jump across" gaps during overshoots). The time brake will be tested in contingencies (example: an obstacle falls across the planned path behind the forward simulator). Transitions involving cognitive/manipulative task-nesting will be explored. We will add the command buffer and interpolator modifications,and HCI controls, to implement and demonstrate time-ratio control during time-synchronized forward simulation. We also plan to attempt demos of simple forms of role-reversal by having the teleautomaton do the path planning, and letting the human rendezvous to telemanipulate along selected portions of the projected path. In the role reversal demo a route planner uses AI techniques to plan a path through a maze. The planner then places the path into the forward simulator and, when necessary, calls upon the human to take over and manipulate through certain path segments. The human then just drives the manipulator along the displayed path segment. This mimics a human taking over the driving of an ALV while the ALV remains under machine cognitive control. This environment will also enable demos of human intervention in cognitive tasks, for example to assist in planning the route if the machine gets stuck in that high-level planning task. We found basic principles such as Fitt's law very useful in thinking about forms of testbeds and hypotheses for our early trials. We need to consider additional system parameters and also the dynamics of the human/machine combination, generate further hypotheses regarding factors affecting performance, and then design experiments to test these ideas. Such work may eventually produce principles and design rules for teleautonomous manipulation systems. 23

6. FUTURE RESEARCH CHALLENGES This initial tele-automation work suggests opportunities for coordination of research in several specialized fields. It also raises issues concerning research equipment infrastructure. Tele-autonomous technology presents new challenges in human computer interaction. We have proposed a set of interface controls that are conceptually simple and easy to mechanize. The controls are generic ones that may be applicable in many different specialized situations. They are also cognitively and manipulatively accessible to the uninitiated by analogy. But many other new human interface aspects haven't been pinned down at all. How is the operator to visualize where they are, who has control of what, and who they give control to next as they enter or leave some subtask within a complex task lattice? What measures can we provide concerning operator performance, and what feedback can we provide? And what about the analysis and design of cognitive and manipulation tasks themselves? Research can perhaps provide better measures of joint human-machine cognitive-manipulative performance. Analyses similar to those in [CAR83] may then lead us to design intermixings of human and machine activity that yield substantial improvements in overall performance. Research challenges arise in robotics, such as the eventual need to perceive, model and forward simulate not only the remote tele-automaton, but also portions of the remote environment itself. Forward simulation will work fine when interacting with static objects, but what about interactions with moving objects? Even if we knew how to specify interactions with moving objects, such work would be severely constrained by the high computational complexity of present methods for representing and simulating mechanical systems. Further basic work, such as that of Hopcroft, on efficient representation and simulation of mechanical systems is required if we are to handle problems of really interesting complexity [HOP87]. 24

More work is needed on methods for path-error specification and associated methods for the time optimization of path following, such as in [SUH87]. Additional work is also needed on autonomous "reflex" actions that the remote robot can perform when encountering uncertainties (particularly those involving contact) not modelled in the forward simulation. We also need augmented AI programming environments that interface in such a way with real-time programming environments as to easily enable rapid estimation of time available for short-term Al planning tasks (enabling us to select among AI methods as a function of available time). We believe that fundamental work can be done in these areas with modest robotic laboratory equipment. AI techniques [WIN84] and expert system technology [HAY83] have matured so that roboticists can now mechanize knowledge-intensive cognitive functions well beyond their reach just a few years ago, and can run these systems on accessible workstations. Thus mixings of manipulation and cognition technologies are now ripe for research exploration. However, some experiments will benefit from multi-dimensional teleoperators or high-tech automation or autonomous system technology. One way to gain access to such expensive equipment is to treat remoteness as a feature: For example, we are negotiating connection of our tele-autonomous control equipment via satellite links with automation systems at several remote sites. As such efforts provide useful testbeds, others might exploit shared access to the same remote facilities. Shared access to capital equipment has obvious costs benefits, but in addition can stimulate standards, collaborations and healthy direct competitions among researchers. Shared access to silicon foundries greatly increased the productivity of the VLSI research community [CON81]; a remote tele-automation facility could play an analogous role in tele-autonomous systems research. 25

7. SUMMARY We have introduced basic functional concepts for tele-autonomous technology and an architectural framework for implementing the technology, using controls over time and position synchrony that enable simple structuring of control transitions. We have proposed hypotheses concerning capabilities of the technology, described our environment for investigating these phenomena, and discussed the results of early tests of some of the hypotheses. The results indicate how telerobotics can be extended to include projection of cognitive activity and autonomous systems extended to accommodate smooth transitions of cognitive or manipulative responsibility between machine and human operator. Through such extensions, the two technologies merge into one of "tele-autonomous systems". Finally, we have also sketched some further lines of research suggested by this initial work. We believe that tele-autonomous systems research can yield methods and systems for improved projection of intelligent, manipulative action at a distance in time and space. This interdiscipline presents interesting new research opportunities to teams having expertise in robotics and automation, artificial intelligence, and the psychology of humancomputer interaction. We envision many possible applications for the resulting technology, not only in space and defense systems, but also in design and production systems, and eventually in personal and recreational environments. 8. ACKNOWLEDGEMENTS We gratefully acknowledge support of the Center for Machine Intelligence and the College of Engineering of the University of Michigan, the support provided by the grant of a high performance IRIS graphics workstation from Silicon Graphics, Inc., and the contributions of Lee Hagerhorst and Lejun Shao to the rapid prototyping of our first experimental systems. 26

9. REFERENCES [CAR83] Card, S. K., Moran, T. P. and Newell, A., The Psychology of HumanComputer Interaction, Lawrence Elbaum Assoc., Hillsdale, NJ, 1983. [CON81] Conway, L., "The MPC Adventures: Experiences with the Generation of VLSI Design and Implementation Methodologies", Second Caltech Conference on VLSI, California Institute of Technology, Jan. 1981. [CON87] Conway, L., Volz, R. and Walker, M., "New Concepts in TeleAutonomous Systems," Univ. of Michigan Robotics Laboratory Video-Report, Feb. 1987. [DAR83] DARPA, Strategic Computing: New-Generation Computing Technology - A Strategic Plan for its Development and Application to Critical Problems in Defense, Defense Advanced Research Projects Agency, Arlington, VA, October 28, 1983. [DAV85] Davis, D. B., "Assessing the Strategic Computing Initiative", High Technology, April 1985. [HAS86] Hashimoto, T. and Sheridan, T. B., Paper to be published in the Japanese Journal of Ergonomics. [HAY83] Hayes-Roth, F., Waterman, D. W., Lenat, D. B., ed., Building Expert Systems, Addison-Wesley, Reading, MA, 1983. 27

[HOP87] Hopcroft, J. and Krafft. D., "The Challenge of Robotics for Computer Science." In Advances in Robotics, Vol. 1, Algorithmic & Geometric Aspects of Robotics. C. Yap and J. Schwartz, ed., Lawrence Erlbaum Associates, Hillsdale, NJ, 1987. [MAR86] Martin Marietta Corporation, "The Autonomous Land Vehicle 2nd Quarterly Review", DARPA ALV Workshop, June 23-27, 1986. [NAS81] NASA, "Shuttle Flight Operations Manual, Payload Deployment and Retrieval Systems," Vol. 16; Flight Operations Directorate, Johnson Space Center, June 1, 1981. [NOY84] Noyes, M. and Sheridan, T. B., "A Novel Predictor for Telemanipulation through a Time Delay", Proc. of the Annual Conference on Manual Control, NASA Ames Research Center, Moffett Field, CA, 1984. [RTI82] RTI, RTI Force Sensing Wrist User's Manual, Robot Technology, Inc., Los Altos, CA, 1982. [SHE86] Sheridan, T. B., "Human Supervisory Control of Robot Systems", Proc. of the IEEE International Robotics Conference, April 1986, pp. 808-812. [SUH87] Suh, S. H. and Bishop, A. B., "Tube Concept and Its Application to the Obstacle Avoidance Minimum-Time Trajectory Planning Problem," Univ. of Michigan Robotics Laboratory paper submitted to the IEEE Journal ofRobotics and Automation. 28

[VER86] Vertut, J. and Coiffet, P., "Teleoperations and Robotics: Applications and Technology," Robot Technology, Vol. 3B, English Trans., Prentice-Hall, 1986. [WIN84] Winston, P. H., Artificial Intelligence, 2nd Ed., Addison-Wesley, Reading, MA, 1984. 29

3ee l 2R move monitor Forwar~ Simul. Contr Local Localw By Sti ctelerobo joystick Contro Figure 2.1 Using forward simulation and predictor display to cope with time delay (after Noyes and Sheridan).

Manipulation time, Tm (sec.) without forw. with forw. sim. 0.0 1.6 3.2 6.4 Communication time, Tc sec.) Figure 2.2: Manipulation time as a function of time delay, Tc. [HAS86, NOY84, SHE86]

X'~ " "~ - ~~move.~~~~~~~~~ j monitor Forward _ RemotJ pulff 1 p u I kt- Simulati / 1-ontrol teleC & Local Ic~/'J joystick _ telerobot ~~~~~. - - %-=.=..~ -................ A_ ~~~~~~~~~~~~~~...j i. rai.........tr'.....rai i:-'.....l time clutch...................".'.........lutch. posn. clutc /]........................................ time be Figure 3.1: Using time and position clutches, time ratio and time brake to control forward simulation path planning in a tele-autonomous system.

S', PC diseng. S, TC diseng. R S, PC disen = path of actual robot (R) = path planned by forward simulatior (S) Figure 3.2: Using time and position clutches to "hand-off" and to "rendezvous" with a manipulation task.

D2 - - Figure 4.1: Simple testbed and example manipulation trial.

TSe/ngage TC & Py disengage TC Pyn PSyn ger~ge PC en C wi TC engwhile C is PC dis PC\ /while TC dis NoSyni eng T dis TC Figure 5.1: Diagram of forward simulator states and the transitions caused by time and position clutches.

Tm, sec 0 TOP D = 500mm Ts, sec a TOP+FS & TOP+FS+TC 120 100:Tc = 2 80 60 - o3:Tc=O 40: 20 =T 2 Wjmax= 1.0, 0.5: Tc=0, 2,4 oI! I 0 5 6.7 10 20 Task difficulty ratio, D/S Figure 5.2: Initial trial results, showing Ts, Tm as functions of system and task parameters for three modes of control.

For Tc = 2 sec: TOP Tm, sec ~ TOP+FS Ts, sec - TOP+FS+TC o' D =500mm | 80/~ o ~D = 250mm 70 o D =125mm o D= 500 50 4 F=1 — 40 I0 |.-a: D =125, 250 30,-~.. I A'D = 125, 250 20 10 Wjmax = 0.5; = 0.25 @ 125, 0.35 @ 250 0 01 _ I,....... 0 5 6.7 10 20 Task difficulty ratio, D/S Figure 5.3: Trials showing Ts, Tm as functions of system and task parameters, for several values of task size scale D.

UNIVERSITY OF MICHIGAN IlllllII II lllllllllllllllll IIIIII IIIIIIIII 3 9015 02653 6022