An Overview of Tactile Sensing by Rajeev Agrawal and Ramesh Jain Robotics Research Laboratory Department of Electrical Engineering and Computer Science The University of Michigan. Ann Arbor, Michigan 48109 July 1986 T ) l\ U.IN EPSITY OF MiSi\'",.'.~ CENTER FOR RESEARCH ON INTEGRATED MANUFACTURING Robot Systems Division COLLEGE OF ENGINEERING THE UNIVERSITY OF MICHIGAN ANN ARBOR, MICHIGAN 48109-1109 lThis work was supported by NASA Ames under Contract No. NASA-G-NAG 2-350

RSD-TR11-46 TABLE OF CONTENTS 1. Introduction 1 2. Tactile Sensors-Some General Considerations 3 2.1 Performance Criteria........................................ 5 3. Some Typical Current or Proposed Tactile Sensors 7 3.1 Remarks...................................................................................... 20 4. Recognition Using Tactile Sensing 21 4.1 Recognition Schemes Employing Only Tactile Sensing.............. 23 4.2 integrating Tactile Sensing With Vision................................... 26 4.3 Comments.................................................................................. 28

RSD-TR-11-88 1 Introduction Tactile sensing, as the name suggests, pertains to perception by'touch', i.e. the sensing of certain qualities by a direct contact between the'object' and the "subject.' These qualities might include: shape, pressure distribution, texture, temperature, thermal conductivity, moisture, slippage, and vibrations. In general, any feature which requires direct contact with the object could be called a tactile feature. The "subject' of interest here is a robotic system. Although the features mentioned above are drawn mainly from human tactile sensing, they could easily be extended depending on the capabilities and needs of the system under consideration. Harmon in [16J,[171 makes a distinction between'simple touch' and'tactile sensing.' He defines tactile sensing as the continuous variable sensing of forces in an array, as opposed to simple touch, which is defined as force sensing at a single point or binary( on/off ) sensing at multiple points. Note that Harmon refers to force sensing only, which besides being the most important tactile feature is also reflective of the current state-of-the-art. To see how and where tactile sensing fits in a robotic system, consider the model shown in Fig.[lJ. The robot gains information about the external world through a set of sensors and a preprocessing unit. Following this, the computer with the help of the knowledge base ( which it may also update occasionally ), directs the manipulators to perform certain actions on the external world. The decision making process of the computer may be fully programmed ( fully automated system ) or may require human interaction from time to time ( semi-automated system ). Given the above set-up, tactile sensors can form an integral part of the sensing requirements of a robotic system, especially if these sensors are placed on the manipulators which would come in contact with the objects. A distinguishing characteristic between different robotic systems is whether or Tactile Sensing 1

RSD-TR-11-86 not they perform any kind of object recognition. Typically, the issues addressed in a task involving only simple manipulations of an object will be: grasping the object properly and maneouvering it around in a stable fashion. These would emphasise factors of position, orientation, force distribution, slippage etc. This is quite different from the requirements of a recognition task which should typically be independent of the above factors. It is possible to combine these two modes together to make a more powerful system. For instance, object recognition may be necessary prior to proper manipulation of the object. A knowledge of the object helps to determine the appropriate grip points. On the other hand, one might want to "feel" around the object in order to perform the recognition task. Most of the current thrust for the development of tactile sensors is coming from the robotics area where object manipulation is the main concern. Another area of application, which is providing impetus to the development of tactile sensors, is prosthetics. The development of these sensors makes available an entirely new type of information which can be used to supplement and/or complement'vision' or any other type of object recognition sensors. Our objective in this paper is to review some of the existing or proposed tactile sensors and to give some typical recognition systems that have attempted to use tactile sensing. This paper does not describe the aspects related to grasping, etc., which are not very relevant to a recognition system ( Refer to [3], [13], [251 for some issues related to grasping). First, general considerations involved in tactile sensing and various performance criteria are discussed. Typical specifications to be expected from these sensors are also described. Thereafter, a representative set of present day tactile sensors is studied. Finally, some of the proposed recognition systems using tactile sensing are described. The last part is basically meant to give the reader a familiarity with these 2 Tactile Sensing

RSD-TR-11-86 systems and may be used as a guideline in their design. Very little has been done till now to effectively combine tactile sensing with other sources of information, like vision. Increasing research interest in this area should yield important results. 2 Tactile Sensors - Some General Considerations This section discusses the issues that are important in the design and evaluation of a tactile sensor and of systems employing tactile sensing. There is no real order since these issues are disconnected, and the following material is just an enumeration of such factors. * Location: A tactile sensor has to come in direct contact with the object. The obvious place to locate tactile sensors in a robot is the manipulators since they come in contact with the object, or could be easily maneouvered to do so. One might make an extension from the human skin, and place such sensors elsewhere on the robot but it is clear that sensors will be of maximum use on the manipulators. * Sensor Array: Considerations of implementation and the need for computer processing of the output data lead to the discretization of this 2-D sensing into a kind of array format. The proper density of the individual tactile elements ( taxels ) in the array will be determined by the Nyquist rate of sampling, i.e. according to the bandwidth of the 2-D spatial Fourier transform. In most cases, however, technology determines the spatial resolution which in turn then limits the kind of tactile images that the sensor can properly sense. ~Preprocessing and Communication: Before the main recognition process is carried out on the data available from these tactile sensors, a certain Tactile Sensing 3

RSD-TR-11-86 systems and may be used as a guideline in their design. Very little has been done till now to effectively combine tactile sensing with other sources of information, like vision. Increasing research interest in this area should yield important results. 2 Tactile Sensors - Some General Considerations This section discusses the issues that are important in the design and evaluation of a tactile sensor and of systems employing tactile sensing. There is no real order since these issues are disconnected, and the following material is just an enumeration of such factors. * Location: A tactile sensor has to come in direct contact with the object. The obvious place to locate tactile sensors in a robot is the manipulators since they come in contact with the object, or could be easily maneouvered to do so. One might make an extension from the human skin, and place such sensors elsewhere on the robot but it is clear that sensors will be of maximum use on the manipulators. * Sensor Array: Considerations of implementation and the need for computer processing of the output data lead to the discretization of this 2-D sensing into a kind of array format. The proper density of the individual tactile elements ( taxels ) in the array will be determined by the Nyquist rate of sampling, i.e. according to the bandwidth of the 2-D spatial Fourier transform. In most cases, however, technology determines the spatial resolution which in turn then limits the kind of tactile images that the sensor can properly sense. ~Preprocessing and Communication: Before the main recognition process is carried out on the data available from these tactile sensors, a certain Tactile Sensing 3

RSD-TR-i 1-86 the deformation of the object itself. This will clearly affect the tactile data coming in, and also might reflect a desirable/undesirable state of affairs in terms of object handling. The pressure ( and thus deformation ) distribution on the elastic layer reflects information both about the shape and hardness of the object surface. The skin stiffness may be used to obtain a desirable combination of the two. The dominance of the shape information can be obtained by using a'soft' skin. Note this entire argument is based on the relative hardness of the object and the elastic layer. 2.1 Performance Criteria Before describing some actual sensors, it is important to establish common ground for the basis of evaluation and comparison of these sensors. Presented in this section are significant criteria for tactile sensors. It is also noteworthy that some of these criteria are related and interdependent. Often there is a trade-off between them. Unfortunately, most of the sensors which have been described have not been extensively evaluated. There is no standard set of well defined criteria which permit an easy comparison of tactile sensors. However as these products proliferate and become more available commercially, their specification sheets would reflect such standards. Note that some of the factors listed here might be specific to pressure sensing. * sensitivity * dynamic range ~ linearity/non-linearity ~ drift Tactile Sensing 5

RSD-TR-11-86 * signal/noise ratio * susceptibility to external inference * response time/recovery time * spatial resolution and sise * spatial response * measurement speed * defect tolerance * reliability * robustness * overall size and weight * preprocessing/communication capability * elastic properties, hysteresis, etc. These criteria for performance evaluation are essentially common to any transducer and for further detais any book on measurement or transducers such as [11] may be referred to. Harmon [16J,[17J did an extensive survey of the needs and requirements of tactile sensing both with regards to research and industrial applications. A brief summary of his'ideal' tactile sensor is included in the table of sensor specifications. 6 Tactile Sensing

RSD-TR-11-88 3 Some Typical Current or Proposed Tactile Sensors Described in this section is current or proposed tactile sensors. This is a representative list showing a variety of transduction principles and bringing out a number of important characteristics, i.e. preprocessing, composite sensing, etc. However, it is not meant to be an exhaustive survey. Some performance measures are also given for each of these sensors. Since a standardized list of criteria does not exist and most of these sensors are currently being researched, an authentic comparison is impossible. Some qualitative remarks are made about the relative weak and strong points. This is presented in the form of a table. 1. Capacitive Impedance sensor: ( Boie [81) Fig.[2J illustrates an exploded view of a sample robotic touch sensor developed by Boie. The topmost layer is a compliant glove that contacts objects and transmits via its elastic constant the contacting force distribution to the elastic/dielectric layer below. The lower layer is rigidly supported by the printed circuit board. The glove and dielectric layer can be viewed as two springs in series under compression where the force information is obtained by measuring the displacement of the dielectric spring. The mechanical point-spread function of the glove can be narrowed, if desired, by suitably segmenting the glove material. Orthogonal sets of conductive strips are arranged on the upper and lower surfaces of the elastic layer. A sampling of the layer thickness map is obtained by measuring the array of capacitors formed by the crossing areas, A;,, of row and column strips. The strip widths and spacing along with any point Tactile Sensing 7

RSD-TR-11-86 force spreading in the structure determine the spatial sampling and resolution. The time required to measure all capacitors determines the temporal sampling. The radio frequency source, Vocos(wt), is connected to the lower set of strips via analog multiplexer Aid. The multiplexer Ij' connects pads to the amplifier input node. The pads are capacitively coupled to the upper strips via an inactive region of the elastic/dielectric layer. This contactless arrangement is an important feature of this method. Cross talk signals are reduced by connecting the unselected strips and pads to ground potential. For each pair of multiplexer addresses (i, j) the r.f. source voltage is connected through the capacitance C(i, j) of strip i to strip j to the input node of the amplifier. (The strip to pad capacitance is arranged to be sufficiently large.) The output signal of the amplifier, VA(i,j,t), is related to the strip to strip capacitance by, VA(i, j,t) = -VD C( cos(wt) where CA is the capacitance in feedback. C(i,j) is related to the localized layer thickness change by, CKA c(,X j) f=.(d - z(i,i)) where A is the strips crossing area, K is the relative dielectric constant, e0 is the permitivity of vacuum, d. is the unloaded layer thickness. The local sampled force is described by the relationship, F(i, j) = Az(i, j) where A is the dielectric/elastic layer spring constant. The applied force is linearly related to measures of the reciprocal crossing capacitances with a constraint of fixed layer constants. Each crossing capacitance, independent of the shunt dielectric loss and series switch resistances, is 8 Tactile Sensing

RSD-TR- 1-86 measured in turn by phase sensitive detection during the interval T., between sequential address advances. Cross layer capacitive impedance sensing is a favorable choice in many re spects. The elastic materials need not be modified. Desirable mechanical properties are generally consistent with low dielectric loss. The method has the virtues of a parametric measurement, that is, the output signal is proportional to the displacement times the drive signal. Capacitors are non-dissipative elements and so generate no noise. Thus, capacitive sensing has demonstrated to have marked advantage in terms of signal to noise ratio and measurement speed. However, capacitive sensing has not fared well in robotics literature to date, where it is described as inappropriate because of noise ( Refer to 1171 ). This misconception has resulted from confusing man-made interference, which can be reduced to negligible levels by proper shielding and connection, and with intrinsic noise related to the basic nature of the detection process. Boie [8J discusses some of the dependence and trade off involved between noise, resolution and dynamic range. 2. A Capacitive Silicon Tactile Imaging Array: (Chun & Wise [9J) This tactile sensor was developed at the University of Michigan. Figure[3J shows the cross-section and the layout of the proposed capacitive tactile imaging array. The basic cell is formed between a selectively-etched, boron-doped thin silicon diaphragm, which moves in response to applied force, and a metalized pattern on an opposing glass substrate to which the silicon substrate is electrostatically bonded. Silicon dioxide is used to isolate the transducer's plates on the silicon from the substrate and to let them function as isolated row lines. hn the layout, row conductors are run vertically across the siliTactile Sensing 9

RSD-TR- 1-86 con wafer in slots which are simple extensions of the capacitive gap recess. Metal column lines run horizontally on the glass under recesses in the silicon, expanding to form capacitor plates over the cell areas. Thus, a simple X-Y capacitive keyboard is formed which has precisely controlled dimensions and a force sensitivity set by the thickness of the silicon plate. A perforated cover plate is overlayed on the array for protection. This plate is in turn covered by a compliant, replaceable pad and outer skin. If the compliance is high this pad can be slit to decrease blooming of applied local force. The access holes coupling force to the cell are filled with a substance ( such as silastic ) which acts as a force transmitter. Effort has been made here to minimize the importance of the pad on determining the performance of the overall array, since such pads are known to be the performance-limiting factor in most reported designs. The dominant structure in determining overall force sensitivity is the silicon diaphragm, whose properties can be well controlled, easily scaled for various applications, and are known to be stable over time and free of hysteresis. While pad designs vary widely for applications which range from enveloping grasp to surface texture measurement, the important feature of this basic cell is that the pad is used only for force transmission and plays a relatively minor mechanical role. The array is read out using a switched capacitor charge integrator giving a resolution of more than 8 bits. In order to simplify the sensor design and fabrication, the readout scheme makes use of off-chip electronics to take care of all multiplexing/ preprocessing. 10 Tactile Sensing

RSD-TR-11-86 3. VLSI Tactile Sensing Computer: (Raibert & Tanner [24] ) This tactile sensor makes use of the VLSI technology to combine transduction, tactile image processing, and communication; thus creating a very effective intermarriage between transducer and preprocessing requirements. Fig. [41 illustrates the physical structure of the proposed tactile sensor. A sheet of conductive plastic is placed in surface-to-surface contact with a custom-designed VLSI circuit. Large metal electrodes are formed on the surface of the integrated circuit by patterning the circuit's superficial metalisation layer. Analog and digital computing elements are present within the structure. Windows in the Moverglass', an insulating layer of SiO2, normally placed on integrated circuits for protection, allow selective contact between plastic and circuit. Where windows are present, current flows between the sensing electrodes and the conductive plastic. Where no windows are present, the overglass insulates the integrated circuitry from the conductive plastic. Tactile forces are transduced by measuring changes in sheet resistivity of the conductive plastic as it is deformed by pressure. To measure the magnitude of deformation and pressure at each point, small test currents are passed from each pair of electrodes through a local region of conductive plastic. The microelectronics within the device form an array of elements that performs a number of sensing and computing operations within the tactile sensor. Once signals proportional to force are generated, circuitry within each cell digitizes and performs computations upon the resulting data. Filtering and simple feature extraction operations are implemented with these computations. Because each tactile cell transduces its own data and a separate computing element is provided for each tactile cell, parallel processing is implemented in a natural way. Once data are processed the results must be made available Tactile Sensing 11

RSD-TR-11-86 to a host computer or controller. This is done by shifting data from tactile cell to tactile cell until all data has reached the periphery of the array. There they are serialized for transmission to the host. For further details of the computing structure refer to [24]. Some comments: * Speed: Considering the margin between typical VLSI systems which operate around 1 is and manipulator controllers which operate in the 5-20 ms range, speed was sacrificed for circuit area thus leading to serialization of many functions. Note once again that the design was primarily intended for manipulation, and if the real time recognition task imposes more severe requirements this point may have to be reconsidered. * Defect tolerance: The need for large arrays would require that large portions of the wafer be defect free, that is, no defective cell can be present if the overall array is to operate correctly. However, with the present state of art in VLSI this is a severe requirement. An alternative to this is fabricating sensors with redundancy. A redundancy of one allows up tQ 1000 elements per array within a reasonable probability ( assuming a Poisson distribution- see [24] ). However, this technique cannot be extended beyond one redundancy because of the increase in complexity. Another alternative is to use algorithms which can tolerate defects. This is viable only if defects in one cell do not block transmission of data from other working cells ( stripe defects ). Solutions to this are again discussed in [24]. 12 Tactile Sensing

RSD-TR-11-86 4. A high resolution sensor: ( W. Daniel Hillis [24 ) The tactile array sensor ( Fig.1[5] ) developed by Hills basically consists of two conductive components: a flexible printed circuit board and a sheet of anistropically conductive silicone rubber ( ACS ). The ACS has the peculiar property of being electrically conductive along only one axis in the plane of the sheet. The printed circuit board is etched into fine parallel lines, so it too conducts in only one dimension. The two components are placed into contact with lines on the printed circuit board perpendicular to the ACS axis of conduction. (The ACS is mounted so that its edges fold around the printed circuit ( PC2 ) where they are pressed against contact fingers on the other side.) The contact points at each intersection of the perpendicular conductors form the pressure sensing cells. The device also includes a separator to pull the conducting layers apart when pressure is released. The sensitivity and range of the sensor depend largely on the construction of this intervening layer. There is a trade-off between sensitivity and range. For a large pressure range, the best separator tested was the woven mesh of a nylon stocking. For high sensitivity, a separator may be deposited directly onto ACS by spraying it with a fine mist of nonconductive paint. The conductive rubber presses through the separator so that the area of contact, and hence the contact resistance, varies with the applied pressure. The pressure/resistance relation is non-linear and, moreover, there is no known model to describe the relationship. However, the relationship has been determined empirically and seems to be a stable characteristic after an initial setting period. The array can be scanned by applying a voltage to one column at a time and measuring the current flow in each row. A potential problem with this method is the introduction of "phantom' tactile images which take Tactile Sensing 13

RSD-TR-11-86 place when it is not possible to distinguish between conduction along two or more different paths. This has been overcome by using an approach similar to the voltage mirror approach [24]. s. Torque sensitive sensor: ( Hackwood et. al. [15 ) This tactile sensor, unlike most of the others, is sensitive to torque, as well as to normal and tangential forces. A schematic diagram of the sensor array is shown in Fig.[61. Each element in the array is composed of three parts: (1) a magnetic dipole, (2) being embedded in a compliant medium and (3) a substrate containing magnetic sensors. The dipole in the compliant medium and the magnetic sensors are in parallel planes. The relative configuration of the magnetic sensors and the dipole is shown in Fig.[6]. These sensors employ the magnetoresistive effect and are thus able to detect both a dipole translation and rotation in the plane parallel to the substrate in contrast to the ones which use Hall effect which is sensitive to dipole translation only. For each tactile element there is four magnetoresistive sensors arranged on the substrate as shown in Fig.[7J. The resistance along the major axis of these sensors is a function of the magnetization parallel to it. The difference between the detected four resistances allow reconstruction of four degrees of freedom of dipole motion, i.e., translation in the X,Y,Z direction and torque about the Z axis. The inherent problem of hysteresis in these type of sensors can be overcome by using the magnetic bubble memory technology. Each of these sensors can be addressed separately. Finally, the compliant medium can be tailored to the particular pressure sensitivity requirement ( typically 1 - lOOg/mm2 for robotic applications ). 14 Tactile Sensing

RSD-TR-11-86 Some comments: * The compliant medium used was Sylgrad which has good elastic and thermal properties. The compliant medium can produce hysteresis, however, the choice is not as restricted as in the elasto-optic tactile sensors which require the material to have special optical properties. * Temperature compensation of the resistors can also be done to take care of temperature variations due to friction, etc. 6. All digital VLSI sensor: (Raibert [23]) This tactile sensor overcomes some of the drawbacks of the sensors described previously. Specifically it obviates the need for conductive elastomers by using a switch closure type technique. This method also does the analog to digital conversion totally mechanically thereby eliminating the need for any analog circuitry. Thus, one avoids the problem of combining the digital circuitry needed for preprocessing with any analog circuitry on the same VLSI substrate. Fig.[8] & [9J show the basic transduction method employed in this sensor. The basic principle used is that the depth of penetration of a layer of elastic pressed against a hole on a rigid surface is directly proportional to the applied pressure and the radius of the hole. One could define'contact pressure' as the pressure required so that the elastic layer just touches the bottom of the hole. If the depth of the hole is fixed, this contact pressure will be a function of the size of the hole. If an electrode were to be placed at the bottom of the hole and the underside of the elastic coated with a conducting medium, then when the applied pressure exceeds the above defined contact pressure the switch would close. One could have a bank of such holes of different sises Tactile Sensing 15

RSD-TR-11-86 and thus with different contact pressures and this would then quantize the applied pressure to whatever resolution is desired. This transducer basically uses the same principle but with a more compact design. A tapered hole with many sensing electrodes is employed which avoids the use of many holes of different sises ( Fig.11O] ). An electrode in the narrow region of the notch will be touched by the grounded elastic material only when the pressure is large, while an electrode in the wide region of the notch will be touched even if the pressure is small. The taper of the notch can be designed to obtain the kind of response characteristic desired. The raw data is encoded into 4 bits/cell ( 15 contact electrodes ), converted into serial form and finally multiplexed with the data from the other cells. This sensor is still in the development stage so the specifications are subject to change. 7. Ferroelectric polymer tactile sensors with anthropomorphic features: (Dario, Rossi et. al. [10] ) By attempting to combine various different sensing elements which operate in different frequency, sensitivity/dynamic range and even sense different features, this tactile sensor is a significant step forward when compared with the previously described sensors. This sensor comes from the bioengineering circles and is designed as much for applications in prosthetics as it is for robotics. Thus there is a natural inclination to mimic the skin, which even otherwise might be a useful model. For this reason the reader might like to refer to the literature on human tactile sensing [261 to better appreciate the design ideology. 16 Tactile Sensing

RSD-TR-11-86 A cross-section of the tactile sensor describing its different components is shown in Fig.[11]. It consists of three basic transduction layers mounted on a rigid printed circuit board which can be either flat or curved in order to conform to the shape of an artificial fingertip. The printed circuit board has a pattern of electrodes defined on the upper side through metalized holes. A thick PVF2 sheet intended to reproduce'dermal' receptors is bonded to the printed circuit board. When the PVF2 film is pressed, electric charge is generated by the piezoelectric effect and a voltage can be measured between the upper and each lower electrode ( to which the charge is transferred capacitively ). A layer of pressure sensitive conductive silicone rubber is laid onto the PVF2 film, which can have a small number of electrodes deposited either on both surfaces or on a flexible printed circuit with which the rubber is in contact on the bottom side. A suitable measuring circuit can detect the variation of electric resistance between the electrodes and provide a measurement of the pressure exerted by the object on the tactile sensor. This layer is affected by several problems peculiar to the presently available conductive rubbers i.e. drift, hysteresis, long time constant, poor mechanical properties, etc. However it is a very important supplement to the PVF2 layer as it is able to measure truly static contact pressure that PVF2, being a piezo-electric material, is inherently not able to detect. It also shields the bottom PVF2 layer from sudden external temperature variations. A third layer on the top of the laminated transducer structure is aimed to detect very small pressure variations or vibrations as needed for texture analysis. This sensor layer is made of a thin film of PVF2 that, being backed by the rubber layer, works primarily as a membrane and is therefore 1seniTactile Sensing 17

RSD-TR-11-88 tive to small forces. The operation of such surface sensor might not require very high spatial resolution. It is possible to define various patterns of largely spaced sensor elements on the PVF2 film by selective polling. In this configuration, while the upper side of the PVF2 film could be uniformly metalized and grounded for better shielding, a pattern of electrodes corresponding to the sensors and related thin, signal transmission tracks could be deposited on the bottom side. Proper connection pads could be provided along the perimeter of the PVF2 film. After electric isolation with a Mylar film, some of the PVF2 sensor elements can be backed with a thin layer of resistive paint that, connected to a DC power supply, can heat the PVF2 sensors up to a temperature of about 37C. When a sensor element touches an object, heat is drained from the heated element at a rate which depends upon the thermal conductivity of the material of which the object is made. Temperature variation rate is detected by the PVF2 sensor via the pyroelectric effect. The resulting signal will be the resultant of the pyroelectric signal and of the piesoelectric signal originated by object pressure. A composite, multi-element transducer like this one requires rather complex electronic circuitry in order to amplify and preprocess the signal originated from the various sensing elements. Conductive rubber sensors can be read quite conventionally by monitoring the change in resistance via suitable current sensing devices. Arrays of piezoresistive sensors can also be scanned and outputs presented sequentially without significant crosstalk between the sensing elements. 18 Tactile Sensing

RSD-TR-11-86 Piesoelectric transducers are charge generators. Being high impedance devices, signal amplification is, in general, more delicate for them than for piesoresistive transducers, especially when very low frequency operation is required. Careful shielding from electrical interference and use of coaxial cables are mandatory to improve signal to noise ratio. Either high input impedance voltage amplifiers or charge amplifiers should be connected to piesoelectric transducers. However, the high cost of good quality amplifiers of such type prohibits the use of individual amplifiers for each cell. Therefore, a more practical solution based on multiplexing circuits which sequentially connect the output of each sensor to a single, high quality FET-input operational amplifier is employed. 8. Lord Tactile Sensor 200 (LTS200): This is a commercially available tactile sensor and is fairly representative of the state-of-the-art tactile sensor technology. It is a deflectometer type of sensor and works on the principle described in Figure[l121. An applied force depresses a rubber spring, and the compression of the spring is measured by the partial interruption of the light beam. Therefore, the sensor actually incorporates transduction at three levels: (1) the transduction from force to displacement of the pin attached to the spring; (2) the change in the light level striking the electro-optic detector due to the displacement of the pin; and (3) the transduction that takes place in the photodetector, which varies the output electrical current as a function of the light striking its surface. All three of these transduction processes must be controlled and all contribute to the determination of the sensitivity of the measurement. Despite the complexity of the relationships involved in these compound transductions, the device is still Tactile Sensing 19

RSD-TR-11-88 basically a deflectometer, it measures deflection derived from a force applied to its surface. This lightweight sensor features a compliant, very low hysteresis touch surface with high strength, tear and abrasion resistance. A microprocessor based interface unit supports the sensor. The interface unit accepts commands from the host and preprocesses data, makes calibration corrections and provides several options for data readout. The communication between the interface unit and the host system takes place on a RS-232C link. 3.1 Remarks The different transduction principles which can be used for tactile sensing is virtually unlimited. This is clear from the diversity of techniques used in the sensors described above. One of the interesting facts to be noted is the realization of'smart' sensors which use a VLSI base with local circuitry permitting necessary preprocessing ( in a parallel fashion which the main computer might not be able to do ) and simplification of communication of data. This kind of an active substrate can, in principle, be used ( instead of a passive PCB ) with virtually any transduction technique. Another significant development is that of composite transducers which combine different transducer layers which supplement and complement each other. These two developments in essence should make the realization of truly powerful sensors possible. It would bear repeating that the list of sensors described is by no means exhaustive but only representative. Moreover, since most of these were still at the research and development stage ( and definitely nowhere near commercial production ) their specifications were only typical and could be modified. A limitation of such a description is that it does not give a dear quantitative picture of 20 Tactile Sensing

RSD-TR-11-86 the-state-of-art commercially available sensors. 4 Recognition Using Tactile Sensing Tactile sensing, by providing another means to acquire information about the external world, could both supplement and complement other sensory information like vision, and thus be useful in many tasks of a robot. Considering the nature of information sensed, it could help in the proper grasping and handling of objects by providing a direct feedback of the force distribution. Alternately it could provide a direct measurement of certain features i.e. texture and shape ( maybe only partial ) and thus help recognize an object. The two tasks mentioned above might also be combined in a natural way; for instance, in a bin sorting problem an object has to be handled and identified at the same time, or it may be that before handling an object it might be necessary to recognise the object or vice versa, that is in order to identify an object it might be necessary to *feel' it. Most of the research so far has been on the use of tactile sensing for object recognition, and although this usually involves the need to grasp the object and possibly handle it as well, we shall concentrate here on the basic issue of recognition and only make references to the relevant work on grasping. The problem of object recognition is a long standing one, and has received considerable attention particularly from the computer vision group. Conceptually the problem is as follows: on the one hand we have some knowledge about the objects to be recognized in the form of models ( statistical or deterministic ) and on the other we have some'sensed information'. It is the goal of the object recognition system to match the two. However, in spite of its conceptual simplicity, the task is a herculean one in Tactile Sensing 21

RSD-TR-11-86 practice. It is not so much a problem of having insufficient information as it is of finding suitable methods to use this information. Most often the information is implicit and not explicit. Consider as an abstraction a function y = f(x) where x and y can be from any abstract set, and then consider the problem of finding z given y. If we know the inverse function (may not be one to one ) we can explicitly find out z in terms of y to within the given multiplicity. However, if the inverse function is not known explicitly we have to scan the entire domain of z to find the solution. The problem in object recognition is very similar. What is therefore required to perform object recognition is some kind of a search technique, and the larger the search space and the range of sensing information available, the more time is required to come to a solution. To compound matters, usually noise is also present in the observation. Thus, usually the goal is to discover efficient heuristics to find hat is calleda satisfying solution' instead of an'optimal solution'. Refer to the paper by Besl and Jain [7] for a detailed discussion of the 3D recognition problem. Moreover, if we permit active sensing we add an additional degree of freedom. This could be an additional nuisance to deal with as we now have an additional parameter that needs to be controlled and we also have to handle all the extra information that is now available. However, if used judiciously it could not only provide greater information, but may also reduce the time required to search for a good solution. In this perspective, tactile sensing could be used alone or along with other sensory information like vision for the recognition task. Moreover, we could do with a single tactile image ( passive sensing ) or with multiple images ( active sensing). Before getting into the details of some of the suggested schemes, it would be worthwhile at this stage to point out some of the salient features of tactile sensing which 22 Tactile Sensing

RSD-TR-11-88 distinguish it from vision which, up till now, has been the single most important sensing modality for the recognition task. Machine vision is a very rich sensing medium, able to provide large amounts of data very rapidly. Vision by its very nature is a sensing of the brightness ( or light intensity ) in the field of view, and thus is a recording of various factors like surface geometry, reflectance and illumination. All of these effects can be tightly coupled in an image and can be difficult to separate. Thus interpreting 2D objects is fine in as much as they can be distinguished by their differences in reflectance characteristics. However when considering 3D objects there is some kind of a projection mechanism which comes into play thereby losing out the third dimension. Much of machine vision processing is involved with recapturing this lost dimension. As stated previously, the sensed feature, intensity of light, is a function of many tightly coupled factors, and this further compounds the problem. On the other hand, tactile sensing is a much slower process with much less information content as it is generally able to'view' or more aptly'feel only a small portion of the object. However, it is able to sense directly the 3D shape of an object, which reduces a lot of time consuming processing There is less noise in the process and there is no coupling effect. Tactile is an active sensing medium requiring a large degree of control, which has traditionally been the most difficult problem in robotics applications. To make better use of the information available it is important to consider the sensory data available from joints and limbs about position, orientation, velocity, forces, etc.. 4.1 Recognition Schemes Employing Only Tactile Sensing Now we shall look into some of the proposed schemes for object recognition using tactile sensing. A possible way to reduce the available information space and make Tactile Sensing 23

RSD-TR-11-86 it more tractable, would be to extract certain important features which still contain enough information to discern between various objects. The feature space could then be used to find a solution. Many useful features have been considered in vision, and some of these can be translated into the tactile domain. Others might be particular to tactile sensing. Ellis [121 has extracted features from tactile sensing like the ones listed below for the purpose of object recognition: - Three-dimensional position of the area of contact of the sensor and object, relative to the robot's base reference frame. - A measure of the planarity of the contact area. - If the contact is with a plane, the surface normal of that plane. - Whether the area of contact seems to be an edge, rather than a gently curved surface. - Whether the contact is with a corner or vertex, rather than a gentlycurved surface. - Whether there are slots present in the area being sensed. - Whether there are holes present in the surface or edge being sensed - A measure of the surface texture of the area of contact. - An estimate of the radius of curvature of an edge or corner. - The manner in which the object deforms in application to a force. In this list, the last two definitely require active touch sensing Texture, while it can be measured passively, probably is better measured actively. However, the features extracted using active sensing in this work basically involve vaying the pressure with which the object is held which is a very simple scheme in 24 Tactile Sensing

RSD-TR-11-86 the general domain of active sensing. Moreover, most of the features discussed here are an extension from the vision area. * Grimson and Lozano-Peres [14J have described a scheme using tactile sensing to identify and locate polyhedral objects with up to six degrees of freedom. The inputs to their recognition system are: a set of sensed points and normals, and a set of geometric object models for the known objects. The recognition process proceeds in essentially the following two steps: 1. Generate Feasible Interpretations: A set of feasible interpretations of the sense data is constructed. Interpretations inconsistent with local constraints ( derived from the model ) on the sense data are discarded. 2. Model test: The feasible interpretations are tested for consistency with surface equations obtained from the object models. An interpretation is legal if it is possible to solve for a rotation and translation that would place each sense point on an object surface. The sensed point must lie'inside' the object face, not just on the surface. The range of possible contact patterns between multiple sensors and complex objects is highly variable and the rich geometric data available from object models can be exploited to reduce the search space to more manageable levels. This method is an instance of a description-based recognition method. However this differs from global feature-based or surface-based description methods by relying only on sparse 3D positions and surface normals, the kind that is generally available from tactile sensors. This kind of description ( Interpretation ree ) also makes possible the use of local constraints ( distance, angle, direction ) in pruning and thus reducing the search space. Refer to 1141 for further details. Tactile Sensing 25

RSD-TR-11-86 4.2 Integrating Tactile Sensing With Vision Next we discuss some schemes which combine tactile sensing with other modalities like vision to perform the object recognition task. * Luo, Tsai and Lin [21] have proposed a hierarchical object recognition system which makes e of a decision tree. First recognition ( or partial recognition) is attempted by using 2D visual information and then, if necessary, the 3D tactile information is employed. This scheme also uses a feature extraction method. Once the features have been extracted, the decision tree is employed to identify the object. The features used are the moment invariantbW of object silhouette shapes. Refer to 121] for details of the ~moment invariants. The recognition procedure begins by taking the top-view visual image of the object ( after it is brought right under the TV camera on the platform ). The object is then discriminated against the features extracted from the visual image. If the object is not discernible with its visual image alone, P pair of tactile images are then taken. MAfter more object features are extracted, the object is discriminated against further. This step may have to be repeated more than once. If the object is still not identifiable, the object may be moved into a different position, and more images taken from this new viewpoint. The main problem that can be foreseen in using a decision tree type approach is that it can be used in situations with certain restricted types of objects and in general a yes/no type of decision is rather inaccurate and difficult to make when employing the decision tree to discern between different object types, especially since the measurements are never noise-free. 26 Tactile Sensing

RSD-TR-11-86 * Another scheme which attempts to combine vision with tactile sensing has been described by Allen [11. The intention here is to form a 2' sketch by using surface descriptions, prior to the actual object recognition stage. The motivation is drawn from an analogous stage in human perception of 3D objects. A stereo image is obtained first. The 3D coordinates of contours and edges can be calculated. There are, however, numerous candidates for filling in the intervening surfaces. Using contour data alone, a single surface patch ( bicubic surface patches are used as primitives here) can be created which interpolates the surface. It is unlikely that this surface will fit the real surface except at the boundaries. By injecting tactile trace information into this set of boundary curves, a better approximation to the real surface can be found. Since a level 1- parametric surface based on boundary data alone exists, a tactile trace is executed from the parametric midpoint of a boundary curve to the midpoint of the opposing boundary curve. Successive refinement can be carried out in this manner. Matching, by using these partial surface descriptions, can now be performed. This problem has been addressed separately. As in any matching scheme, a transformation must be effected between the model in a canonical form and the particular instance of object under scrutiny. It is here that tactile sensing can be a further aid in the recognition process. If the object recognition system makes a hypothesis about the nature of the object, tactile sensing can be used to verify model features in the scene. Further, if the object recognition system cannot decide amongst candidates, the tactile sensing system can be invoked to arbitrate. This is another of active sensing that requires control strategies. Tactile Sensing 27

RSD*TR-11-86 4.3 Comments Having discussed a few recognition systems, which make use of tactile sensing either alone or in conjunction with vision, let us now consider some of the general issues involved, and some basic guidelines for such a general purpose recognition system. Since tactile sensing involves direct contact with the object, it is required to first'spot' the object of interest and unless the location of the object is fixed a priori, it is perhaps useful to employ vision for this purpose instead of'blindly' feeling around. By using vision initially, an approximate description of the object can be obtained. Since tactile sensing provides rather incomplete information about the object ( unless the object size is smaller than the sensor ) it is also a good scheme to drive the tactile sensing. There should be a specific motivation behind any instance of tactile sensing, i.e. verifying the location of a hypothesised surface or edge. A description based approach could be used for this purpose, as adopted by Allen [11. After having obtained an initial description based on vision alone, successive refinements can be made by actively employing tactile sensing. Later on in the matching stage, tactile sensing could again be used to resolve ambiguities or to directly sense certain discerning features. It seems desirable to have an objective in mind while performing tactile sensing for the reasons cited previously ( partial information ). An important issue that surfaces in tactile sensing is'grasping' the object and problems related to this issue when'feeling' an object. Some of these problems have been addressed in [31, [131, [25]. A schema type approach as employed by Overton [22] could be a powerful way of structuring the recognition process. The entire recognition process here is carried out by a set of sequentially activated schemas. At each step, the result of one schema activates another appropriate schema depending on the knowledge base. Each of 28 Tactile Sensing

RSD-TR-11-88 these schemas is a set of sensing and/or processing operations to be performed. All the relevant side issues like positioning the sensors and grasping are resolved within the schema, or by making calls to appropriate system routines. It also seems clear that by exploiting the specific nature of the type of objects one is expected to encounter, a much simpler and faster process could be implemented. To overcome the inflexibility of such a domain dependent approach, the general recognition system could be constructed in a hierarchical fashion by combining simple subsystems. It is possible to perform low level processing in parallel ( possibly by hardware), thereby greatly reducing the time required for recognition. These operations could include various data driven feature extraction processes i.e. edge detection etc.. Tactile Sensing 29

RSD-TR-11-86 References [11 Peter Allen,'Surface Descriptions from Vision and Touch,' IEEE 1984 Int. Conference on Robotic., 84CH2008-1, pp. 394-399. [2J Peter Allen,'irualy Driven Tactile Recognition and Acquisition.' [3] Rusena Bajcsy, Michael J. McCarthy, and Jerry C. Trinkle,'Feeling by Grasping.' [41 R. Bardelli, D. De Rossi, and P.C.Pinotti,'Pieso and Pyroelectric Polymers Skin-like Tactile Sensors for Robots and Prothese', 13th Int. Symposium on Industrial Robota and Robots7, Vol. 2, CH18, pp. 45-56, 1983. [5] Begej Laboratories Inc.,'TSA-32-V4 Optical Tactile Sensor Array,' Technical Bulletin No.l April 1985. [6J Harry L. Benjamin,'The Development of a Production Robot Tactile Position Sensor,' 1th Int. Symposium on Industrial Robots and Robots7, Vol. 2, CH18, pp. 57-76, 1983. [71 Paul Besl and Ramesh Jain,'An Overview of Three- Dimensional Object Recognition,' Robot Systems Division Technical Report ( RSD-TR-19-84 ), CRIM, The Univ. of Michigan. [8J R. A. Boie,'Capacitive Impedance Readout Tactile Image Sensor,' IEEE 1984 Int. Conference on Robotic., 84CH2008-1, pp. 370-378. 30 Tactile Sensing

RSD-TR-11-86 91 K. J. Chun and K. D. Wise,'A Capacitive Silicon Tactile Imaging Array,' IEEE, CH2127-9/85, pp. 22-25, 1985. [10] P. Dario, D. De Rossi, C. Domenici, R. Francesconi,'Ferroelectric Polymer Tactile Sensor with Anthropomorphic Features,' IEEE 1984 Int. Conference on Robotics,84CH2008-1, pp. 332341. [111 Earnest O. Doeblin, Measurement Systems: application and design,McGrawHill, 1966. [12] R. E. Ellis, Eztmction of Tactile Features by Passive and Active Sensing. [131 Samuel M. Goldwasser,'Computer Architecture for Grasping,' IEEE 1984 Int. Conference on Robotics, 84CH2008-1, pp. 320325. [141 W. E. L. Grimson and Tomas Lozano-Peres,'Model- Based Recognition and Localization from Tactile Data,' IEEE 1984 Int. Conference on Robotics, 84CH2008-1, pp. 248-255. [151 S. Hackwood, G. Beni, L. A. Hornal, R. Wolfe and T. J. Nelson,'A Torque-Sensitive Tactile Array for Robotics,' Int. Journal of Robotics Research,Vol. 2, No. 2, pp. 46-50, 1983. [16] Leon D. Harmon,'Touch-Sensing Technology: A Review,' Society of Manufacturing Engineers: Technical Report, MSR80 -03. [17] Leon D. Harmon,'Automated Tactile Sensing,' Int. Journal of Robotics ResCarch, Vol. No. 1 pp. 3-32, 1982. Tactile Sensing 31

RSD-TR-1 1-86 [18] Leon D. Harmon,'Automated Touch Sensing: A Brief Perspective and Several New Approaches,' IEEE 1984 Int. Conference on Robotic., 84CH2008-1, pp. 326-331. [191 W. D. Hillis,'A High Resolution Imaging Touch Sensor,' Int. Journal of Robotics Research, Vol. 1, No. 2, pp. 3344, 1982. [20] Lester E. Krueger,'Tactual Perception in Historical Perspective: David Katz's World of Touch,' from Tactile Perception: A Sourcebook, by William Schiff and Emerson Foulke, Cambridge University Press, 1982. [21] R. C. Luo, W. H. Tsai and J. C. Lin, Object Recognition with Combined Tactile and Visual Information. [22] Kenneth J. Overton,'The Acquisition, Processing, and Use of Tactile Sensor Data in Robot Control,' COINS Technical Report 84-08, Univ. of Mass., Amherst. [23] Marc H. Raibert,'An All Digital VLSI Tactile Array Sensor,' IEEE 1984 Int. Conference on Robotica, 84CH2008, pp. 314-319. [24] Marc H. Raibert and J. E. Tanner,'Design and Implementation of a VLSI Tactile Sensing Computer,' Int. Journal of Robotics Research, Vol. 1, No.3, pp. 3-18, 1982-83. [25] J. Kenneth Salisbury and John J.Craig,'Articulated Hands: Force Control and Kinematic Issues,' Int. Journal of Robotics Research, Vol. 1, No. 1, Spring 1982. 32 Tactile Sensing

RSD-TR-1 1-86 [26) William Schiff and Emerson Foulke, Tactual Perception: A Sourcebook, Cambridge University Press, 1982. [271 Carl E. Sherrick and James C. Craig, TShe Psychophysics of Touch,' from Tactile Perception: A Sourcebook, by William Schiff and Emerson Foulke, Cambridge University Press, 1982. Tsactile Sensing 33

0 m CL ao 0 CDC ocn~co c E 0 og EE t~~~~~~ 0 CL. U) U) 0 E 0 0~~~~~~~~~~~~~~~' H U) Lu a C/)Q _ _ _ _ _ _ _c E <~~~~~~~~ coCo ~~o 0 c- - Cu o U)~~~ 0 C/)~~~~~~~~~~~~~~~~~~~~~ ca~~~~~~~~~~~~~~= n'~~~~~~~~~~~~~~~~~~~~~~~~. -orr 0r

CONDUCTIVE STRIPS ELASTIC/DIELECTRIC LAYER SHIELD Aii VD COS Wo t r A~ ~/ —-~- /P ANALOG MULTIPLEXER, *_ - / -i" FLEXIBLE PRINTED CIRCUIT BOARD _/d~~;g N ANALOG MULTIPLEER "j" VAi,it)= - V(COS wot)Cij/CA Fig. 2 Exploded view of a sample 6x6 Capacitive Impedance Sensor

"FORMAS. POO OUTER SKIN LA PLATE // //////S/UP~OO/RT //////// TOP VIEW Fig. 3 Cross-section of the Silicon-based Tactile Imager (top) and the Layout of two cells (bottom).

Pressure-sensitive / elastic material _' /'"' Surface,.::::::.::..:.~: -metal electrodes Computing elements Silicon substrate Fig. 4 Physical structure of the VLSI Tactile Sensing Computer.

PC2 PCI PC2 Fig. 5 Physical structure of the Tactile sensor developed by Hillis.

F- -7 -s -, - - ESX3, 1/,,., ~ ~ ~~~~~~~~~~~~~~~~~~~L~~~~~~~ 1111,il 11 ~_11 I 2) -~~~~ -~~~~~ -~~ -~ -~~ - Randomirzine field Fig. 7 Magnetoresistive sensors' configuration. 6 Torque sensitive sensor. op view of a 7x7 array Cross section to show dipole(1), edded in a compliant medium(2), ontact with the substrate which ains a magnetoresistive element(3). Schematic operation of the device ormal forces.

Pressure P VI 4 M..... Elastic Material 6T( Fig. 9 Fig. 10 Top View of Notch IHIJI 1. 1 I I Il All digital VLSI sensor Depth of penentration is directly propotional to the pressure (Fig. E Fr~ig.~ 8 ~and to the radius of the hole (Fig. Fig.*~~ 8 This basic idea is used to form the Notch used in this sensor.

"EPIDERMAL" PVF2 SENSOR RESISTIVE PAINT CONDUCTIVE RUBBER "DERMAL" PVF2 PRINTED CIRCUIT MATRIX SENSOR BOARD Fig. 11 Cross-section of the Composite Tactile Sensor developed by Dario, Rossi et. al.

TACTILE ARRAY SINGLE SITE LOAD ELASTOMER IC. STRUCTURE LENS PHOTO EMITTER PHOTO DETECTOR ZFRO DEFLE CT IOWN MDERA RL DEFLE CTON Fig. 12 The Deflectometer used in Lord Tactile sensor 200.

Sensor type Resolu- Size Dynamic Sensiti- Comments & Source tion Range vity Low hysteresis. 0 Typicalr: 1 -2mm 1 Ox1i0 1 - 100gm 5 gm Correctable non-linearities allowed. Harmon Robustness. Capacitive: 2.5 mm 8 x 8 50,000 Good mechanical and dielectric properties. Bole I: dynes/sq cm Good signal/noise ratio. Access time - 20 us for 8 pixels. 2 Capacitive(VLSI): 2 mm 8 x 8 1 - 45 gm 15 - 100 Range can varied easily. Chun & Wise mv/gm 8 bits resolution. Resistive(VLSI): 1 mm 6 x Preprocessing & Communication circuitry imple3 Raibert, Tanner mented on VLSI substrate. Defect tolerant. Trade off between speed and serialization. Resistive(ACS): Good mechanical durability. Stable electrical 4 Hillis 1 mm 16 x 16 1 - 100 gm characteristics after initial setting period. Pressure-resistance relationship non-linear. Sensitive to torque also. Low hysteresis. 5 Magnetoresistive: 2 mm 7 7x7 1 - 100 gm __________ Susceptible to external magnetic interference. Hackwood et al. Good elastic and thermal properties. Piezoelectric, -2 Linear response. 7 Pyroelectric: 3 mm 8 x 16 10 Newton 140 - 740 Uniform frequency response. Large bandwidth. Rossi et al. mV/N Crosstalk- 3. Deflectometer Fatigue life - 100 x 10E6. 8 (Optoelectronic): 1.8 mm 10x 16 5 gm Overload protection. Lord (LTS 200) * Specifications of an ideal sensor as suggested by Harmon. Table of sensor specifications

UNIVERSITY OF MICHIGAN 1139015 02330 9910111111 3 9015 02330 9910