Errata p. 26 last line: Delete "s" on "follows". p. 29 line 4: Insert the word "linear" between "a" and "mapping". p. 33 line 27: Replace G by e in the bracketed expression. p. 59 5th line of Sec. 3. 8: Insert "of" between "that" and "finding". p. 63 line 2: Insert "a" between "is" and "case". dC(t ) dC(t ) p. 66 line 20: should be dt dt1 p. 83 Eq. 5. 3: Add the condition p > 1. p. 165 Footnote: Change "Dranc" to Kranc".

Technical Report No. 164 6137-7-T MINIMUM PEAK AMPLITUDE CONTROL by F. M. Waltz IN I Approved by: B. F. Barton for COOLEY ELECTRONICS LABORATORY Department of Electrical Engineering The University of Michigan Ann Arbor Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in The University of Michigan May 1965

TABLE OF CONTENTS Page ACKNOWLEDGEMENTS ii LIST OF ILLUSTRATIONS v LIST OF SYMBOLS vii LIST OF APPENDICES xii X1i11 ABSTRACT CHAPTER I: INTRODUCTION 1 1. 1 Brief Statement of the Problem 1 1. 2 Review of the Literature 3 1.2. 1 Historical Foundations 3 1.2.2 Pontryagin's Maximum Principle 5 1. 2. 3 Dynamic Programming 6 1.2.4 Direct Methods 9 1.2.5 Current Work 10 1. 2. 5. 1 Mathematical Reformulations, Extensions, 11 and Reinterpretations 1. 2. 5. 2 Computational Advances 12 1. 2. 5. 3 Restatements and Revision of Basic Goals 13 1. 3 The Literature and the Minimum Peak Amplitude Problem 14 CHAPTER II: THE LINEAR PROBLEM 16 2. 1 Detailed Statement of the Problem 16 2. 2 Discussion of Assumptions and Restrictions 17 CHAPTER III: SOLUTION OF THE LINEAR PROBLEM 23 3. 1 Preliminary Definitions and Results 23 3. 2 Existence of an Optimal Control 25 3. 3 Some Generalizations 35 3. 4 The Form of the Optimal Solution 37 3. 5 Two Examples Involving Nonunique Optimal Control 42 3. 6 Proper Systems 49 3. 7 Two Examples Involving Proper Systems 55 3. 8 Problems with Unspecified Final Time 59 3. 9 Optimal Control Laws 69 CHAPTER IV: THE INVERSE PROBLEM 72 4.1 A Time-Optimal Problem 72 4. 2 The Solution to the Time-Optimal Problem 72 4. 3 The Inverse Relationship 74 4. 4 A Necessary Condition for the Existence of the Inverse 78 Relationship 4. 5 Summary and Conclusions 79 CHAPTER V: THE RELATED LINEAR PROBLEM 80 5.1 Motivation 80 5. 2 Statement and Solution of the Related Linear Problem 82 5.3 The Limiting Process 86 5. 4 Examples of the Limiting Process 92 iii

TABLE OF CONTENTS (Cont. ) Page CHAPTER VI: NONLINEAR PROBLEMS 100 6. 1 Introduction 100 6. 2 Statement of the Nonlinear Problem 101 6. 3 Statement and Partial Solution of the Related Problem 102 6.4 Two Examples 105 CHAPTER VII: COMPUTATIONAL ALGORITHMS AND EXAMPLES 113 7. 1 Preliminary Discussion 113 7. 2 An Algorithm for Proper Systems 117 7. 3 Another Algorithm 126 7.4 Some Additional Examples 130 CHAPTER VIII: SUMMARY AND CONCLUSIONS 139 8. 1 Introduction 139 8.2 Conclusions 139 8. 3 Suggestions for Future Research 142 APPENDICES 143 REFERENCES 177 DISTRIBUTION LIST 183 iv

LIST OF ILLUSTRATIONS Figure Title Page 1. 1 An example in which the cost functional is related to peak 3 instantaneous input power. 2. 1 An example in which the input directly influences the output. 21 3. 1 The sets S1 and S for Example 1. 44 3. 2(a) Typical sets S1 for systems which are neither proper (rotund) 55 nor smooth. 3. 2(b) Typical sets S1 for systems which are proper (rotund) but 55 not smooth. 3. 2(c) Typical sets S1 for systems which are smooth but not proper. 55 3. 2(d) Typical sets S1 for systems which are both smooth and 55 proper (rotund). 3.3 The set S1 for Example 3. 56 3.4 The set S1 for Example 4. 58 3. 5 C(t1) vs. t1 for Example 5. 61 3.6 C(t1) vs. t1 for Example 6. 63 3.7 C(t1) vs. t1 for Example 7. 64 3. 8 C(t1) vs. t1 for Example 8, showing discontinuous derivative 68 at t1 = loge2. 4. l(a) A typical curve of cost vs. final time for a minimum peak 77 amplitude problem. 4. l(b) C vs. t1 for the inverse problem, showing only those points 77 which satisfy all the conditions of the maximum principle. 5. 1 The limiting process illustrated for Example 9. 94 5. 2 Upper and lower bounds on minimum peak amplitude provided 96 by related problems for Examples 9 and 10. 5. 3 The limiting process illustrated for Example 10. 98 5. 4(a) u (t) for various values of p for Example 11. 99 p 5. 4(b) Upper and lower bounds on C, as a function of p. 99 6. l(a) The optimal control up(t) for the related nonlinear problem 109 for various values of p, for Example 12. 6. l(b) Bounds on the optimal peak amplitude provided by lu pl100 and 109 Ilupllp for Example 12. v

LIST OF ILLUSTRATIONS (Cont.) Figure Title Page 6. 2(a) The optimal control up(t) for the related nonlinear problem 112 for various values of p, for Example 13. 6. 2(b) Bounds on the optimal peak amplitude provided by Iju p10 and 112 llup|lp for Example 13. 7. 1 Constructions to show the continuity of the mapping of aUk 116 onto as1. 7. 2 Simplified diagram of computer program for proper systems. 124 7. 3 Successive iterations of the computational algorithm for a 126 2-dimensional example. 7. 4 Simplified diagram for second algorithm. 128 7. 5 Convergence of the second algorithm for a 2-dimensional 130 problem. 7. 6 C - vs. - t1 for an undamped oscillatory system, for various 132 final conditions. 7. 7 C - vs. - t1 for a system with a stable node, for a particular 133 set of boundary conditions. 7. 8 C - vs. - t1 for a system with a stable focus, for various 134 final conditions. 7.9(a) C vs. t1 for a time varying system. 136 7. 9(b) The optimal control for a particular choice of t1. 136 7. 10 The variation of the minimum peak amplitude as a function 138 of the constant K (and the initial mass) for Example 18. m vi

LIST OF SYMBOLS General notes: Underlining is used to denote vector quantities. The components of an mcomponent vector quantity v are denoted by v1, v2,..., v. A symbol with a horizontal line above it usually denotes an optimal value of the indicated quantity. A dot above a symbol denotes the time derivative of the indicated quantity. Symbol Description First Reference a,a A,A(t) b,b, b(t) b11(t), b12(t), etc. B, B(t) B B* B** c c c, c -p-n C,C(u) C C,C p' n C oo0 d D, D(tl) E,E1,Ei E,E(t) i E, E(t) the given initial state the system matrix the desired final output elements of the matrix B(t) the input matrix a certain Banach space the conjugate of the Banach space B the conjugate of B* a vector in k-dimensional Euclidean space an optimal value of c for the minimum peak amplitude problem optimal value of c for the related problem the cost associated with the control u the minimum peak amplitude for a given problem constants appearing in the solution of the related linear problem the limiting value of C an arbitrary vector in Ek the output matrix errors (used in the computational algorithms) a certain k x r matrix 1 16 1 42 16 31 31 32 40 38 85 1 35 85 90 114 16 113 21 vii

LIST OF SYMBOLS (Cont.) Symbol Description First Reference f (x, t), f(x,u, t) F(c,t1) G, G(t) h,h(c) H, H(x, u,, t) I J,J(u) J,J x u k K K(t) L, L(z) I LII Lr Lr r Lr p' q' 1' o0 m Mi M 1 n n N,N,N,etc. g z' e N N( ) p q n-dimensional real Euclidean space nonlinear vector functions characterizing system behavior a certain nonlinear function of c and t the goal of the control problem the cost weighting matrix a point on the surface of the set S a Hamiltonian-like function the identity matrix a cost functional Jacobian matrices the number of components in the output a constant a function of time t in L1 the mapping taking controls into final outputs the norm of the mapping L Banach spaces an arbitrary constant > 1 an open interval on the ith coordinate axis of Ek a linear manifold in state space the order of the controlled system an arbitrary constant > 1 a neighborhood of a point in some topological space a positive constant a positive integer a constant > 1 used in the related problem the index conjugate to p 16 101 66 24 17 113 73 23 102 103 16 30 176 27 28 83 86 33 37 4 86 29 30 174 80 84 viii

LIST OF SYMBOLS (Cont.) Symbol Q(g) Qi Q r R1 R s S1 S t Description the set of bounded measurable z(t) which yield the point g a certain open set in Z; the inverse image of M. the terminal constraint function the number of input or control variables the closed unit sphere (ball) in B* or Z the closed sphere of radius C in B* or Z a dummy variable of integration points in S1 or S the image of R1 under the mapping L the image of R under the mapping L time First Reference 25 33 103 4 27 34 23 29 27 35 1 to ti T T T T1, T1(C) T u,u, u(t) u,u(t) u.(t) U(a,b, T) vij(t1, t) v**(z ), V-*( ) the initial time the final time as a set: the set of points in the interval [t0,tl] as an algebraic quantity: t1 - to as a superscript: indicates transposition the subset of T on which Iv(t)| or IV Tc| 0 the subset of T on which Iz(t)| = Lz|1 the input or control variable or vector an optimal control for the minimum peak amplitude problem the elements of a sequence of vector-valued controls the set of admissible controls causing the boundary conditions to be satisfied an arbitrary (row) vector an element of the matrix V(t1,t) bounded linear functionals in B** 1 1 1 80 16 37 176 1 26 9 17 28 28 32 ix

LIST OF SYMBOLS (Cont.) First Symbol Description Reference V, V(t1,t) the k x r matrix DX(t1,t) B(t) G t) 24 IlVII the maximum gain of V over all c in Ek and all 89 t in T W a matrix appearing in the complete controllability 30 condition W(c ) a nonlinear matrix function of c 126 x,x, x(t) the state variable or state vector 1 X, X(t, s) the fundamental matrix or state transition matrix 23 y, y, (t l) the output variable or output vector 16 z,z(t) the vector G(t) u(t) 24 z,z(t) a z(t) having minimum peak amplitude 25 z(v) a bounded linear functional defined over allv(t) 31 in B z,z.Zp the optimal control for the related problem of 84 index m, n, or p Z the Banach space of equivalence classes of 27 bounded measurable controls 6,6(t1) a small positive number 66 ~~e ~ set inclusion symbol 1 e a small positive number 66 Trl~ ~ the minimum eigenvalue of the matrix W 89 0 a number between zero and one 29 0 a constant vector in Ek 102 0 the angle between two vectors in Ek 113 ^0 ~ a Lagrange multiplier 73 _', /(t) a Lagrange multiplier vector 11 x

Symbol Special Symbols: I l, Ixi, etc. II II II v 11 IIV ll 1 Illl 11 v lp as, aR, etc. LIST OF SYMBOLS (Cont.) Description the Euclidean norm of the vectors v, x, etc. the essential supremum of the Euclidean norm of the vector v(t) the L -norm of the Euclidean norm of the vector v(t) the L -norm of the Euclidean norm of the vector v(t) P the boundaries of the sets S,R, etc. First Reference 1 24 31 80 114 xi

LIST OF APPENDICES Page APPENDIX A: OPTIMAL CONTROL PROBLEMS AS PROBLEMS IN 143 THE CALCULUS OF VARIATIONS APPENDIX B: DYNAMIC PROGRAMMING AS A COMPUTATIONAL 156 TECHNIQUE APPENDIX C: THE FUNCTIONAL ANALYSIS APPROACH TO 162 LINEAR TIME OPTIMAL PROBLEMS APPENDIX D: THE CONTROLLABILITY ASSUMPTION 168 APPENDIX E: CERTAIN PROPERTIES OF THE SPACES Lr AND Lr 172 p oo xii

ABSTRACT MINIMUM PEAK AMPLITUDE CONTROL by Frederick Marshall Waltz The purpose of this study is to develop methods of solution for minimum peak amplitude control problems; i. e., optimal control problems with specified initial and final conditions on the state variables and in which the cost functional is given by supremum T ( u(t where t0 is the initial time, t1 is the final time, u(t) is a measurable vector-valued control variable, G(t) is a matrix of weighting functions, and the symbols I| denote the Euclidean norm of the enclosed vector. This problem can be identified with various practical problems, such as the problem of determining the input signal to a given electrical device (e. g., a filter) which produces the desired output while requiring minimum peak input power, and the problem of determining the smallest thrust-limited steerable rocket engine capable of performing a specified task. The problem is initially formulated in terms of linear time-varying systems which can be modelled by differential equations of the form x = A(t)x + B(t)u y = Dx where x is the state vector, y is the output vector, the dot indicates differentiation with respect to time, and where A(t), B(t), and D are given matrices satisfying a certain complete controllability assumption. A theorem guaranteeing the existence of an optimal control (similar to the one given by Neustadt) is proven for this class of systems using a functional analysis approach, and a theorem about the form of optimal controls is obtained. It is shown by xiii

means of examples that the optimal control is not unique, in general. A suitably restricted class of systems (with wide engineering applicability) is described for which the optimal control is unique, and takes on the simple form G T(t) B (t) XT(t, t) DTc u(t) =CG (t) -T(t (t ) DTc I a. e. on [to0tl] where X(t1,t) is the state transition matrix for the given system, C is a nonnegative constant, c is a constant unit vector, and the superscripted T denotes transposition. The original problem of finding an optimal control function u(t) has thus been reduced to the much simpler problem of finding constants C and c such that the u(t) determined by the above expression causes the desired boundary conditions on the output to be satisfied. The implications of the uniqueness or lack of uniqueness of optimal controls for these problems are discussed in terms of the rotundity and smoothness of the reachable set (the set of all points in state space that can be reached in the given interval, starting from the origin and using controls with costs not exceeding some specified amount). Examples are presented to illustrate the various kinds of behavior that can be obtained. The relationship of the original class of problems to a class of minimum time problems in which a bound is placed on the quantity | G(t) u(t) | is investigated, and it is shown that while all such minimum time problems correspond to minimum peak amplitude problems, the converse is not true. As a possible approach to nonlinear minimum peak amplitude problems, to which functional analysis techniques and classical variational methods are not readily applicable, a so-called related problem (involving a generalization of the Lp-space norm as a cost functional, and solvable by classical variational techniques) is defined, and it is shown that for linear systems the limiting form of the solution of this problem is an optimal solution of the original problem. Corresponding results for nonlinear systems are not proven, but the technique is applied successfully to two nonlinear examples. Finally, computational algorithms for the solution of linear minimum peak amplitude problems up to tenth order are presented and discussed, along with additional numerical examples. xiv

CHAPTER I INTRODUCTION 1.1 Brief Statement of the Problem In its simplest form, the problem to be considered here can be stated as follows: From the class of bounded measurable vector-valued input or control functions u(t) defined on the interval T = [tot1] which cause a given linear system described at each moment of time by a state vector x(t) to transfer from the given initial state x(t0) = a at the given initial time to to the desired final state x(t1) = b at the given final time t1, choose one for which the cost functional C(u) = ts Iu(t)I is a minimum. Here I u(t)l is the Euclidean norm (or, synonymously, the amplitude) of the vector u(t) defined at each moment of time as the square root of the sum of the squares of the components of the vector u(t) at that moment. The following generalizations and modifications of this problem are also treated here: a) The final value of the state is required to lie in a specified linear manifold in state space, rather than at a fixed point b. b) The square root of a symmetric positive definite quadratic form in u(t) is used in place of the Euclidean norm of u(t) in the cost functional. c) The final time t1 is left unspecified initially, and must be chosen along with u(t) so as to produce the lowest possible cost. 1 A discussion of state variables and the formulation of problems in terms of state variables is given by Tou, Ref. 74. 1

2 d) Optimal control laws2 for certain of the problems considered here are discussed, and the general form required for these control laws is described. e) A generalization is made to certain nonlinear problems. Certain other generalizations and variations of the basic problem are also discussed briefly. We shall refer to all the problems noted above as minimum peak amplitude control problems. This problem formulation encompasses practical problems in various fields. An example from electrical engineering is as follows: The given system is the amplifierfilter shown in Fig. 1. 1. It is assumed that the vacuum tube interelectrode capacitances can be ignored and that the tube is operating in an essentially linear region and draws no grid current over the range of inputs involved. The state variables x1, x2, x3 are chosen as the voltages across the capacitors, and the input voltage is denoted by u(t). The input u(t) is to be chosen so as to cause the final state x(t1) = [x1(t1), x2(t1), x3(t1)] to lie in some specified manifold in state space and so that the cost C(u) = tuplu(t)I is a minimum. The instantaneous input power to this circuit is simply u, ( so that minimization of C(u) = Sup I u(t)I is equivalent in this case to minimization of the peak instantaneous input 3 power to the circuit. As another example, let the system in question be a rocket vehicle propelled by a single steerable rocket engine,the maximum thrust of which is to be specified. The thrust produced by the engine is regarded as a vector in three-dimensional Euclidean space which can vary in both magnitude and direction as a function of time. The components of this thrust vector along the three coordinate axes of the space are considered as the components of the input vector u(t). The Euclidean norm of u(t) is thus simply the magnitude of the thrust at a given moment of time. A typical minimum peak amplitude problem in this A control law is a rule or formula which states the value of thae control that is to be used at each moment of time as a function of the state at that moment and the desired final state. An optimal control law is one which results in a control which is optimal- i. e., a control which causes the cost to be minimum. This control law (which specifies the operating characteristics of the optimal feedback controller) is to be contrasted with the original formulation of the problem, which requires the determination of u as a function of timea result which is often termed open-loop or programmed control. This follows from the fact that if we minimize the peak value of a function, we also minimize the peak value of its square and of any constant times its square.

3 Fig. 1. 1. An example in which the cost functional is related to peak instantaneous input power. 4 case is the problem of determining the smallest thrust-limited engine that is capable of performing a specified task (e. g., putting a certain satellite into a certain orbit), and of determining the steering pattern to be used. From these two examples, it can be seen that the techniques to be developed here for the solution of such problems are essentially system synthesis techniques; that is, they give information about the smallest signal generator or the smallest rocket engine that will do the required job, as an aid in system design. If one already has a specific signal generator or a specified rocket engine at hand, the pertinent problem is then not the synthesis problem, "What is the minimum amplitude signal that will do the job?" but the analysis problem, "Will this device do the job, and if so, how well?", with performance being evaluated on some other appropriate basis, such as with a time or a fuel consumption criterion. Since numerous techniques for the solution of such analysis problems are available and widely known (Refs. 1 through 20), these problems will not be discussed in detail here except as they relate to the problem at hand. 1. 2 Review of the Literature 1. 2. 1 Historical Foundations. The problem described above is known as an Examples of engines that are primarily thrust-limited, rather than, for example, energylimited, are the engines which do not carry their own energy source, such as a "solar sail" or a light-powered ion engine.

4 optimal control problem. 5 Optimal control problems are the subject of an extensive and rapidly expanding body of literature. Interest of the engineering and scientific community in such problems began in earnest within the last fifteen years (Refs. 1, 2, 3,4). The mathematical foundation for much of the work being done today is found in the calculus of variations, developed long before this, beginning with the study of first-order necessary conditions for extremality (i. e., optimality) by Euler and Lagrange in the eighteenth century (Ref. 5, p. 54). Second-order conditions, which allow one to classify extremals as maxima, minima, or inflection points, (analogous to the classification of stationary points of ordinary functions by consideration of second derivatives) were then developed (Ref. 6, p. 23). With the publication of Bolza's work on variational problems with differential side conditions in 1913 (Ref. 7), all the needed elements were present for solution of optimal control problems which involve no restrictions on the state and control variables (except those implicit in the differential equations and boundary conditions). A discussion of these results and their relationship to optimal control problems (in their "modern" formulation) is given in Appendix A. Mathematicians (Bliss, Refs. 6, 8, and others) continued to refine and extend these results, and Valentine (Ref. 9, published in 1937) introduced a simple device which in effect allowed the n-component state vector x(t) and the r-component control vector u(t) to lie at each moment of time in closed sets in (n+r)-dimensional space, rather than open sets, as in previous works. This made possible a generalization of great potential application to engineering problems, because of the fact that in many practical problems u(t) or x(t) or both must be confined to closed regions in their corresponding Euclidean spaces. However, this method of Valentine received virtually no attention outside mathematics circles at the time. Interest in these results became widespread among engineers only after current engineering problems, involving such things as minimum-response-time servo systems and thrust control of intercontinental or interplanetary rockets, stimulated the development of two new approaches to problems of this type: Pontryagin's maximum principle (Refs. 10 through 15) and Bellman's dynamic programming (Refs. 17,18). An optimal control problem, in the sense used here, is a problem in which a controlling variable must be chosen so as to produce some desired effect and so as to optimize (i. e., maximize or minimize) some performance or loss functional.

5 1. 2. 2 Pontryagin's Maximum Principle. After its original formulation, applicable to time-optimal problems (Refs. 10, 11), the maximum principle was extended to a formulation suitable for use with much more general "cost" functions (Refs. 12, 13), then to a version applicable to problems with bounded state variables (Refs. 13,14), and, most recently, to an integral formulation (Ref. 15) that applies to problems in which the admissible controls are restricted to closed sets in some function space (rather than being restricted, moment by moment, to some set in Euclidean space). Pontryagin and his associates have relied for the most part on variational methods to prove their results, using a particular kind of constrained or "one-sided" variation in place of the "two-way" variations of the classical calculus of variations. Berkovitz and others have shown (Refs. 19,20,21) that many of these results can be obtained (under conditions on the system and on the class of admissible controls which are more restrictive than those assumed by Pontryagin and Gamkrelidze) from the classical calculus of variations using Valentine's method. (See Section A. 3. 2 of Appendix A). In fact, Berkovitz's results go beyond those of Gamkrelidze, in that they also apply to problems with joint constraints on the values of x(t) and u(t) rather than separate constraint sets for x(t) andu(t). Here again, however, Berkovitz's conditions are more restrictive than those of Gamkrelidze. The major contributions of Pontryagin and his associates to the optimal control field fall into three categories: (a) Mathematical results: The new theoretical contributions (i. e., those not contained or implied in previous works) are relatively few, the major contribution being not the maximum principle itself, but the proof of the maximum principle under less restrictive conditions on the system differential equations than those imposed by Bliss and other earlier workers (see Ref. 22, p. 4, and Section A. 3. 1 of Appendix A). Another contribution of considerable importance lies in the Pontryagin formulation of optimal control problems. By defining the control variables as separate entities, distinct from the state variables and subject to different types of constraints (a distinction that is not made in the usual calculus of variations formulation), Pontryagin has been able to simultaneously reduce the order of the

6 system of differential equations that must be considered, provide a concise notation for expressing the necessary conditions for optimality, and lay the foundation for analogies with Hamiltonian mechanics. (b) Optimal control philosophy: Classical calculus of variations tends to center its attention on the cost functional — the quantity to be minimized —relegating everything else to the status of "side conditions. " Pontryagin's approach, in common with other modern treatments, puts more emphasis on the physical system (represented by its differential equations). For many practical engineering problems, this latter orientation seems more appropriate, since the system is an objective thing (presumably known or knowable), while the "cost" is much more subjective and open to choice. This orientation leads plausibly to the problem of determining the optimal control for the same system with various cost functionals, to the so-called inverse problem of optimal control, and other practical considerations. (These points are discussed in more detail below. ) Furthermore, the emphasis that the Pontryagin approach places on the control variables is appropriate to engineering problems, where the specification of these variables is often the primary goal. (c) "Popularization": There can be no doubt about the fact that the work of the Russian school has caused much attention throughout the world to be given to the whole field of optimal control and the calculus of variations, and has stimulated considerable effort in these areas. This is in itself a contribution of some importance. The maximum principle is not without drawbacks, of course. Some of these are discussed below. 1. 2. 3 Dynamic Programming. Dynamic programming, developed by Bellman (Refs. 17, 18) at about the same time that the Russian school was developing the maximum principle, serves to eliminate four disadvantages inherent in conventional variational methods: a) the two (or more) point boundary value problems that result from the application of such methods, b) the need for continuity (with respect to the state variables)

7 in the functions used to describe system behavior and in their partial derivatives, c) the indirectness of these methods (i. e., the fact that they work with auxiliary variables which are even further removed from "physical reality" than are the equations which presumably describe system behavior), and d) the inapplicability of such methods to stochastic problems. These points are discussed in some detail in Appendix B, which also gives a brief exposition of the method of dynamic programming as a computational technique. Discussion here will be limited to a statement of the fundamental idea behind the method: For cost functionals in which the total cost (the cost over the whole interval) is the sum of the costs on a partitioning of the complete trajectory into subarcs, each of the subarcs is itself an optimal trajectory connecting the end points of the subarc. Therefore, if the total interval is from t0 to t1, and if t' is some intermediate time, it is a property of optimal trajectories that the control action from t' to t1 must be optimal for that subarc, regardless of what happended before t'. This is a statement of the "principle of optimality." This principle leads to a computational technique which, in its discrete-time form, starts at the final time and computes the optimal trajectory in the vicinity of the final point and then works backward toward the initial time, step-by-step (i. e., subarc by subarc), tabulating the optimal trajectories at each step. As pointed out in Appendix B, this process yields a family of optimal trajectories with various initial and/or final conditions, from which is selected the trajectory with the desired boundary conditions. While dynamic programming, as described above, is applied mainly in the numerical solution of problems, the same approach can be used to generate a partial differential equation, the solution of which gives the optimal control for continuous-time optimization problems. One would expect that this technique, when applied to a problem to which calculus of variations is applicable, would turn out to be equivalent to the calculus of variations. This is indeed the case for suitably restricted classes of problems, and many authors (Bellman, Ref. 28, Kopp, Ref. 23, Dreyfus, Ref. 24, Pontryagin, et al., Ref. 13) have used this continuous-time version of dynamic programming to derive (with more or less rigor and with various restrictions on the class of problems considered) the Euler equations, the maximum principle, and/or other results of the calculus of variations.

8 This connection was further explored by Kalman (Refs. 25, 26), who has shown (Ref. 27) that the dynamic programming approach is analogous to the Hamilton-Caratheodory approach to the calculus of variations. As noted above, one of the major advantages of dynamic programming is that it can be used in problems to which variational methods are not applicable; for example, problems in which the derivatives required by the variational methods do not exist. However, as pointed out by Pontryagin, Boltyanskii, et al. [Ref. 13, pp. 69-73], dynamic programming does not have the rigorous mathematical foundation possessed by the variational methods, so that in some cases in which dynamic programming may be used (as a heuristic tool) there is no assurance that the results obtained are indeed optimal. These same authors go on to discuss a weakness in the continuous-time version of dynamic programming, which is that a certain function which appears in the derivation is required to be differentiable. They show examples of a very simple and basic sort (solvable by means of the maximum principle) for which this requirement is not met. The discrete-time computational version also suffers from a weakness that is not apparent from the discussion above and in Appendix B, which can be summed up as a problem of system modeling: In this computational technique, the system is characterized by difference equations. If the original system characterization is in terms of differential equations, it can happen that important aspects of the behavior of the system are lost in the conversion to difference equations.6 As a result of this, dynamic programming might in some cases produce a control which was far from optimal, without giving any indication that such an event had taken place. The following example, while admittedly somewhat strained in that it involves a differential equation which does not have a unique solution for certain initial conditions, nonetheless shows that drastic things can happen in the conversion from differential equations to difference equations: Note that the differential equation x = 2 xl 2 + u(t), with initial conditions x(O) = 0 has two unforced solutions (i. e., solutions for which the forcing function u(t) is equal to zero): x(t) = 0 and x(t) = t2. In contrast, the difference equation [x(ti + At) - x(ti)]/At = 2 x(t(i) + u(ti)] or x(ti + At) = x(ti) + At[2 x(ti)-2 + (ti)] has only the single unforced solution x(t) = 0 for the initial condition x(O) = 0. Suppose that we were given the problem of choosing u(t} so as to force this system from x(O) = 0 to x(1) = 1 while minimizing the integral of u (t). The differential equation formulation indicates that this can be accomplished with zero cost [i. e., u(t) = 0 throughout] but the difference equation formulation overlooks this possibility.

9 1. 2. 4 Direct Methods. Classical variational calculus and its modern extensions as well involve the solving of a set of auxiliary equations (which go under various names, depending on the formulation: Euler-Lagrange equations, Hamilton equations, adjoint equations, Lagrange-multiplier equations, etc.). There are also other optimization methods, lumped under the title of "direct methods," which do not involve the solution of such auxiliary systems, but instead determine (or approximate) the optimal control by working directly with the functional to be optimized. One important class of direct methods, of which the Ritz method and the method of finite differences are representative, involves three basic steps (see Akheizer, Ref. 72, or Gelfand and Fomin, Ref. 73): a) the construction of a minimizing sequence; that is, a sequence of controls [uj(t)] having the following properties: i) each control uj(t) is in some given metric space U, ii) each control uj(t) causes the system to satisfy the desired boundary conditions, iii) the "cost" associated with the controls uj(t) approaches as a limit the infimum of the "cost" over all controls satisfying i) and ii) as j approaches infinity; b) a proof that the elements of the sequence converge (in the metric defined for the space U) to a function (called the limiting function) which satisfies i) and ii); c) a proof that the limiting function has as its cost the infimum mentioned in step a). Fortunately, theorems are available which eliminate the need for carrying out each of these steps in every case. For example, step c) can be eliminated if the cost functional can be shown to be lower semicontinuous (in the metric defined for the space U) at the limiting function (see Gelfand and Fomin, Ref. 73, pp. 8 and 194). And, as another example, under conditions which are from a practical standpoint neither extremely restrictive nor difficult to test for the satisfaction of, the Ritz method produces a sequence which is guaranteed to satisfy all the conditions of steps a), b), and c). The reader is referred to the stated references, and the references given therein, for detailed expositions of these and similar methods. We limit ourselves here to the following statements: Both the Ritz

10 method and the method of finite differences involve the construction of a minimizing sequence by the minimization of a succession of functions (not functionals) of m variables, for progressively larger values of m. From a computational standpoint, this is a simpler problem than the original problem, involving the minimization of a functional, and as a result these methods have been used successfully in a number of practical problems. In fact, depending on the degree of approximation desired, the process can often be terminated after only a few steps. On the other hand, convergence can in some cases by very slow, so that these methods by no means eliminate the need for the indirect variational methods discussed above. The approach used by Faulkner (Ref. 40), Bryson (Refs. 41 and 42), and Kelley (Refs. 43 and 44) might also be considered a direct method in that the adjoint equations are not explicitly solved in these methods. Instead, a successive-approximation approach is used to approximate a control which causes the adjoint equations (Lagrange multiplier equations) and the maximum principle to be satisfied. These methods are primarily computational implementations of classical or modern variational methods, and are therefore classified here with the computational advances discussed in the next section. 1. 2. 5 Current Work. Since the development of dynamic programming and the maximum principle, work in the field of optimal control systems has tended to fall into three categories: a) Mathematical reformulations, extensions, and reinterpretations: the viewing of existing results in new ways; the proving of existing theorems by different or more elegant methods. b) Computational advances: improvement of existing computational methods and the development of new methods. c) Restatements and revision of basic goals: a recognition of the unrealistic nature of many optimal control problems, as posed; efforts to use optimal control theory in ways more meaningful to practical problems. These three categories are discussed in more detail below. 7Faulkner refers to his approach as a direct method. Faulkner refers to his approach as a direct method.

11 1. 2. 5. 1 Mathematical Reformulations, Extensions, and Reinterpretations. Considerable effort has been devoted to exploring the interrelationships between the calculus of variations, the maximum principle, and dynamic programming (as mentioned in preceding sections), resulting in a widespread appreciation of the common foundation underlying the whole of optimal control theory. Geometrical interpretations of the optimal control problem (Roxin, Ref. 28, Fligge-Lotz and Halkin, Ref. 29) have provided a great deal of insight into the nature of optimal trajectories and into the techniques for solution of optimal control problems. The key idea here is the concept of the reachable set: the set of all points in state space that can be attained in a given time starting from a given initial point and using controls from the admissible set. With this interpretation, the Lagrange multiplier vector j[(t) of the calculus of variations and the maximum principle becomes simply the outward normal to the reachable set, the principle of optimality is a consequence of the fact that an optimal trajectory must stay on the boundary of this set, and the maximum principle is simply a statement of the fact that the Lagrange multiplier vector must have a nonpositive inner product with the optimal state velocity vector, which is tangent to the surface of the reachable set. Krassovski (Refs. 30,78), Kulikowski (Refs. 31-34), Kirillova (Ref. 35), Kranc and Sarachik (Refs. 36, 37), Neustadt (Ref. 77), and others have used certain results from functional analysis (Holder's inequality, principally) to solve linear time-optimal problems in which the admissible controls are restricted to closed sets in some function space, rather than having their values restricted, moment-by-moment, to some set (however complicated) in Euclidean space. (This approach, because it is closely related to the subject matter of the present work, is presented in some detail in Appendix C. ) The contribution of these papers lies not so much in the problems that they are able to solve (which can be solved in other ways) or in increased convenience of obtaining numerical solutions (which is lacking), but in the simplicity and elegance of the mathematical proofs of optimality that they involve, and in their application of functional analysis concepts to a field which is after all the study of the extremization of functionals. Formulations involving more general function spaces than the Banach spaces used in these papers can be expected to follow.

12 In a treatment which involves elements of functional analysis in its formulation and proofs, and which utilizes geometric ideas throughout (including a generalized reachable set and many other ideas from the geometric interpretations mentioned above), Gamkrelidze (Ref. 15) has achieved a synthesis of many concepts and results which were previously only loosely related. The question of existence of optimal controls is also receiving some study (Markus and Lee, Ref. 38), but much remains to be done in this area. It is obvious that an optimal control cannot exist unless there exists at least one control which causes the desired conditions to be satisfied. Therefore, the work on the controllability of systems (Refs. 25,60,69) is also pertinent to this problem. The theory for linear systems is virtually complete, but little has been done on the controllability of nonlinear systems. Sufficient conditions for optimal control have also received some attention (Refs. 13,76). Here again, the results for linear systems are much more complete than those for nonlinear systems. 1. 2. 5. 2 Computational Advances. The advent of large-scale digital computers has made it possible to solve many optimization problems that were once too long or complex for solution by existing methods. Even now, however, problems of rather low dimension —10 or so —may sometimes outstrip the capabilities of the largest and fastest computers available. Thus, there is frequently a need for computational techniques which reduce computation time or storage requirements, relative to straightforward implementations of the basic mathematical methods. In the case of dynamic programming, which typically requires only a modest amount of arithmetic capacity but large amounts of storage, this has led to methods for reducing storage requirements at the expense of extended computing time. This and other computer programming techniques have made possible the solution of engineering problems of considerable size by means of dynamic programming. The two-point boundary value problems which result from the application of variational methods must be solved iteratively, in general. To this end, interest in the method of gradients, or method of steepest descent, first applied to variational problems in 1908 by Hadamard (Ref. 39), has revived. -recent work on the subject having been done

13 by Bryson, et al. (Refs. 41 and 42) and Kelley (Refs. 43,44). Graham (Ref. 45) has generalized this technique to apply to "multiple-arc" problems —problems in which a finite number of discontinuities exists in the time derivatives of the state variables. 1. 2. 5. 3 Restatements and Revision of Basic Goals. As can be seen from the previous discussion, much of the work that has been done in optimal control theory is highly theoretical and mathematical in nature. In applying these results to practical engineering problems, it is often necessary to make many simplifying assumptions (or just plain guesses). The question then arises, "What good are techniques which give very precise and often intricate solutions to mathematical formulations which are only rough approximations of the real problem/'?" One of the principal areas of difficulty involves the cost function itself. Corresponding to each cost function there is a specific optimal control (or set of optimal controls), but the choice of cost function is often a highly subjective matter, and it may be difficult or impossible to incorporate all the important factors into a single mathematical function in a meaningful way. (A similar statement could be made about the constraints on the state and control variables, also.) One attempt to overcome this is the use of a vector-valued cost function (Zadeh, Ref. 46), which measures each control by a number of different criteria. Zadeh argues that, while such an approach may not produce a complete ordering of the controls (i. e., there may be no control which is as good as or better than every other control with respect to all of the criteria), it may eliminate a good many controls from consideration (as being not optimal on any count). It may then be possible to define a single cost functional which is meaningful over the class of controls which survive the first test, and in any case the system designer is better able to see how the various factors affect the total cost —a relationship that is often obscured when a single cost functional must be specified a priori. Another approach is the study of what is usually called the inverse problem of optimal control (Refs. 47,48, for example): Determine the class of problems for which a given control or control law or type of control law (e. g., a linear control law) is optimal. If this class is large enough to encompass the uncertainties in the mathematical model and the

14 different cost functions that can meaningfully be applied, then one is justified in using that control or control law. Many systems of engineering importance include significant stochastic variables, measurement errors, or unavoidable computation errors, which render the typical calculus of variations solution to the completely deterministic control problem inappropriate or impractical. Therefore, work is being done on optimization and prediction techniques applicable to stochastic problems (Refs. 49, 50, 51, to name a few). Also, the determination of control laws, which specify the control that should be used as a function of some or all of the state variables, is being emphasized (Refs. 52, 53). In large or complex systems (complete chemical factories or public utilities systems, for example), another factor becomes important: While it may be possible to find an over-all optimum control policy for the whole system, such a policy might become so complicated and interrelated as to require excessive "hardware" or computer capacity for implementation; it may have an adverse effect on stability under parameter changes; etc. A more practical approach (and hopefully yielding results not too far from the over-all optimum) would be to optimize subsystems, which presumably require simpler control systems, and then have one or more levels of supervisory control systems to set goals for the subsystems. Such multilevel control is being investigated in many of its aspects (Refs. 54, 55, 56). 1. 3 The Literature and the Minimum Peak Amplitude Problem With the exception of the functional analysis approach, the optimization techniques discussed thus far involve minimization of some functional expressible as an integral (or a summation). In the problems solvable by these techniques, the total cost is, in general, affected by every segment of the trajectory, however small. In the minimum peak amplitude problem, on the other hand, the total cost is not expressible as an integral, and may be determined by the magnitudes of the control variables on a very small subinterval of the total interval, with the controls at all other times having no (direct) effect on the cost. Thus, of the available techniques, only the functional analysis approach is directly applicable to the mini mum peak amplitude problem.

15 In the present work, functional analysis methods are used to derive the basic results. These results are then shown to be closely related to those of a certain time optimal problem studied by Krasovskii (Ref. 78). The significant differences between the two problems are discussed. Finally, a limiting process used by Kirillova (Ref. 35) is generalized to the problem at hand, which leads to a technique that is applicable to certain nonlinear systems.

CHAPTER II THE LINEAR PROBLEM 2. 1 Detailed Statement of the Problem The problem to be stated in this section already includes some of the generalizations noted in Section 1. 1. Other generalizations and variations will be introduced at appropriate points in the discussion. The reasons for and the implications of the various assumptions and restrictions made here are discussed in the.section immediately following this statement of the problem: Consider a time-varying deterministic linear dynamic system with input or control vector8 u = u(t) = [u(t),.. u (t)]T, state vector x = x(t) = [xl(t), x (t)]T, T and output vector y = y(t) = [Yl(t),.., yk(t)] related by x = A(t) x + B(t) u (2. 1) (t) = D x(t) (2.2) where x is the time derivative of the state x; A(t) and B(t) are given n x n and n x r matrices, respectively; the elements of A(t) and B(t) are bounded measurable functions of time defined on the interval T = [to,t1]; and D is a given constant k x n matrix (1 < k < n) of rank k. The matrices A(t) and B(t) are further assumed to be such that the output y is completely controllable on the interval T; that is, it is assumed that corresponding to every point a in n-dimensional Euclidean space En and every point b in k-dimensional Euclidean space Ek, there exists at least one bounded measurable control defined on T which causes the system to 8 To conserve space in the body of the text, column vectors such as u and x will often be written out as transposed row vectors, the superscripted T indicating the transposition operation. 16

17 transfer from the given initial state x(t0) = a to any state for which the output y takes 9 on the the value b at specified time tl. This complete controllability assumption, for which a necessary and sufficient condition is given in Appendix D, is different from the one used by Kalman and others (Refs. 25, 60, 69). This point is further discussed below. In terms of these definitions and assumptions, the minimum peak amplitude problem to be considered initially can be stated as follows: From the set U(a, b, T) of bounded measurable controls u(t) which cause the system characterized by Eq. 2. 1 to transfer from the given initial state x(t0) = a at given initial time to to any state for which Y(t1) = b at given final time t1, choose as an optimal control u any one for which the cost functional C(u) = teu IG(t)u(t)l (2.3) is a minimum with respect to all u e U(a,b, T). Here G(t) is an r x r matrix of bounded measurable functions defined on T, the inverse of which exists for all t in T and also consists of bounded measurable functions defined on T. As in Section 1. 1, the symbols | denote the Euclidean norm of the vector enclosed. 2. 2 Discussion of Assumptions and Restrictions The assumptions and restrictions made in the above problem statement are discussed and explained below: The restriction that A(t) and B(t) consist of bounded measurable functions is sufficiently weak to allow a large number of physical systems to be modeled by Eq. 2. 1, and yet strong enough to guarantee that the solutions of this equation can be written in a particularly simple form — a form which plays an important role in the results obtained here. This restriction could be relaxed in minor ways (e. g., by requiring only essential boundedness instead of boundedness) if desired, but such relaxed conditions are of little interest in most practical problems. 9 The original problem was stated in terms of achieving a certain goal at a specified final time, and therefore the complete controllability assumption has been formulated in the same terms. The implications of this assumption are discussed in the next section. 10 This is discussed in detail in Section 3. 1.

18 The matrix G(t) allows different weighting factors to be applied to the various terms which appear in the cost functional, and allows these factors to be varied with time. In the rocket problem mentioned in Section 1. 1, the flexibility provided by the inclusion of this matrix was not needed, since the thrust was simply the Euclidean length (norm) of the vector u. In such a problem, G(t) is set equal to the identity matrix. By including G(t) in the initial formulation, we can apply our results both to this problem and to other problems with more general cost functionals —for example, to certain electrical networks in which it is desired to minimize the maximum instantaneous input power and in which the components of u represent the current and/or voltage inputs to the circuit. The components of G(t) are then related to the input resistances and/or conductances of the circuit, in such a way that the expression uT(t) G (t) G(t) u(t) (the square root of which is the quantity involved in the cost functional) becomes the total input power at time t. The matrix G(t) is required to be invertible in order to avoid the possibility of degenerate problems. Such problems are excluded from consideration here because of the mathematical difficulties that they present. This is not to say that degenerate problems are not valid problems —they may arise naturally when one or more of the control variables "costs nothing, " i. e., is available in such quantities that it may be applied in unlimited amounts without incurring any cost. Examples might be the use of atmospheric air and sunlight as control variables in a chemical process: compared to that of the other input variables, the cost of these might be so small that it could be ignored. The cost functional has been defined in terms of the supremum, but for most practical purposes we could use the essential supremum just as well. For reasons of convenience, the results derived here are obtained using a cost functional defined A degenerate problem is one in which the boundary conditions can be satisfied by a control which is different from zero on a set of measure greater than zero and which nonetheless results in zero cost. If G(t) has a nontrivial null space, then it could happen that the set U(a,b, T) contained a nonzero control u(t) which lay entirely in this null space at each moment of time, resulting in G(t) u(t) being identically zero. This would imply in turn that the cost C( u) was zero.

19 in terms of the essential supremum. The optimal control for such a functional turns out to be identical to that for the functional originally specified, except for possible differences on sets of measure zero. We shall consider all controls which differ from each other only on a 12 set of measure zero as being equivalent, and therefore in this problem there is no significant difference between these two cost functionals. The specification of the final conditions on the system in terms of a final condition b on the output vector y is actually a statement to the effect that the final value of the state x(t1) must lie in some (n-k)-dimensional linear manifold M in n-dimensional state space En, where M is defined as the set of all points x(tl) e En for which D x(tl) = b. The formulation in terms of the output y lends itself to application in a number of practical problems. For example, if the system in question is a linear bandpass filter, some of the state variables (those representing internal voltages or currents, for example) may be of no direct concern in a given application. The matrix D is then chosen so that the "output" consists of only those state variables which are of interest (or some desired linear combination of them). If the problem is one of steering a projectile to intercept a moving target in three-dimensional space, only the position of the projectile at the moment of interception (and not the velocity, angular velocity, etc. ) might be important —a fact which could be taken into account by an appropriate choice of D. In such a case,the final condition b would represent the position that the projectile must occupy if the interception is to take place at the desired final time t1. With this idea in mind, we can see that the requirement that the output be completely controllable on the interval T is less restrictive than the complete controllability condition often imposed (Refs. 77,78), in that those state variables which do not enter into the output vector y (through the matrix D) need not be controllable at all. We require controllability of only those parts of the system which concern us. However, in another sense this type of complete controllability is more restrictive than that defined by Kalman (Ref. 26) 12As we shall see in the next chapter, the control functions enter into the expression for the output only in the integrand of a Lebesgue integral. Thus, control functions which differ only on a set of measure zero all yield the same output. See Natanson, Ref. 79, p. 137.

20 in that it requires that every possible output be attainable at precisely the specified final time t1, (rather than merely at some time greater than t0). Kalman's definition of controllability is appropriate to minimum-time problems and other problems in which the final time is unspecified, but is of little use in a problem with fixed final time. To illustrate this point as defined above, but was completely controllable in Kalman's sense; i. e., for any given initial state and final state, there exists some time t' at which the final state can be achieved. If the smallest possible value of t' is greater than the specified t), then obviously the requirements of the original problem can not be met, since we can not achieve the desired goal at the specified time tl. On the other hand, if it is known that a t < tl exists, it does not necessarily follow that the goal can be achieved at t (and achieving the goal at t may be the essence of the problem, as in the swinging of a baseball bat across the plate at the precise moment when the baseball thrown by the pitcher is crossing the plate and in the proper position to be hit past the second baseman on the hit-and-run play)o To see that complete controllability at tT does not necessarily imply complete controllability at t > ta, consider the following (admittedly rather artificial) example: Let ther le system in question have two parts, a completely controllable part characterized by the state variables x1(t),..., x(t) and an uncontrollable part characterized by the state variables xm+ (t),.., x (t). Let the output matrix D be a continuous matrix function of time (as is done in Section 3. 8 below) such that on the interval [t0,ti] the output y(t) involves only the controllable state variables xl(t),.., xm (t) and such that on the interval [ti,co) the output y(t) involves only the uncontrollable state variables x n(t),..,x (t). (Here t'i is larger than t' to allow for a continuous transition from one form of D to the other. ) This system is clearly not controllable for any tj > tV', but is completely controllable for all tl in (to,ti]? The complete controllability condition stated in Section 2. 1 is essential to all the results obtained here for the linear problem in that it plays a key role in the existence theorem of Section 3. 2, which is in turn needed in all other results. Problems which do not 13 See Kalman, Ref. 26.

21 satisfy this condition may well be valid and important problems, but they generally cannot be solved by the methods presented here. Two possible variations of the problem, as posed, will now be discussed: A more general system characterization than that represented by Eqs. 2. 1 and 2. 2 is one in which the control variables influence the output not only through their influence on the state variables, but also directly, i. e., Eq. 2. 2 is replaced by y(t) = D x(t) + E(t) u(t) (2. 4) where E(t) is a k x r matrix function of time defined on the appropriate interval which is 14 bounded in norm at each time t in this interval. Such a situation can arise even in very simple cases. An example is the idealized system shown in Fig. 2. 1. For this system, the equations corresponding to Eqs. 2. 1 and 2. 2 are x = 1 1 RC + RCu y = -x + u The equation for the output y is obviously of the form of Eq. 2. 4 and not that of Eq. 2. 2 state Hx(t) t C,1C input u(t) R output y(t) I IO Fig. 2. 1. An example in which the input directly influences the output. 4The norm IE(t)|I of the matrix E(t) is here defined at each moment of time as IsuP I E(t)u where as always the symbols I I denote the Euclidean norm of the enclosed - vector. This system is idealized in the sense that lead inductances have been neglected. Inclusion of a lead inductance in series with the input capacitor would eliminate the effect described here. Nonetheless, this situation can be closely approximated in practice.

22 Another variation that has some practical applicability involves problems in which both the initial state and the final state are specified as lying in linear manifolds in state space. Both of these variations include interesting and useful problems, but they will not be treated in detail here because of the increased complexity that they introduce into the derivations and proofs. However, at appropriate points in the discussion, comments will be made to show how the techniques used here can be extended to apply to these variations.

CHAPTER III SOLUTION OF THE LINEAR PROBLEM 3. 1 Preliminary Definitions and Results The basic results of this chapter are presented as a series of lemmas and theorems. Before proceeding with these, we make certain definitions and note certain results from the theory of linear differential equations which will be needed in the remainder of this work: For A(t), B(t), and u(t) consisting of bounded measurable functions defined for t e T, the solution x(t) of the vector-matrix differential equation x = A(t) x + B(t) u with boundary condition x(t0) = a can be written as t x(t) = X(t,t0)a + f X(t,s) B(s) u(s) ds (3.1) to where X(t, s) is an absolutely continuous invertible n x n matrix, the solution of the matrix differential equation dt X(t,s) = A(t)X(t,s); t,s E T with boundary condition X(t,t) = I (the identity matrix) for all t e T, (see, for example, Coddington and Levinson, Ref. 57, Chapters I and II). Since the emphasis of this work is on optimization problems, rather than on the solution of systems of linear differential equations, X(t, s) will be assumed to be known in most of the following. 16 If the matrix A is constant, independent of t, this assumption presents no great difficulty, since straightforward methods for obtaining X(t,s) are available in such cases (Refs. 57,75). We note, however, that for arbitrary A(t), the determination of X(t, s) in analytical form may be difficult or impossible, so that for such problems the assumption that X(t,s) is known is not to be taken lightly. In such cases, numerical solution is often the only practical approach, and even this can be very complex. 23

24 Applying the above expression for x(t) to the problem at hand (see Section 2. 1), and noting that y(t1) = Dx(t1), we see that the desired final condition y(t1) = b will be satisfied if and only if u(t) is such that the equation (t) = b = D[x(t,t0)a + f X(t1,s)(s ) (s)ds] (3. 2) is satisfied. (Here and in the sequel the integral sign with subscripted T implies integration over the interval T = [tt1] ). This problem will now be restated in a form more convenient for analysis: Upon defining17 g = b - DX(tl,t0)a (3.3) V(t1,s) = DX(t,s) B(s) G l(s) (3.4) z(t) = (t)t u(t) (3.5) 1l = ess. sup. Iz(t) (3.6) we can rewrite Eq. 3. 2 as g = f V(t, s)z(s) ds (3.7) T By assumption, G(t) is a matrix of bounded measurable functions, the inverse of which is also a matrix of bounded measurable functions. Therefore, every bounded measurable u(t) corresponds to a unique bounded measurable z(t), given by Eq. 3. 5, and every bounded measurable z(t) corresponds to a unique bounded measurable u(t), given by u(t) = G (t) z(t). There is, therefore, a one-to-one correspondence between functions u(t) satisfying Eq. 3. 2 17 Here we use the essential supremum rather than the supremum (as originally specified) in the definition of the norm, because the essential supremum is here more convenient from a mathematical standpoint. As noted in Section 2. 2, these norms are considered to be equivalent for purposes of this work, in that any control which is optimal with respect to the essential supremum norm can be converted into a control which is optimal with respect to the supremum norm simply by reducing the amplitude of the control at any moment at which it exceeds its essential supremum to a value not exceeding this essential supremum.

25 and functions z(t) satisfying Eq. 3. 7, which establishes the complete equivalence of the original problem formulation (see Section 2. 1) and the following formulation: From the set Q(g) of bounded measurable vector functions z(t), t c T, which satisfy Eq. 3. 7 for a particular choice of _g (determined from a and b be means of Eq. 3. 3), 18 - choose as an optimal control z(t) any one for which 11ll < 1 z1ll for all z Q(g) (3.8) The complete controllability condition on the original system (see Section 2.1) likewise has an equivalent formulation in terms of these definitions, as follows: Lemma: The system defined in Section 2. 1 has an output which is completely controllable on the interval T a) if and b) only if for every g e Ek, there exists at least one bounded measurable vector function z(t) satisfying Eq. 3.7. Proof: The proof of part a) follows simply by taking u(t) = G l(t) z(t). Part b) is proven by assuming that for some g there exists a bounded measurable u'(t) satisfying Eq. 3. 2, but that there is no bounded measurable z(t) satisfying Eq. 3. 7. But z'(t) = G(t) u'(t) is a bounded measurable function satisfying Eq. 3. 7, which is a contradiction. Q. E. D. We note for future reference that any bounded measurable control differing from the z(t) of this lemma only on a set of measure zero will also satisfy Eq. 3. 7 so that this lemma actually guarantees the existence of at least one class of controls - a class of controls which differ only on sets of measure zero. 3. 2 Existence of an Optimal Control Whether or not an optimal control exists for a given optimal control problem is a question not only of theoretical interest but of considerable practical importance as well, since from an engineering standpoint it is important to know whether or not we are seeking to do the impossible when we try to find an optimal control. In clarification of this 18 For convenience, z(t) and u(t) will both be referred to as "controls" in this work. The distinction in any particular case will be clear from the context.

26 point, let U(a,b, T) be the set of admissible1 controls which cause the system in question to satisfy the boundary conditions x(t0) = a, y(t) =b, and consider the two possible situations in which an optimal control may fail to exist: a) the set U(a, b, T) may be empty, a possibility which is ruled out for problems considered in this work because of the complete controllability assumption; and b) U(a, b, T) may contain an ininite number of controls but the cost functional C( ) may not take on a minimum over this set, i. e., there may be no control u(t) such that C(u) = infimum / C(U) = u E U( ab, T) C(u) [If U(a,b, T) contains only a finite number of controls, then clearly the infimum is taken on for some u e U( a, b, T) and hence an optimal control exists. ] In the second situation, no matter what control we choose, there exist other controls in U(a, b, T) with lower cost. The engineering solution to such a problem is to use 20 a control with cost sufficiently close to the infimum noted above, rather than seek a true optimal control. In this section a theorem is presented which guarantees the existence of a minimum peak amplitude control for the specified-final-time problem presented in Chapter II. This theorem is important not only for the reasons stated above, but because it underlies all succeeding results in this chapter as to the form and properties of minimum peak amplitude controls. We alert the reader to the fact that statements below such as "Let z(t) denote any one of the minimum peak amplitude controls..." are meaningful only because of the existence theorem given here. Such statements occur again and again in the proofs and examples to follows. 19 What constitutes an admissible control of course varies from problem to problem. A typical definition might be the set of vector functions of time defined on the interval T which are piecewise continuous and have values lying in some prescribed set in the appropriate Euclidean space at each moment of time t e T. Another is the one given in Chapter II for the class of problems considered in this work. 20 How close this must be will of course depend on the problem at hand. Typically, other factors, such as ease of generation of the proposed control, are brought into play to help determine the final choice of control.

27 Theorem 3. 1: Every linear system of the type described in Section 2. 1 has a minimum peak amplitude control for every set of finite boundary conditions n k x(t0) = a, y(t) = b; a E, b E Discussion: Every pair of boundary conditions x(t0) = a, y(t1) = b determines a specific g by means of Eq. 3. 3. Furthermore, there is at least one pair a, b (namely a = 0, b = ) which yields any given value of g. In light of this and the remarks of Section 3. 1, it is clear that Theorem 3. 1 will be true if and only if, for each system of the stated form, and for every k g e Ek, there exists a control z(t) satisfying Eq. 3. 8. 21 The existence of such a z(t) will be shown in a series of steps as follows: a) Controls z(t) will be considered as points in a Banach space Z with norm defined by Eq. 3. 6, and Eq. 3. 7 will be shown to be a linear continuous open mapping L(z) of k such points into points in Ek b) It will be shown that the closed unit hypersphere R1 (often called the closed unit ball) in Z maps into a closed convex absorbing set S1 in Ek, the boundary points of which are the images of points on the boundary of R1. c) The hypersphere in Z will then be expanded (or contracted) until the goal g lies precisely on the boundary of the cork responding image set in Ek. This point is then the image of a point on the surface of the hypersphere in Z, and the existence of the desired z is thereby established. 2Since the minimum peak amplitude problem is a special case of the problem treated by Neustadt in his Theorem 1, Ref. 77, (although different from any of the special cases he considered), this result follows immediately from said theorem. Certain of the conclusions of the theorem in the next section likewise are immediate consequences of the same theorem. The proofs presented here use many of Neustadt's arguments, but in a different order and with a somewhat different emphasis. The existence proof below is based on a suggestion by Prof. W. A. Porter of The University of Michigan, but follows Neustadt on the key point of the compactness of the image set in Ek.

28 Proof of Theorem 3.1: a) The properties of the mapping defined by Eq. 3.7: Consider all essentially-bounded measurable r-component vector functions of time defined on T. Group these vector functions into equivalence classes according to the equivalence relation that functions which differ from each other only on a subset of T of measure zero are in the same class. Let z(t) denote a representative member of the generic equiv22 alence class, with norm defined by Eq. 3. 6. This space Z is a Banach space2 -- an r-dimensional version of the familiar L space. For V(t1, t) a k x r matrix of bounded measurable functions of t, the operation J V(t, t) z(t) dt T defines a linear mapping L(z) of points z in Z into points in Ek. It is, in fact, a mapping of the space Z onto the space Ek, since, by the complete controllability assumption (as restated in the Lemma of Section 3. 1), there exists at least one z(t) E Z which maps into any given g E. The norm of this mapping is defined as lILll = sup L(z) - sup IJ V(t t) z(t)dt. Using the definition of the Euclidean norm of a vector (see Section 1.1), the inequality v z < _v| Izl (which is valid for all r-component row vectors v and r-component column vectors z), and the fact that, if li zl = 1, I z(t)| < 1 for almost all t e T, it is easily shown that I| LI < (t - t0) kr |vij sup where Ivijsup is the supremum over all t e T and over all i,j (i = 1,...,k; j = 1,...,r) of the absolute values of the elements vij(tl,t) of the matrix V(t1,t). Since every element of 22ee Appendix E for a further discussion of this space. See Appendix E for a further discussion of this space.

29 V(tl,t) is bounded for all t E T, because of the assumption of Chapter II, Ivij Isup < oo and therefore L is bounded. This implies that the set S1 is bounded also. From the fact that S1 is bounded, ( and from Thm. 47. A, Ref. 80, which states that if the image under a mapping from one normed linear space to another of the closed unit sphere is bounded, then the mapping is continuous), it follows that L is a continuous mapping, i. e., if z e Z and g E are such that L(z) =g, then for every neighborhood N ofg in Ek there exists a neighborhood N of z of Z such that N is contained23 in L(N ). g z - g z But since L is a continuous linear mapping of Z onto E k, it is also an open mapping (see Thm. k 50. A, p. 236, Simmons, Ref. 80), i. e., every open set in Z maps into an open set in Ek b) The image in Ek of the unit hypersphere in Z: Now consider the closed unit hypersphere R1 centered on the origin of Z, and let S1 be the image in Ek of R1 under the mapping L. That the set S1 is convex can be shown as follows: Let s1 and s2 be any two points in S1, and let z1 and z2 be any two points in R such that L(z1) =s1 and L(z2) =s2. Because and 2 are in R1, 11 Zl < 1 and 11z211 < 1. Convexity of S1 will have been shown if we can show that 0s1 + (1-0) s2 e S1 for all real numbers 0, 0 < 0 < 1. Denote 0s + (1-) s2 by s. Since s = L(z1) + (1-0) L(z2) = L(0z1) + L[(1- ) 2] = L[0zl + (1-0) z2], because of the linearity of L, and since 1 + (1-0)z2II 1 1 + II(1-0) 2 = ll z II + (1-8)1 21 I< + (1- = 1 by the triangle inequality and the bounds on 11 z1 1 and 11z2 11 noted above, it follows that s is indeed the image of a point in R1, and hence that s S. The set S1 is also symmetrical about the origin in Ek (which implies that if s e S1, then -s e S1), since L(-z) = -L(z), because of the linearity of L, and -z e R1 if z R. - 1. The set S1 is an absorbing set (i. e., for every nonzero vector g e Ek, S1 contains at least one vector s with Euclidean norm bounded away from zero which is colinear -g 2The symbol L(Nz) represents the image of the set Nz under the mapping L; that is, L(Nz) is the set of all points in Ek which are images under L of points in Nz. This notation for the image of a set is used throughout this work.

30 with g) because of the complete controllability assumption. To show this, we exhibit one such vector s = L(z ), which we obtain by normalizing (to unit norm in Z-space) the control -g -g used in Appendix D to prove the sufficiency of the condition for complete controllability on the interval T. This procedure yields zg(t) = K vT(tt)W1 -g= L(g) = J V(t t) V (tlt) W dt K Wv = and IS9 Klg/ isgl = K 1' where W is a constant k x k matrix given by w = f V(tl,t)VT(tl,t)dt T and K, the constant which adjusts z to unit norm in Z-space, is defined as g K ess. sup. TT(tt)W -1. - teT = V' t I Because of the complete controllability assumption (see Appendix D), W is positive definite, and therefore, its inverse exists and has bounded eigenvalues. Since every element of V (tl, t) is uniformly bounded in absolute value on T by the finite number vijv defined 1ij sup above, the matrix V (tlt) W-1 is uniformly bounded in norm24 on T. Denote any such bound by N. Then IVT(tl,t)W lI < N[gl on T, which yields K < N[lg, which in turn yields I| s >. Since N is less than infinity, s is a vector in the same direction as — g and with Euclidean norm bounded away from zero, as claimed. Every point of S1 which is on the boundary of S1 is the image of some point on the boundary of R1. This can be shown as follows: Assume that there exists some 24 The norm IIA I of any matrix A, operating on vectors x, is here defined as |Ai = sup JA XI A II Ix = IAx =.

31 boundary point s' of S1 which is in S1 and which is the image of an interior point5 z' of R. Because L is an open mapping, every neighborhood of z' maps into an open set containing s'. Since z' is an interior point of R, thereis a neighborhood of z' which is entirely contained within R1 which maps into an open set containing s'. But every open set containing s' contains points which are outside of S1. This is a contradiction, since every point of R1 maps into some point of S1, and, therefore, s' cannot be the image of an interior point of R1. Since s' is the image of some point in R1, by definition, it then follows that it is the image of a point on the boundary of R1, as claimed. Finally, the set S1 is closed (in terms of the usual Euclidean metric on Ek). To show this, we adopt for the moment a different point of view from that taken above: Consider the set of r-component measurable row vector functions of time v(t) defined on the interval T, the Euclidean norms of which are integrable on T. Define the norm of v(t) by llvlll = f |v(t)| dt and denote by B the vector space consisting of these functions with this norm. This is a 27 Banach space — an "r-dimensional" version of the familiar L1 space. The rows of the matrix V(tl, t), considered as functions of t, are elements of this space. The conjugate space B* of this space is the space of bounded linear functionals z(v) defined over B, with norm given by 11Z11 = sup z(v) z l IIll 11z I It is easily shown that all functionals of the form 2The set R1 has interior points, of course. For example, the open sphere of unit radius centered on the origin consists wholly of interior points of R1. 2General references for this part of the proof: Neustadt, Ref. 77; Hille and Phillips, Ref. 84, pp. 26-39; Simmons, Ref. 80, pp. 211-242; Taylor, Ref. 81, pp. 160-252. 2This space and the representation theorem for functionals on this space are discussed in Appendix E.

32 z(v) = f v(t)z(t) dt T are in the space B*, where z(t) is an essentially bounded r-component column vector function of time defined on T. Not so obvious, but nonetheless true, 28 is the fact that every functional in B* is representable in this form. Furthermore, the above definition of the norm on B* takes on a particularly 29 convenient form in terms of this representation, namely, = ess. sup. Iz(t)= l 11z11 = tE T lz^)[ lzll where 1l z 11 is defined in Eq. 3. 6. 30 Thus,every functional z in B* corresponds to an equivalence class in Z, and every equivalence class corresponds to a functional in B*. Furthermore, the norms of the corresponding members of the two spaces are the same. The two spaces are therefore isometrically equivalent to each other (Dunford and Schwartz, Ref. 85, p. 65), which allows certain conclusions which are drawn below about B* to be applied to Z also. Let B** denote the second conjugate space of B; i. e., the space of all bounded linear functional v**(z) defined on the elements z of B* (i. e., on the elements z(t) of Z), and then consider the natural embedding of B in B**, in which a correspondence is established between each element of B and an element of a subset of B** by means of the relation v**(z) = z(v) 28 Appendix E. See Appendix E. See Appendix E. 30The representation for a given functional is not unique, since any vector z'(t) which differs from z(t) only on a set of measure zero could be used in place of z(t). Thus, the situation is analogous to that encountered in the definition of the space Z above, and the kernel functions z(t) can be grouped into equivalence classes in exactly the same way as were the controls occurring in the definition of Z.

33 Now consider the weak * topology on B*; that is, the weakest topology on B* for which all the functionals in B** which correspond to elements of B under the natural embedding remain continuous. We now complete the proof that S1 is closed in three steps: 1) The closed unit sphere R1 of Z is a compact Hausdorff space in the weak* topology. This follows from Theorem 49.A, Simmons, Ref. 80, p. 233, (which states that the closed unit sphere of the conjugate space of a normed linear space is a compact Hausdorff space in the weak * topology), and from the fact that Z is isometrically equivalent to B*. 2) The continuous linear transformation L(z) defined above, which maps elements of Z (i. e., B*) into E, is continuous in the weak* topology on Z and the topology generated by the Euclidean norm on Ek. To show this, we note that L(z) actually consists of k functionals from B**, each of which maps B* into E1. By the definition of the weak* topology, each of these k functionals is continuous in the weak* topology. This implies (by the definition of a continuous mapping, Simmons, Ref. 80, p. 93) that for each of these functionals the inverse image of each open interval in E is an open set in Z in the weak* topology. Consider any one of these functionals v**(z), and any open interval M. on the ith coordinate axis of E, and let Qi be the open set in Z which is the inverse image of M. under the mapping v**. Then Qi is also the inverse image under the mapping L(z) of the "open strip" in Ek defined by [g:g e Ek, g G M], since v**(z) is just the ith component of L(z). The set of all such open strips for all i = 1,...,k forms an open subbase for E (Simmons,

34 Ref. 80, pp. 99-104). We have thus shown that the inverse image of each subbasic open set of Ek with respect to the mapping L(z) is an open set. This implies that L(z) is continuous in the stated topologies, as claimed (Simmons, Ref. 80, Theorem 18. E, p. 103). 3) The set S1 is closed (in terms of the usual Euclidean metric on Ek). To show this, we first apply Theorem 21. B, Simmons, Ref. 80, p. 111, which states that if L is a continuous mapping from a compact topological space into another topological space, then the image of the first space under this mapping is a compact subset of the second. Since, from part 1) above, R1 is a compact Hausdorff space in the weak* topology, and since from part 2) above, L is a continuous mapping of R1 into Ek, (in the topologies already mentioned for these spaces), it follows that S1 is a compact subset of Ek But from Lemma 7, c), Section I. 5. 6, Dunford and Schwartz, (Ref. 85), and from the fact that the metric space Ek is also a Hausdorff space (Simmons, Ref. 80, p. 134), it then follows that S1 is closed (in the topology generated by the usual Euclidean norm on E ). c) Determination of the minimum cost: k Now consider any given goal point g e Ek. This point is either inside, on the boundary, or outside of S1. If it is on the boundary of S1, it is in S1, since S1 is closedo Because every point of S1 which is on the boundary of S1 is the image of a point on the boundary of R1, and because there is no interior point of R1 which maps into a boundary point of S1, the existence of an optimal control z(t) is proven in this case, and 11 z = 1. If g is outside (inside) of S1, we choose a larger (smaller) closed hypersphere R in Z, centered on

35 the origin, such that its image S in Ek contains g as a boundary point. In precise mathematical terms, this process involves the evaluation of the Minkowski functional1 of the set S1 at the point g (denote its value at g by C. ) and then the choosing of R as the closed hypersphere in Z centered on the origin and with radius equal to C. Then S = L(R) contains g as a boundary point, S has the same properties of compactness, convexity, etc., as Si, and the reasoning used above for the case in which g was on the boundary of S1 again applies. Since S1 contains vectors in every possible direction which are bounded away from zero in norm, C will always be finite if IgI is finite. The existence of an optimal control z(t) with norm C is thus established. This completes the proof of Theorem 3. 1. 3.3 Some Generalizations At the end of Section 2. 2, two generalizations of the minimum peak amplitude problem, as originally stated, were discussed. In the first of these, a more general system description, in which the control entered directly into the output, was involved. Note from Eq. 3. 1 (which applies to these problems also) that the control can be perturbed arbitrarily at a single moment in time (which is of course a set of measure zero) without affecting the state x(t), but that for this class of problems such a perturbation does affect the output y(t) at that moment (see Eq. 2. 4). Note also that in the problem considered here the value of y(t) is important only at the final time t1, no requirements having been placed on it at any other time. This implies the following: Suppose we consider some control u(t) for which y(t1) is not equal to the desired goal g. If no component of the vector g - y(t1) lies in the null space of the matrix E(t) at time ti, then we can choose another control u'(t) which differs from u(t) only at the time t1 and which causes the corresponding y'(t1) to equal g; namely u'(t) = u(t) t E [to,t1) u'(t1) = u(t1) + Au where Au is such that E(tl) Au = g - Y(tl). In terms of the essential supremum cost functional, u(t) andu'(t) have the same cost. Thus, if this is the cost functional which is meaningful in a given problem, it is clear that the optimal control problem for systems of the type 3See pp. 134-136, Taylor, Ref. 81.

36 shown in Eq. 2. 4 can be solved in the same way as the problem as originally posed (Section 2. 1), if we first subtract from the goal any component of g which lies in the space spanned by the columns of E(t) at time t1. We leave this component of g to be achieved by an appropriate choice of Au at t = tl. If E(tl) happens to be of rank k at t = tl, then the optimal control consists only of a Au of the appropriate magnitude and "direction" at t = t; with the control being zero for all other t e T. The essential supremum of the Euclidean norm of such a control function is of course zero. If, on the other hand, we wish to use the supremum cost functional, the situation is different, since the required value of U|'(t1) may be greater than the supremum of (u'(t)I over the rest of the interval. We must therefore carefully choose u(t) on [t0, t1) so as to leave unaccomplished only that much of the over-all goal g as can be attained using a u'(tl) for which u'(t1)l does not exceed t e [tl) |0 u(t). Further reasoning along this line then leads to the following conclusions: The existence proof of the previous section can be modified to apply to this problem by replacing the image S1 of the closed unit hypersphere R of Z with a set obtained by constructing at each point of the boundary of S1 a hyperellipsoid corresponding to all the points that can be reached from that point as a result of the direct contribution of the term E(t) u(t) in Eq. 2. 4 using controls with norm not exceeding unity at this last instant. The union of S1 and all these hyperellipsoids is then the reachable set for the generalized problem, and all the rest of the steps in the existence proof go through with little change. To compute numerical values of the optimal controls for such problems, we can follow the procedure just outlined for the existence proof, or else use the following iterative technique: A guess C is made as to the value of the minimum peak amplitude, and the goal g is replaced by the set of all points in Ek from which g can be reached using a control u'(t1) for which Iu'(tl)l < C. The problem of choosing the optimal control on the interval [to, tl) to force the system into this set is then solved, and the corresponding peak amplitude C' is determined. If C' is less than the original guess C, the original guess is reduced and the procedure is repeated. If it is greater, the original guess is increased, and so on, until the two values are equal.

37 The second generalization mentioned at the end of Section 2. 2 involved initial states as well as final states in some linear manifold. This likewise presents no problems with respect to the existence of optimal controls, for the following reasons: If the initial state lies in some linear manifold M (rather than at a single given point), then the goal point g must likewise be replaced by a linear manifold, defined as all points in Ek of the form b - X(t,t0) x(t0); x(to0) M Such a manifold is a closed set. The step in the existence proof in which the reachable set S1 is expanded (or contracted) until it just touches g thus has an analog here, and the existence proof once again goes through with little change. Actual computation of optimal solutions is considerably more difficult, however, and is not investigated here. If the initial time to is not specified in advance, but must be chosen as part of the problem solution, it is no longer necessarily true that an optimal solution exists. The difficulties here are the same as those which occur when the final time is not specified in advance. These difficulties are discussed in Section 3. 8. 3. 4 The Form of the Optimal Solution The following lemma, which may be regarded as a vector version of Holder's inequality with p = oo, will be useful in the proof of Theorem 3. 2 below. Lemma: Let v(t) andz(t) be r-component vector functions3 of time defined on the interval T and belonging to the spaces B and Z (defined in the proof of Theorem 3. 1), respectively. Then f v(t)z(t)dt < C f Iv(t)Idt (3.10) T T where C = 1lzl = ess sup | z(t), and where T1 is the subset of T on which Iv(t)l / 0. Furthermore, the equality is taken on in this inequality if and only if z(t) satisfies 3Note that v is a row vector and z is a column vector.

38 vT(t) z(t) = C vTt) for almost all t e T (3.11) This lemma places no restrictions on z(t) on the set T - T1. Proof: j v(t)z(t)dt < | J v(t)z(t)dt < Jf Iv(t)z(t)| dt T T T < J |v(t)| Iz(t)|dt < C f v(t)|dt T T The first inequality becomes an equality if and only if f v(t) z(t)dt is nonnegative. The T second becomes an equality if and only if the scalar quantity v(t) z(t) is of the same sign T (excluding points at which it is zero). The third becomes an equality if and only if v (t) and z(t) are colinear almost everywhere on T. The last inequality, which is just the usual scalar form of Holder's inequality, becomes an equality if and only if z(t) = ess. sup. |(t)| = C almost everywhere on T1. These four conditions combine to show that the equality can be taken on in inequality 3.10 if and only if Eq. 3. 11 is satisfied. We note in passing that the condition of equality says nothing about z(t) on the set T - T1; i. e., the set on which | v(t)[ = 0. The implications of this fact for the problem at hand will be discussed later. Theorem 3. 2: Every minimum peak amplitude control z(t), with norm 11 z1 = C, the existence of at least one of which is guaranteed by Theorem 3.1, is of the form T V (t1,t)c z(t) = C vT(t ) for almost all t e T1(c) |v (tit) | 1 _ (3.12) On the set T - T1(c), z(t) is restricted only by the condition | z(t)[ ~ C for almost all t e T- T1(c) Here c is a nonzero vector in Ek and T(c ) is the subset of Ton which | VT(tl,t)c | 0. Moreover, c must be an outward normal to the set S at the point g, where S is the image under the mapping L of the set R = [z: z E Z, 1z|11 < C ]. Also, if the outward normal to

39 33 S at g is not uniquely defined, 33 then any one of the outward normals may be used as c. Finally, - sup (.>~ o.(c,g) c 0~ |V^ l, )c|dt T(3. 13) C- c/O fI vT(tl't)cdt T and this supremum is taken on only for c's which are outward normals to S at g. Proof: From Theorem 3. 1, we know that an optimal control exists for every goal g e E. Let the minimum peak amplitude corresponding to the given g be denoted by C, and let z(t) denote any one of the minimum peak amplitude controls yielding this g; i. e., L(z ) = g. Let c be any outward normal to the set S at the point g. (Because of the way S is constructed, g is on the boundary of S. ) Finally, let z(t) be any control in the set R, and let s = L(z). Clearly, s is in S. 34 Now form the inner product (c,g) of c and g. Then (cg) = s PS (c,s) (3.14) as can be seen by considering the hyperplane of support of S at g to which c is the normal, and noting that because of the convexity of S (established in the proof of Theorem 3. 1) no vector in S has a component normal to this hyperplane which is greater than that of g. Now, since every point in R maps into S, taking the supremum over all s e S is equivalent to taking the supremum over all z e R. Equation 3. 14 can thus be rewritten (c, L(z)) ( L(z Z e R - - or j -T V(t,t) z(t)dt = sup j T V(t,t) z(t)dt (3. 15) T - eRT 33Where S does not have a unique outward normal, such as at a corner, we say that any vector which is an outward normal to a hyperplane of support of S at that point is an outward normal to S at that point. 3Here and in the sequel,the inner product of two vectors _ and 1 in m-dimensional Euclidean space is denoted by (8, 1) and is given by m (,aZ) = Z ii i=l

40 Using V (t1,t)c as v(t) in the above lemma, we have that cT V(tl,t) z(t)dt < ess. TP. |]z(t)| JT IvT(tl,t)cldt T T1''T where T1 is the subset of T on which VT(tl,t) t i 0. Furthermore, the equality is taken on only if z(t) is of the form C VT(t, t)/ | VT(t1,t)c| a. e. on T1. Since we must find the supremum of this integral over all z(t) in R, it is thus clear that we must choose z(t) of the form C V(t1,t)c /| VT(t1,t)c a. e. on T1. Also, since z(t) must be in R, it follows that Iz(t)| < C a. e. on T - T1. Substituting this result into Eq. 3. 15 then gives Tf - -T V~VT(tlt)f c T CT V(t,t) z(t)dt = C T V(tlt) - dt C VTtlt)c dt T 1 T i |VT(tl,t)c| T -T But this is precisely the situation covered by the above lemma, with c V(t1,t) taken as the row vector v(t). Therefore, from this lemma, it follows that z(t) must be of the form vT(tl,t)c z(t) = C T(,t) for almost all t T1() Since z(t) was defined as any one of the optimal controls yielding g, the first part of Eq. 3. 12 is established. The second part of Eq. 3. 12 is obviously true, since the essential supremum of I z(t) is known to be C, and a function can exceed its essential supremum at most on a set of measure zero. We now show that no vector which is not an outward normal to S at g can be used in place of c in Eq. 3. 12: Let c be any vector in E which is not an outward normal to S at g, and note that max (Cs (c,g) < s - (Like Eq. 3. 14, this follows from the convexity of S: If c is not normal to S at ^, then it is normal to S at some other point, call it h. Then (c,_g) will be less than (c,h), by the same argument used to establish Eq. 3. 14. ) Repeating the steps used in deriving Eq. 3. 15 from Eq. 3. 14 then gives (cg) = J cTV(t1,t) z(t)dt < C j I(t1,t)cIdt T T

41 VT(tlt)c We now use the above lemma to conclude that z(t) can not be of the form C T (tt)' since if it were of this form the equality would be taken on in the lemma, which would contradict the above inequality. The final statement of the theorem is now easily proven: Since from the above inequality (c,g) f > v (tit)c dt T for all c which are not outward normals to S at, and since, from Eq. 3. 15, _- (cg) jC I (tt)cldt T for all c which are outward normals to S at A, the final assertion of the theorem follows at once. Q. E. D. Discussion: A comparison of the foregoing theorems and proofs with the results for other problems and approaches reveals a close similarity in the underlying concepts. For instance, the set S is clearly the reachable set at the time t1 for the given system, starting from the origin at time t, and for z(t) restricted by |z(t)[ < C. The fact that the optimal control involves c, a normal to this reachable set, is equivalent to the fact that the Lagrange multiplier vector 2(t) arising in the calculus of variations solution of many linear systems problems is normal to the final manifold at the final time. Finally, the fact that Iz(t) I = C on T1(c) reflects the fact that the trajectory lies on the boundary of the reachable set at such moments. The question naturally arises as to the nature of the optimal control on the set T - Tl(c). In fact, since c itself may not be uniquely determined (i. e., the reachable set may have a "corner" or "edge" at the point in question), a second question concerns the different optimal controls which result from different choices of c. These two questions will be investigated in turn. The examples in the next section show that if the set T - T1(c) has measure greater than zero, the optimal control is not uniquely determined, in general.

42 3. 5 Two Examples Involving Nonunique Optimal Control Example 1: The system to be considered here is characterized by x 0 x 01 1 b 1(t) b12(t)u 1 x2 J Lo 0 _xx2J 0 b2(tJ u2 where 1 l-t 0 < t < 1 bll(t) = < 0 1<t<2 0 0<t<1 b12(t) = < 12(t) = -(t-1)(2-t) 1 <t 2 r 0 0<t< 1 b22(t) = < t-1 l<t<2 The state x(t) is taken as the output for this example, so that D = I. We note in passing that the matrix B(t) is a continuous function of time on the given interval. From the set U(0,b, T) of bounded measurable controls u(t) defined on the interval T = [0, 2] which cause this system to transfer from the initial state x(0) = 0 to the desired final state x(tl) = b at specified final time tl = 2, choose as an optimal control any control u(t) for which C( u ) = es sSp | u(t)| takes on its minimum value with respect to all u(t) e U(0,b,T). For this system X(tl, t) = 0 1 Hence, for tl = 2,

43 1-t l0 F[t:Il O<t< 1 0 j0 V(t1,t) = V(2,t) = X(2,t) B(t) = 0 tUsing Eq. 3.1, x1(2) 1 1-t 0 u1(t) dt2+ 0 0 u(t) 1 = f A dt + J 1 dt x^2(2) 0 0 LJ0 u2(t) 1 L t-1 u2(t) i (1-t) ul(t) 2 0 = f dt + f dt (3.16) 0 0 1 (t-1) u2(t) Therefore, xl(2) = f (l-t)ul(t)dt 0 and 2 x2(2) = (t-1) u2(t)dt (3.17) 1 35 From these expressions it follows at once that the set S1 defined in the proof of Theorem 1 3. 1 (in this case, the set of all points x(t) in E which can be obtained from Eq. 3. 16 using u(t) for which I u(t)[ < 1, t e T) is the square S1 shown in Fig. 3. 1. Suppose that the desired final condition on x(2) is x(2) = b = = [0. 8, 0. 6] Following the procedure of part c) of the proof of Theorem 3. 1, or alternatively, using Eq. 3.13, we obtain that C = 1. 6. An outward normal to the set S at g is c = [1, 0] and its direction is uniquely defined as can be seen 35 Any point (g1, g2) inside or on the boundary of this square can clearly be obtained by choosing u(t) = [2gl,0]T, 0 < t < 1 and u(t) = [0, 2g2]T, 1 < t < 2, for example. Such controls satisfy the condition Iu(t) < 1, t e T. By applying the lemma of Section 3. 4 to Eqs. 3.17, we can also show that neither Ix1(2)1 nor 1x2(2)1 can exceed 2, that the value x(2) = can be taken on only if u(t) = [ 1,0] T almost everywhere on [ 0, 1], and 1alue x1(2) 2 carl ut: La~en on Ollly U U\- that similarly the value x2(2) = can be taken on only if u(t) = [0 1]T almost every~~where on [ a1, 2]. where on [l, 2].

44 from inspection of Fig. 3. 1. Using these values of C and c in Theorem 3. 2, we have that u(t) = z(t) = (1.6) [, 0 t < 1 I0 with u(t) being thus-far undefined on the interval 1 < t < 2 except for the requirement that u(t) < 1. 6 almost everywhere on this interval. The optimum control is not uniquely determined on this interval, and in fact there are an infinite number of optimal controls for thie particular choice of final condition. One family of optimal controls is given by (1.6) 0 < t < 0^ u(t) = { (1.6) 1 < t < 1 + K 1 [<t_ 2 3 where 4 < K < 1, as can be verified by direct substitution in Eq. 3. 16. Any number of other optimal controls could be devised. 2(2) S.8 S c 5 /s x1(2) -.8 -.5.(2) -.5 -.8 Fig. 3. 1 The sets S1 and S for Example 1.

45 Let us now consider the situation at one of the corners of S1. To be specific, choose the corner at (, A). Here the direction of the normal to S1 is not uniquely defined, so that all vectors of the form c = [cos 0, sin 0]T, 0< 90 <2, are outward normals here. It is interesting to note that, except for the two c corresponding to 0 = 0 and, = every one of these normal vectors yields the same control u(t) = [1, 0] T 0 < t < 1; u(t) = [0,1], 1 < t < 2, defined uniquely except at the point t = 1. (This can be verified by direct substitution in Eq. 3. 12 of Theorem 3. 2. ) Furthermore, this control is included in the set of incompletely specified controls determined by the substitution of the two limiting normals in Eq. 3.12 of Theorem 3. 2, so that the theorem is not contradicted. Finally, any control which differs from this control on a set of measure greater than zero can not yield x(2) = [, A], as indicated in Footnote 35 following Eq. 3. 17. In summary, this example has shown both a situation in which the direction of the normal to S (or S1) is uniquely determined but the optimal control is not, and a situation in which the direction of the normal is not uniquely defined but the control is (to within a set of measure zero). One must not conclude from this that the optimal control is always unique when the direction of the normal is not uniquely defined, as can be shown by the following example, which is similar to this one, but in three-dimensional space: Example 2: From the set U(O,b, T) of bounded measurable controls defined on the interval T = [0, 3] which cause the system characterized by 1 0 b11(t) b12(t) b13(t) x 0 0 1 x + 0 1 ~ x= O O 1 x+ ~ b22(t) 23(t) u 0 0 00 0 b33(t) to transfer from the given initial state x(0) = 0 to the final state x(t) = b at specified final time t1 = 3, choose as an optimal control any one for which C(u) = ess SPT u(t)I is minimum with respect to all u e U(0,b,T). Here,the coefficients bij(t) are all zero except as follows: b11(t) = (1-t) on the interval [0, 1]; b12(t) = -3(t-1) (2-t)(3-t) and b22(t) = 3(t-1)(2-t) on the

46 interval [1, 2]; b12(t) = 2(t-2) (3-t)2 b23(t) = -(t-2)(3-t), and b33(t) = t-2 on the interval [ 2, 3]. Since for this problem 1 X(tl,t) = O 0o tl-t ~ (tl-t)2 o tl-t 0 1 it is easily shown that 1-t 0 0 V(2,t) = X(2,t) B(t) = [ 0 0 - 0 0 0 0 o o 0 0 0 0oJ 0 3(t- 1) (2-t)0 0 O 0 0 0 0 0 t-2 1<t<2 2<t<3 Repeating the procedure of the first example, we obtain as the set S1 a cube centered on the 3 IL 1 1 origin of E and with vertices at (~2, +~, +i) in E. Equations analogous to Eqs. 3. 17 can then be analyzed to show that a) on any face of the cube, the situation is analogous to the first part of the first example —the normal direction is uniquely defined but it does not uniquely specify a control when substituted into Eq. 3. 12 of Theorem 3. 2, and the optimal control is,in factnot unique. b) on any corner of the cube the situation is analogous to the second part of the first example —the normal direction is not uniquely defined, but all normals (except the limiting ones —those which are normal to one of the edges or faces) uniquely specify (to within a set of measure zero) a unique optimal control.

47 c) on any edge of the cube a new situation arises —the normal direction is not uniquely defined, and the optimal control is not unique. However, even here all the different normals (except the limiting normals - those also normal to one of the faces) are equivalent, in that, upon substitution into Eq. 3. 12, they all determine the optimal control to the same degree —i. e., at moments in time at which they determine a specific control, all normals determine the same control, and all leave the control undefined at the same moments. These examples illustrate that the optimal control is not necessarily unique, that the existence of a unique normal to the set S1 (or S) is not a sufficient condition for the existence of a unique optimal control, and that the existence of a unique optimal control does not imply that S1 has a unique normal direction at the given point. However, as we shall see later, all three of these conclusions become invalid for suitably restricted classes of systems. These examples also pose an interesting possibility, which we note as an aside at this point: We can regard the cost functional C(u) as imposing an ordering on the controls in the set U(a,b, T). In these examples the given cost function does not necessarily impose a complete ordering on these controls. We are then free to impose a secondary cost function on the set of controls which are optimal with respect to the original cost function. (In fact, if we are to end up with a specific optimal control, we must apply some rule for deciding which of the optimal controls to use. ) For example, we might use a minimumenergy criterion to further order the optimal controls. Alternatively, however, we could use the freedom available to us to satisfy some other objective, such as the avoidance (if possible) of some specified region in state space or the minimization of the excursion into such a region. Furthermore, it is possible that even this secondary criterion might not provide a complete ordering, so that additional criteria could be applied. This problem (which we designate as a "hierarchical" optimal control problem) can, therefore, be viewed as a sequence of optimization problems each with a more restricted set of admissible control functions, the new admissible set being the set of control functions which are optimal with respect to the previously applied criterion. Whenever this admissible set has only one member, the problem terminates. If,at a particular stage,the admissible set is empty,

48 an inappropriate or overly restrictive criterion must have been applied previously. This process has certain similarities with the vector-valued cost function approach of Zadeh (Ref. 46), but is not identical in that the various criteria are applied sequentially here, rather than simultaneously. The fact is, however, that in a significant fraction of the optimal control problems that have been studied, the original cost functional is such that a single optimal control is determined, thus leaving no opportunity for the application of a hierarchical approach. The modification of this approach outlined below might still be useful in engineering problems, nonetheless: In Section 1. 2. 5. 3 we mentioned the difficulties involved in the choosing of a single cost functional which accurately reflects, in the proper proportion, all of the factors that are important in a given problem. The functional actually chosen is often a compromise of some sort. From an engineering standpoint, it thus seems unreasonable that the control which emerges as having the minimum "cost" (in this compromise sense) be regarded as preferable to every other control, including those with "costs" almost as small as the optimal cost. That is, since the cost functional is a guess or a compromise anyway, we are putting too much faith in our mathematics if we insist on using only the control which is optimal this sense. A more reasonable approach would be to look at all the controls which have costs close to the optimal cost (within some reasonable engineering tolerance, say 10%, or whatever figure is appropriate to the problem at hand), and choose among these on the basis of some other appropriate criterion, such as minimum energy, ease of generation by existing devices, minimum frequency bandwidth, etc. This idea forms the basis for a revised hierarchical optimization technique: a) Find a control which is optimal with respect to the primary cost functional, (chosen, as best one can, to embody the most important elements of the cost) and determine the appropriate cost. b) Take as the admissible set of controls all those with costs within some tolerance of the minimum cost. c) Apply the secondary cost functional to determine an optimal control from this set, and the appropriate cost. d) Take as the admissible set of controls all those that "survived" step b) and are within some tolerance [not necessarily the same as in part b)] of the minimum cost determined in part c).

49 e) Apply a third cost functional, and proceed as in the previous steps until the process terminates in the selection of a single control. The resulting control will be within the chosen tolerence of the minimum cost (with respect to the original cost functional) and yet might be considerably better (in the secondary, tertiary, etc., senses) than the control which is truly optimal in the first sense. 3. 6 Proper Systems The conditions imposed thus far on the types of systems considered have been sufficient to guarantee the existence of an optimal control, and have allowed the establishment of Theorem 3. 2, which gives information about the form of the optimal control. But, as we have seen from the examples of the last section, they are not sufficient to guarantee that the optimal control be unique. While it is usually not of prime engineering importance that optimal controls be unique, it is sometimes worthwhile to know whether or not there is a unique optimal control for a given problem. For example, if a certain control u(t) is known to be optimal, but this control is undesirable for some reason (e. g., difficult to generate, requires wideband equipment, etc. ), then we might wish to seek a "better" optimal control if we knew that -u(t) was not the only optimal control. If, on the other hand, u(t) were known to be the unique optimal control, such efforts would be futile. Another difficulty with the class of problems considered thus far which is of greater practical importance than the lack of uniqueness of the optimal controls is the fact that when the optimal control is not uniquely defined by Theorem 3. 2, we have no simple and convenient method of determining what numerical values of the control to specify. In contrast, when Theorem 3. 2 completely specifies the optimal control almost everywhere on T, the problem of finding an optimal control reduces to the relatively straightforward one of - k finding some vector c in E such that the control determined by Eq. 3. 12 satisfies Eq. 3. 7. In order to insure that Theorem 3. 2 completely specifies an optimal control, we must make additional restrictions on the class of systems considered. (Such restrictions will also guarantee that the optimal control is unique, a result which is incidental to our purposes here.)

50 If we restrict attention to systems which satisfy all the conditions of Section 2. 1 plus the additional condition that, for all c e E, _c 0, the quantity V (t, t) c is different from zero almost everywhere on T, then the difference between the set T1(c) defined in Theorem 3. 2 and the set T will be a set of measure zero. Theorem 3. 2 can then be applied to determine an optimal control defined almost everywhere on T for every choice of c. Furthermore, this will be a useful result, since many systems of engineering interest meet the above stated requirement. This line of reasoning motivates the following definition and theorem: DEFINITION: A system of the type described in Section 2. 1 is proper on the interval T if, for every vector c e Ek, C i 0, the quantity V (tl,t)c is different from zero almost everywhere on the interval T. We note in passing that this definition is analogous to LaSalle's definition (Ref. 59) of a proper system, which can be stated as follows: A system characterized by an equation of the form of Eq. 2. 1, and with A(t) and B(t) continuous matrix functions of time defined for all t, is proper if, for every c e En c 0, the quantity BT(t) X(t,t) c t is not identically zero on any time interval of length greater than zero. These two definitions are not completely comparable, since the second definition applies to a more restricted class of systems than the first, but for problems to which they both apply, the second is more restrictive than the first in two respects: a) The second definition involves every possible time interval on the whole time axis, while the first makes restrictions on the system only on the interval T. b) For k < n, the matrix D, which enters into the definition of V(t1, t), might be such that a given system would not satisfy the second definition, but would satisfy the first. To show this, we have only to write out the two quantities IB(t) X (tl,t)cn and IGT(t) BT(t) XT(t, t) DT I that appear in the two definitions. The n-vectors D c lie in a k-dimensional subspace of En, because D is of rank k, by assumption. Ifallthevectors in whichcaused

51 IBT(t) XT(tlt)c to be identically zero on some subinterval of T happened to be orthogonal to this manifold, then the stated result would take place. On the other hand, the second definition cannot be satisfied if the first is not, since every k in the null space of G (t) B (t) X (tl,t)D corresponds to - -k T T n Tsome c in the null space of B (t) X (tt), namely c = D ck Theorem 3. 3: Every system of the type described in Section 2. 1 which is proper on the interval T has an optimal control which is defined almost everywhere on T by Eq. 3. 12 of Theorem 3. 2. Furthermore, this control is unique to within a set of measure zero. Proof: We first dispose of the cases in which C = 0. In such cases, Eq. 3. 13 shows that the optimal control is zero almost everywhere on T, which obviously satisfies this theorem. Therefore, we need to consider only cases for which C > 0. For such cases, we first note that the first assertion of the theorem is obviously true, since the condition that the system be proper on the interval T is precisely the condition that Eq. 3. 12 determine z(t) almost everywhere on T. To prove the uniqueness of this control, we consider any two outward normals c1 and c2 to the set S at the point s and the corresponding optimal controls z1(t) and z2(t). (By Theorem 3. 2, every optimal control has some such normal associated with it so that by considering all such normals we are indeed considering all the optimal controls. ) Both z1(t) andz2(t) must satisfy Eq. 3. 7, and therefore, g = f V(tlt) zl(t)dt T g = j V(tt) z2(t)dt T Subtracting these gives o = f V(tl,t) [zl(t)- z2(t)]dt T Forming the inner product of each side of this equation with c then yields

52 0 = f CT V(tl,t) [z(t) - z2(t)]dt T But since z (t) C VT(tl t)- l = V t1,t)c1 almost everywhere on T, from Eq. 3. 12, the quantity clTV(tl,t) can be written as VT (tl, t C z T(t) almost everywhere on T. Thus, 0 = c -t Z(t) z (t) - Zl t) z2(ti dt T C We note from Eq. 3. 12 that | zl(t)l = C and| z2(t)I = C almost everywhere on T. Using this T result and the fact that the inner product z Z2 of any two vectors zl and z2 is no greater atn z2 -1 - -2 than the product of the Euclidean norms of the two vectors, we have that T) (t) zl (t) - (t) ) > 2 - = 0 almost everywhere on T. Thus, the integrand of the above integral is never negative (except perhaps on a set of measure zero), and yet the integral is equal to zero. Therefore, the integrand must be zero almost everywhere. Since r VT(tl,t)cll is different from zero almost everywhere on T, it must follow that zlT(t) zl(t) - zT(t) z2(t) = 0 a. e. on T. Exactly similar reasoning with the roles of zl(t) and z2(t) interchanged shows that T T z2 (t) z(t) - z (t) z(t) = 0 a. e. on T. Adding these two expressions then gives T T T T z (t) z(t) - (t) (t) (t) z(t-(t) = 0 a. e. on T. But this is just the square of the Euclidean norm of the vector zl(t) - z2(t). Therefore, I zl(t) - z2(t)I = 0 a.e. on T,

53 which implies that zl(t) = z2(t) a. e. on T. Since zl(t) and z2(t) were any two optimal controls, the theorem is proven. Lemma: If a given system of the type described in Section 2. 1 is proper on the interval T, then the requirement of Section 2. 1 that the output be completely controllable on the interval T is redundant. Proof: Consider a system which meets the description of Section 2. 1 with the complete controllability requirement omitted, and suppose that IVT(tl,t)cI c 0 a. e. on T for all E, c / O. Then cT V(t1,t) VT(t1,t)c dt > 0 for all c Ek, _c 0. T This is a statement of the fact that the matrix J V(t, t) VT(t,t)dt T is positive definite, which is shown in Appendix D to be a necessary and sufficient condition that the given system output be completely controllable on the interval T. Q. E. D. In testing a given system to see if it meets the conditions of Theorems 3. 1 and 3. 2, we may thus omit the test for complete controllability on the interval T if the system satisfies the condition used to define systems which are proper on the interval T. Before proceeding with some examples of proper systems, we note certain implications involved in this approach: Had the original Banach space Z been reflexive and rotund, the existence and uniqueness of the optimal control would have followed at once (see Porter and Williams, Ref. 83, p. 42). In this reference it is shown that it is the rotundity of the space that results in a unique optimal control. In the problem considered here, the control space is not rotund, and the optimal control is not unique in every case. By restricting attention to proper systems, we obtain the same result as if the space had been rotund, namely, a unique optimal control. This is more than just a chance similarity, as we see from the following considerations: In a rotund space, the unit hypersphere has no "flats, " or, in other words, is strictly convex. The image of such a hypersphere under a continuous linear mapping of the type studied

54 in Section 3. 2 is also strictly convex, as is easily shown using the linearity of the mapping and the fact that points on the boundary of the image set can be the images only of points on the boundary of the original hypersphere (as proved in Section 3.2). It is the strict convexity of the image set (the reachable set) which results in a unique optimal control. In the problem considered here, the unit hypersphere in Z has "flats. " (As a simple example of this, consider the two controls z1(t) = 1, 0 < t < 2 and z2(t) = 1, 0 <t < 1; z2(t) = 0, 1 < t < 2. Both have peak amplitude equal to 1, and hence,are on the surface of the unit hypersphere in Z. Any linear combination of these of the form 0zl(t) + (1-0) z2(t), 0 < 0 < 1, also has peak amplitude equal to 1. Hence,Z is not strictly convex. Examples of this type for vector-valued controls z(t) are easily concocted along the same lines. ) Examples 1 and 2, Section 3. 5, show that without the restriction that the system be proper on the interval T, the set S1 may also have "flats. " With this restriction, however, S1 is always strictly convex. This can be shown as follows: Theorem 3. 2 states that every optimal control has associated with it at least one normal vector. Theorem 3. 3 states that for proper systems each such normal uniquely and completely determines an optimal control. Since each input to a linear system of the type considered in this work gives rise to a unique output (Ref. 57), this means that each normal vector can correspond to one and only one k point on the surface of S1. If no two points on the surface of the convex set S1 in E have the same normal, then S1 must be strictly convex. Thus, be restricting attention to proper systems, we guarantee that the image S1 of the unit hypersphere in Z is strictly convex. A further restriction could be made also, to obtain properties analogous to those of spaces having "smooth" unit hyperspheres; that is, hyperspheres with no "corners" (see Porter and Williams, Ref. 83), which guarantees that the reachable set has a unique normal at every point of its boundary. It can be shown that the unit hypersphere for the control space Z used here also has "corners. " But a suitably restricted class of systems, which we shall refer to as "smooth" systems, are such that the reachable set has no corners. Figure 3. 2 shows examples of the appearance of the reachable set for systems which are proper and/or smooth, or neither. These points are also discussed by Kriendler (Ref. 61). Smooth systems are not investigated in detail here, however, because the distinction between smooth and nonsmooth systems is not of great engineering significance.

55 Fig. 3. 2(a) Typical sets S1 for systems which are neither proper (rotund) nor smooth. Fig. 3. 2(b) Typical sets S1 for systems which are proper (rotund) but not smooth. Fig. 3. 2(c) Typical sets S1 for systems which are smooth but not proper. Fig. 3. 2(d) Typical sets S for systems which are both smooth and proper (rotund). 3. 7 Two Examples Involving Proper Systems Example 3: From the set U(O,b, T) of bounded measurable scalar functions u(t) defined on the interval T = [0,1] which cause the system (a double integrator) characterized by - X + U X= 0 0 x+[1] to transfer from the initial state x(O) = 0 to the desired final state x(1) = b, choose as an optimal control any one for which C(u) = ess. sup. u(t) te T is minimum with respect to all u(t) e U(O,b, T). For this system the matrix X(tl,t) is 1Xt) X(t et) = Lo t -t 1

56 so that V(tl,t) = V(l,t) = X(l,t) B(t) = [lt. This system is proper on the interval T, since vT(t1,t)c = (1-t)cl + c2, which, for nonzero c, can be zero on the interval T only at the isolated point C1 1 c- c2 For this problem the set S1 defined in the proof of Theorem 3. 1 is as shown in Fig. 3. 3, from which it can be seen that in this case S1 is rotund but not smooth. That S1 is as shown can be verified by substituting vectors c corresponding to all possible normal directions in Eq. 3. 12, using C = 1 (this gives all points on the boundary of S1), and making use of the fact that all interior points of S1 can be obtained for some appropriate choice of C < 1. The optimal control for a particular goal b can be obtained from this plot of the set S1 as follows: Draw a line from the point b to the origin, extending it outward if necessary until it intersects the boundary of S1. Determine the direction of the normal to S1 at the point of intersection, and use this result in Eq. 3. 12 to determine the optimal control. For more accuracy, an iterative computational algorithm of the type discussed in Chapter VII could be used. x2(1),- 1 xx(S) -1 Fig. 3. 3 The set S1 for Example 3.

57 We note at this point that Example 3, in addition to being a minimum peak amplitude problem in the sense used in this work, is also a member of the class of problems considered by Neustadt (Ref. 77) and others, for which the cost functional was defined as, max sup ui(t) i=1,...,r tt e T i where ui(t) is the ith component of the vector u(t). (For scalar u(t), this cost functional is identical to the one used in this work. ) The next example, on the other hand, is one in which these two cost functionals are not the same, since it involves a vector-valued control. Example 4: From the set U(O,b, T) of bounded measurable controls u(t) defined on the interval T = [0, 1] which cause the system characterized by 0 1 0 0 0 x= 0 0 x0 + 1 0 u O 0 1 0 1 1 0 0 (t) = x(t) o 0 1 to transfer from the state x(O) = 0 to the desired final condition y(1) = b, choose as an optimal control any one for which C(u) = ess. sTup. Iu(t)| is a minimum with respect to all - ET -T u(t) e U(O,b,T). For this problem 1 tl-t 0 X(tl,t) = 0 1 0 0 0 1 so that = V(1-t 0 V(t,t) = V(l,t) = DX(l,t)B(t)= 0 1 L

58 36 This system is easily shown to be proper36 on the interval T, so that Theorems 3. 1, 3. 2, and 3. 3 apply. The resulting set S1, which is both rotund and smooth, is as shown in Fig. 3. 4. The points on the boundary of this set are obtained by evaluating Eq. 3. 12 for vectors y2(1) 1.0 S1.y1(1) -1.0 Fig. 3. 4 The set S1 for Example 4. c c E2 in every possible direction. If we denote by 0 the angle that the vector c makes with the x1 axis, then the boundary of the set S1 is given by 36It is, in fact, proper in LaSalle's sense also (Ref. 59).

59 1+ sec 1 sec 0 sgn [cos 0] tan2 o log < 0 < 2'tan e 1+ sec 0 J sgn [sin 0] tan 0 log tan 0 ~ 0, i 2 y(1) = i 1., t~0=0;, = +; 0 = 7T as can be verified by substituting the z(t) obtained from Eq. 3. 12 into Eq. 3. 7 and carrying out the integration. The optimal control for any given goal b can be obtained as described in Example 3. Comparison of Figs. 3. 3 and 3. 4 shows that, while it may happen that a system which is proper on the interval T is "smooth, " i. e., has a set S1with uniquely defined normal direction at each point of its boundary (see Fig. 3. 4), this is not necessarily true of such systems (see Fig. 3. 3). 3.8 Problems With Unspecified Final Time In this section we consider minimum peak amplitude problems in which the final time t1 is not specified in advance. That is, we are to choose not only the control u(t) but the time interval over which it acts, so as to minimize the peak amplitude of the control. 37 Since the control can be determined from previous results once the optimal value of t is known, the only new aspect of this problem is that finding the optimal t1. Two versions of this problem are considered here. In the first version, no restrictions are placed on t1 except that it be greater than the initial time t0. In the second version, t1 is restricted to lie in a closed and bounded8 interval, i. e., to < t' < t1 < t" < o, 37The optimal value of t1 is the one which allows the peak amplitude C to take on the lowest possible value, compared to the values of C corresponding to any other choice of t1. 8The interval is required to be closed and bounded as one condition of an existence proof, as well as to give a reasonable model of those engineering problems in which unlimited time intervals are not allowed. The existence of solutions is discussed in detail below.

60 where t' and t" are given. In both versions, problems involving "moving targets" are included by allowing the desired final condition vector b to be a function of t1. For further generality, the k x n matrix D appearing in the definition of the output vector y (see Eq. 2. 2) is also allowed to be a function of t. So that the results of the previous sections can be applied to this problem, we require, in analogy to Section 2. 1, that for each fixed value of tl in the allowed range, the k x n matrix D be of rank k and the output Y of the system be com1 pletely controllable on the interval [to, tl]. Under these restrictions, it follows from Theorem 3. 1 that there exists at least one minimum peak amplitude control (with unique minimum peak amplitude C) for each allowed choice of final time t1. Thus, the minimum peak amplitude C can be regarded as a function of t1. Upon rewriting Eq. 3. 13 to show the tl-dependence explicity, and making use of the fact that the magnitude of c does not affect C(t1), so long as c # 0, we have that max (c, g(t)) C(t1) = c =a (3.18) to J V (tlt)cl dt or, from Eq. 3.15, ( (t1), g(t1)) C(t ) = (3. 19) 1 f V (tt, t)c(t 1) dt to Here c is an outward normal to the boundary of the reachable set at time tl, as defined in Theorem 3. 2. Since C(tl) is a function of t1 (not a functional), the problem of determining the minimum value of C(t1) over all allowed values of t1 is a problem in the ordinary calculus (not the calculus of variations). As in all such problems, it may happen that C(t1) does not take on a minimum value on the open interval (to0, o). The following simple examples illustrate this point:

61 Example 5: Consider the problem of determining the minimum peak amplitude control u(t) which forces the system characterized by x = -x + u from the initial state x(0) = a = 0 to the final state x(tl) = b(tl) = 1, where 0 < t1 < oo. One -t+s can easily show (Ref. Section 3. 1) that X(t, s) = e, and hence that u(t) must satisfy g(t) = -t! t1 -tl+s b(t) - e a = S e tl u(s)ds. From Theorem 3. 2 0 -t +t 1 u(t) = C(t1) - = C(t1) e c with the plus sign being used if g(t1) is positive and the minus sign if g(t1) is negative. Thus g(t ) += C(t1) f e tl ds += C(t ) 1-e ) 0 C (t) 4 2 asymptote 0 1 2 Fig. 3. 5 C(t1) vs tl for Example 5.

62 which yields |g(t1)j I b(t1)- a C(t1) t= _ t = T = -t 1-e 1-e 1-e This function is plotted in Fig. 3. 5, from which we see that there is no value of t1 e (0, co) which causes C(tl) to be minimum. We note, however, that if t1 is restricted to lie in some closed and bounded interval [t',t], then the problem does have a solution, namely t = t, u(t) = C(t"), etc. Example 6: In this examplewe keep the same simple system as in Example 5 but change the boundary condition as follows: t1 x(0) = a = 1 x(t) = b(t1) = e (Since the final condition varies with t1, this is a "moving target" problem. ) Using the results of the previous example, g(tl) _ b(t) - e a e't e t C(t -t -t -t 1.1 1. 1-e1 11-e 1-e This function is plotted in Fig. 3. 6, from which we see that there is no value of tl E (0, co) which causes C(tl) to be minimum. As before, however, if t1 is restricted to lie in some closed interval [t',t"], where t' > 0, then the problem has a solution, namely t1 = t', u(t) = C(t'), etc. If the interval to which t1 is restricted is [0, t"], then another interesting result is observed: The minimum cost is zero, and the corresponding final time is t1 = 0, since no control effort is required to force the system from x(0) = 1 to x(t1) = x(0) = e = 1. Thus, the curve of C versus tI is discontinuous at t = 0 in this case. These two problems involve functions C(tl) which are not only continuous in t1 except at t1 = 0, but are also continuously differentiable except at t1 = 0, and yet they have no solution in the case where t1 is allowed to lie anywhere in (0, co). On the other hand,

63 it can happen that problems having a very "badly-behaved" C(t1) function may nonetheless possess a solution, even on the open interval (0,oo). The following example is case in point: C((t1) 1I. - / t 0.2.4.6.8 1.0 Fig. 3. 6 C(t1) vs t1 for Example 6. Example 7: Let the system be the same as in Examples 5 and 6, and let the boundary conditions be x(O) = a = 0 x(t1) = b(t1) = e g1 [sn (cos.5 2 tl~s ( O.12 ~ 2 where the signum function sgn (y) is here defined as follows: sgn(y) = 0 -1 y> y =0 y<0 Proceeding as in the previous examples we get that C(t) = e [sn (cos 5) + 2] tsgn (c o -15 ]

64 Investigation of this function for all t1 > 0 shows that it takes on its absolute minimum value C = 4 at t1 = loge2. Figure 3. 7 shows C(tl) plotted as a function of t1. A slight modification of this problem, namelyone in which the final condition is x(tl) = b(tl) = e 1 sn (cos tl2) + 2 (t1) 20 \ 15 I / oscillates between upper / and lower curve faster and faster as t1- 0. 5. 1att loge2 Ii 11 I I 1| 1' 0.25 5I.75 III III! I I II iI i i I j!ll **111 4I' I, 111 I II II I Il(. II I 11 111I' iI m inimu at t= log 2 1 e t 0.25 o 5.75 1.0 Fig. 3. 7 C(t1) vs t1 for Example 7.

65 would result in a problem which had no solution on any interval, open or closed, which included tl = loge2 as one of its points. These three examples, taken together, show that the question of the existence of nonexistence of solutions to unspecified final time minimum peak amplitude problems is not a simple one, and that the rules obtained for fixed final time problems do no apply. However, we can state the following lemma: Lemma: For problems satisfying the conditions of Section 2. 1 at each tl in the allowed interval [t',t"], t < t' < t" < oo and in which C(tl) is continuous in t1 on this interval, the unspecified final time problem has at least one solution. Proof: This lemma is merely a statement of the fact that a continuous real function defined on a closed interval takes on its infimum. One way to prove this is to note that a closed bounded interval on the real line is a compact metric space (with the distance between two points ta and tb being taken as Ita - tb ). Then, from Theorems 4. 15 and 4. 16 of Rudin (Ref. 82, p. 77), the image of this space under the continuous mapping assumed to exist between t1 and C(tl) is also a compact metric space, and therefore C(tI) takes on its infimum. Q.E. D. The following lemma gives a sufficient condition that C(tl) be continuous in tl: Lemma: C(tl) is continuous in t1 on the interval [t',t"] if b(tl) and D(t1), the vector and matrix which define the final manifold, are continuous in t1 on the same interval. Proof: We note that if b(t1) and D(tl) are continuous in tl, then g(t1) =b(tl) - D(tl) X(tl,t) a -1 and V(t1,t) = D(t1) X(t1,t) B(t) G (t) are also, since X(t1,t) is absolutely continuous (and hence,continuous) in t1 (see Coddington and Levinson, Ref. 57, pp. 42-43). The quantity (C,,(tl)) t f v T(t,,t)scIdt to

66 appearing in Eq. 3. 18 [which we denote by F(c, t)] is continuous in t1 on the interval in question, since it is the ratio of continuous functions and the denominator is bounded away from zero (uniformly with respect to vectors c satisfying the contraint tc = 1) for t1 > t' > to due to the complete controllability assumption of Section 2. 1. Note also that F(c, t1) is independent of the magnitude of c, for all c O 0. Thus, for every tl in [t',t"], every c O0, and every e > 0 there exists a 6(t) > 0 independent of c such that F(c, t + At1) - F(c, t) < e for Atl < 6(t1). From this it follows that for j At l < 6(tl), max max ma1 F(c, t+At) - m F(2 t < e ~ * I - 1" = 121 1I But since |C(tl+At1)-C(tl) = lIcI =1, Ic) 2 1 F(2 1 from Eq. 3. 18 and from the definition of F(c, t1), it is clear that C(t1) is continuous in t1. Q.E. D. As a final point in this section, we investigate the possibility of determining the minimum of C(t1) by differentiating C(t1) with respect to t1 and setting this derivative equal to zero. We assume that C(tl) is piecewise continuous and has a piecewise-continuous derivative with respect to t1. (In order to be able to carry out the required operations, we must also assume that c(t1), the normal to the reachable set which appears in Eq. 3. 19, satisfies the same continuity and differentiability conditions. ) In order to locate the minimum of C(t1), if it exists, we must then investigate the stationary points and points of discontinuity of dC(tl) dt, and, in problems involving t1 restricted to a closed interval, the end points of this interval. The derivative of C(tl) with respect to t1 can be computed from Eq. 3. 19. In this derivation we make use of the facts that V(tl,t) = D(t1) X(t1,t) B(t) G (t), g(t1) =b(tl) - D(t) X(tl,t0)a, and -- X(tlt) A(t) X(tl,t). To simplify the notation, explicit mention of the functional dependencies of C(t1), c(t1), g(t1), V(t1,t), D(t1), X(t1,t), A(t1), and B(t) will be omitted in this derivation unless needed to avoid confusion:

67 dC dt ridg dc dtl L dtV I'l vT dt to (,' I t VT c dt] IVT(tl tl)- I -T dc T- -T dV Tt 12 d — VV c+2c -- V c t dt _ - - dt 02 ivT' dt d7 dt. I tl to V TI dt- - C vT t,,t, c dC 1 dt1 t1 T- j V cd -T dc- -T dD -TAX + - ~-c X(tt )a - c DAX(tl, t)a T-. t dt Tt0 I-1 Vc - -T vT I - -Td f XBG - dt-C DAf XBG - dt dT We note that the term multiplying dt- is identically zero, due to the satisfaction of the final condition. Further collecting of terms, plus substitution of Eq. 3. 1 (evaluated at t = t1) and use of the u(t) given by Theorem 3. 2 in two places, gives _|-(d D (t)- db dC 1 0 - C |VTCt1 c | - cdt x(t dt1 dC 1 (3.20) dt1 tl T1 =f V c dt -_ to - c DAx(tl) The stationary points of C(t1) occur when this quantity is equal to zero. Thus, C(tl) will have a stationary point at a particular value of t1 if the following condition is satisfied: dA+ || (c dtl t (3. 2d) -T - IdT ( D(t1) t(. c D(t A(tl) x(tl) + C(t)I V (ttl)z + _c, dtI x(t) - dtI: 0 (3.21) 1~~~~t,

68 Unfortunately, this condition is of limited usefulness, since there is no way to tell in advance whether or where C(tl) will have a discontinuous derivative. As noted above, the minimum value of C(tl) may occur at just such a point. The following example, involving a very simple system description and constant boundary conditions, nonetheless gives rise to a C(t1) with a discontinuous derivative. The same thing can happen in higherorder systems also. Example 8: Let the system be the same as in Examples 5, 6, and 7, and let the boundary conditions be x(O) = a = 1, x(tl) = b(tl) = 1 Then, following the same procedure as before, ~1~~T 2 we have that g(tl) = - e and hence that -tl 1- e asymptote 0 1 2 t Fig. 3. 8 C(t1) vs t1 for Example 8, showing discontinuous derivative at tl = loge2.

69 This function is plotted in Fig. 3. 8, from which we see that the optimal solution is t1 = loge2, -t C(t1) = 0, u(t) = 0, x(t) = e. However, it is clear that C(t1) does not possess a derivative at t1 = loge2, and that this is thus obviously not a stationary point of C(tl). One conclusion that can be drawn from the discussion and examples of this section is as follows: Unspecified final time minimum peak amplitude problems may exhibit behavior which is difficult to analyze or predict, even in the simplest cases. Therefore, a good engineering approach to such problems uses a combination of graphical and analytic techniques —graphical techniques to determine the range or ranges of t1 in which minima should be sought, followed by analytical techniques (where applicable) to locate the minima exactly. This is in fact the procedure that was used in Example 7 to find the minimum at t = loge2. 3. 9 Optimal Control Laws As defined earlier, an optimal control law is a rule which states what control function should be applied to the controlled system as a function of the state of the system, so that the resulting trajectory will be optimal. This law sometimes takes a very simple form, as in the by-now classical "bang-bang" second-order servo problem (Ref. 13), in which the phase plane is divided into two regions, one in which the optimal control always takes on the maximum allowed value, and the other in which it takes on the minimum allowed value. More complicated examples have also been worked out, involving more than two regions, but still preserving the simple rule that every state in that region is associated with the same value of the control (see Ref. 13, pp. 23-45). In the problems considered here, the control law generally takes a more complicated form than this. Theorem 3. 2 states that at those moments at which a minimum peak amplitude control is uniquely defined, it is of the form V (t,t)c z(t) = G(t)u(t) = C T(tt (3.22) T (t, t) where C and c are determined from the boundary condition equation

70 t T ~_,l V (tl, t) c b - DX(tl,t0)a = C V(t t) t dt t V0 1 vT(tlt)c We can also write this in terms of the state x(t) at time t, thus: tl VT(ts)c b- DX(tt) x(t) = C f V(ts) vTt ds (3.23) t v(t, s)-_c Since this expression does not involve the initial state a or initial time to, we can in principle construct a device which determines the optimal control at each moment as a function of the state x(t) at that moment, plus of course the desired goal state b and final time t1; such a device must determine the state x(t) at time t, then solve Eq. 3. 23 to determine C and c, and then use these values in Eq. 3. 22 to determine u(t). This is not as formidable a problem as it may appear to be at first, since if the given equations were a perfect model of the physical system and if the initial computation were errorless, C and c would remain the same throughout the whole trajectory. The effect of imperfect modeling of the system, unforseeable external disturbances, and computation errors is to require, in practice, that C and c be corrected during the time interval of the problem, but if these errors and disturbances are typically small, infrequent correction would normally suffice, so that the computation of C and c need not necessarily be carried out in real time. If Eq. 3. 22 does not uniquely specify the optimal control at every moment of time, [i. e., if V (tl,t) c is zero on some nontrivial subset of the interval], complications are introduced, in that some other means must be provided for computing the control at such times, but, in principle, the above discussion still applies. Another way of specifying the optimal control law is possible, and is especially convenient if the system is proper, which implies that the quantities C and c completely determine the optimal control. This other way amounts to a tabulation of the optimal control for each point in state space; that is, we consider the (k+l)-dimensional Euclidean space

71 having k of its coordinate axes associated with the k components of the goal state 9(tl,t) b - DX(t1,t) x(t) and the (k+l)st axis associated with the time remaining, t - t, and tabulate, for each point in the half space for which t1 - t is positive, the numbers C, c appearing in the expression for the optimal control (Eq. 3. 22) corresponding to that goal and amount of time remaining. Such a tabulation is impossible (since an infinite number of points are involved), but two simplifications allow the same end to be accomplished more reasonably: a) Only the values of C and c for unit vectors g(tl,t)/| g(tl, t) need be tabulated, since the control for other values of g(tl,t) can be obtained simply by multiplying by Ig(tl,t) ( the value of C corresponding to a unit vector in the desired direction. b) In the usual engineering problem, some tolerance is allowed, both with respect to the satisfaction of the desired final condition and with respect to the achieving of minimum cost. Thus, adjacent points in the abovementioned (k+1)-dimensional space could be grouped together in regions and treated as equivalent, thus reducing to a finite number the number of values of C and c that must be tabulated. Approaches other than straight tabulation are possible, also, such as a tabulation of level curves or (in two or three dimensional problems), the storing of the information as X-Y plots or 3-D cams. In any case, the problem of tabulating a vector function of time for each.g(tl, t) has been reduced to the storing of k + 1 numbers C, cl,..., ck (actually k numbers, since c is a unit vector) for each g(t1,t). Really, the amount of the goal that remains unaccomplished.

CHAPTER IV THE INVERSE PROBLEM 4.1 A Time-Optimal Problem Associated with each minimum peak amplitude problem there is a certain 40 minimum time problem. 4The following discussion of this minimum time problem offers additional insight into the original problem, and provides an alternate approach to the solution of that problem. It will be shown that under certain conditions the two problems are, in a sense, inverse to each other. We begin by posing the time-optimal problem: In the class of bounded measurable controls u(t) which cause the system x = A(t) x(t) + B(t)u(t) y(t) = D(t) x(t) to satisfy the boundary conditions x(to) = a; y(t1) = b(t) at some unspecified time t1 and for which I G(t) u(t) I < C for all t e T, find one for which tlis a minimum. A, B, D, G,x,y, and u are as specified in Sec. 2.1. C is now a given constant. 4. 2 The Solution to the Time-Optimal Problem We solve this problem with the aid of Pontryagin's maximum principle, which states that 40 Krasovskii (Ref. 78) has considered a special case of this problem, approaching it from the functional analysis viewpoint. 72

73 a necessary condition that the control u(t) and the corresponding x(t) and y(t) be optimal is that there exist a constant vector c, a nonpositive constant 0o, and an absolutely continuous nonzero vector A(t) satisfying aHll(t) Qi - x-t e T1 i t=T 1 i =1,...,n (4.1) x t1 const. H(tl) + aQ =0 (4. 2) 1 (t1) = const. such that at every moment of time t E T, u(t) maximizes H(t), (for fixed x and _) over all controls for which IG(t) u(t)l < C, where H(t) = o0 + ((t), A(t)x(t) + B(t) u(t)) (4. 3) Q = (c, y(t1) - b(t1)) = (c, D(tl)x(t) - b(t1)) (4. 4) Since only the last term of Eq. 4. 3 depends on u(t), the required maximization of H(t) is carried out by maximizing the quantity'T (t) B(t) u(t) subject to the constraint IG(t) u(t) < C. Rewriting this quantity as _T(t) B(t) Gl(t) G(t) u(t) reveals that T T whenever GT (t) B (t) _ (t) 0, u(t) must be chosen so that G(t) u(t) is colinear with T T G (t) B (t) _.(t) and of the maximum allowed length. Thus G(t) ut) C GTt) BT(t) (t) (4. 5) IG-T(t) BT(t) 4(t) -T T whenever G (t) B (t) _/(t) is not equal to zero. At those times at which this quantity is equal to zero, the maximum principle cannot be used to determine u(t). * pT Carrying out the indicated operations in Eq. 4.1 gives i = -A (t)i, (t DT(t) which has as its solution 4,(t1) = D (t1) c, which has as its solution _(t) = xT(t t) DT(t1) c (4. 6)

74 as can be verified by direct substitution and the use of the identity X(t1, t) X(t, t1) = I. Here X(tl, t) is the state transition matrix defined in Section 3.1 in connection with the solution of the original problem. Substitution of this result into Eq. 4. 5 gives VT(t t) c G(t) u(t) = C VT(t I whenever IVT(tl,t) c I / 0 IvT t t)c (4. 7) G(t) u(t) I < C but otherwise undetermined whenever I V (t, t) c = 0 where V(tl, t) is as defined by Eq. 3. 4. We note that this result is identical in form to the optimal control for the original problem, as given by Theorem 3. 4, except that the unknown C has been replaced by the given value C, and t1is now considered as a variable, rather than a given constant. Since the boundary conditions for the two problems are also the same for each ti, we might expect the two problems to be closely related. These considerations motivate the following definition: 4. 3 The Inverse Relationship Definition: A minimum peak amplitude problem and a minimum time problem are said to be in inverse relationship to each other (or, alternatively, to be inverses of each other) if they involve the same system and boundary conditions and if the control which is optimal for the minimum peak amplitude problem (for a certain t1 and resulting peak amplitude C ) is also optimal (with minimum time tl) for the minimum time problem in which C is chosen as the bound on the amplitude of G(t) u(t). That is, in the two problems the roles of cost functional and constraint on u are interchanged, the cost functional of one being the constraint on the other, and vice versa. We ignore controls which differ from the optimal control only on a set of measure zero. Three questions now arise: a) Are there any pairs of problems which are inverse to each other? The answer is yes. Every time-optimal problem involving a proper system and a

75 magnitude constraint on the Euclidean norm of G(t) u(t) which has a solution41 is 42 inverse to a minimum peak amplitude problem. This follows4 from the identity of form of the solutions to the two kinds of problems, and from the uniqueness of the solution to the fixed final time minimum peak amplitude problem for systems which are proper on the interval T. b) Are all minimum peak amplitude problems inverse to some minimum time problem; i. e., are the two classes of problems completely equivalent? The answer here is no. To illustrate this, suppose that we have solved a minimum time problem and obtained three distinct solutions (with corresponding final times tla, tlb, and t1c, tla < tlb < tic) all of which satisfy all of the given necessary conditions for time optimality. Obviously, only tla is truly a time optimal solution, but, by reasoning identical to that above, all three correspond to minimum peak amplitude problems. Thus, the minimum peak amplitude problems corresponding to tlb and tic do not possess minimum time inverses. c) Since all minimum peak amplitude problems do not have minimum time inverses, can we find conditions which tell us whether or not a given minimum peak amplitude problem (for a specific choice of tl) is in inverse relationship to some minimum time problem? In answer to this, we shall find a general necessary condition for the existence of the inverse relationship. However, a general set of necessary and sufficient conditions which apply for all boundary conditions is not likely to be obtained. 4Many such problems have no solution, since C may be too small. For instance, Example 5 of Sec. 3.8 shows a system for which there are no minimum-time solutions if C is chosen less than unity. See Fig. 3. 4. 42 4To illustrate this reasoning in more detail, assume that we have solved the minimum time problem with specified C, and obtained a minimum time t1. If we now consider a minimum peak amplitude problem with this same tl chosen as the final time, we know that there is a unique optimal solution. Since the control for the minimum time problem satisfies all the conditions of this problem, it must be the unique solution to the minimum peak amplitude problem. Thus, every minimum time solution is also the optimal solution to a minimum peak amplitude problem, as claimed.

76 To facilitate discussion of these points, we introduce for each minimum peak amplitude problem to be considered a cost-vs-final time graph of the type shown in Fig. 4. la). We imagine that we have solved the minimum peak amplitude problem for the given system and boundary conditions and for a series of choices of final time t1. Along the abscissa of the graph we plot the final time tl, and along the ordinate the corresponding values of cost C. Such a graph is used in minimum peak amplitude problems as follows: For any given t1 we can read off at once the corresponding optimal cost C. If t1 is unspecified, we can select from the graph the point (or points) for which C takes on its absolute minimum value, or, alternatively, we can see that C has no absolute minimum value for finite t1. Referring to Fig. 4. la), we note in passing that for this example the times t1c and tle correspond to local minima and maxima of C, respectively, and hence that both satisfy the zero-slope condition used for the determination of t1 (see, Eq. 3. 21), even though neither corresponds to an absolute minimum of C. This graph also gives information about minimum time problems, for which it is interpreted as follows: For any given bound C on I G(t) u(t) I, the corresponding minimum time (if one exists) is determined by entering the graph at that point C on the ordinate and determining the minimum abscissa which gives that value of C. For example, referring to Fig. 4. la), a problem for which I G(t) u(t) I must be less than or equal to the specified value Cb, has a minimum time solution with final time tlb. Solutions which require final times tld or tlf are not time-optimal solutions. This graph also clearly illustrates the above statement that all minimum peak amplitude problems do not have minimum time inverses. Minimum peak amplitude problems with times between t1 and tg, in this example, do not possess minimum time inverses. Fig. 4. lb) provides a further clarification of these points. Every point on this curve corresponds to a solution of the inverse (time-optimal) problem satisfying all the conditions of Pontryagin's maximum principle. The gap in the curve between

77 cost C Cb Ch 0 tla tlb tlc tld tle tlf tg lh Fig. 4. l(a). A typical curve of cost vs. final time for a minimum peak amplitude problem. cost C C \ a Cb Cc Oh i rA _. 4_ _ I __ _ _ _ _ _ _ _ _,_ _ 1._ u tla tlb tlc le tf lg lh Fig. 4. l(b). C vs. t1 for the inverse problem, showing only those points which satisfy all the conditions of the maximum principle.

78 tlc and tie implies that there are no trajectories involving these final times which satisfy the conditions of the maximum principle. (It turns out that the trajectories in Fig. 4. la) corresponding to these times satisfy all the conditions of the maximum principle except the one that /o be non-positive. ) It is obvious that the trajectories corresponding to final times between tie and tlg are not time optimal, but this is not a contradiction of the maximum principle, since it gives only necessary conditions. 4. 4 A Necessary Condition for the Existence of the Inverse Relationship From the above standpoint, one immediately sees that a necessary condition that a given minimum peak amplitude problem with specified t1 have a minimum-time inverse is that the slope of the curve of C - vs - t be non-positive at that value of t, assuming that the slope is defined there. That this is not sufficient is attested to by the problems with tie < t1 < tig in Fig. 4. la). In Section 3.8 the slope of C was evaluated as dt [f C(t, s)V c ds [ - [CI V (tltl)I c - c D(t )A(t)x(t1 ) T dD(t1) db(t1) - c (tl) + c d(4.8) dt dt The relationship of this slope to the necessary condition for time optimality given by Eq. 4. 2 will now be derived: Upon substituting Eqs. 4. 3, 4. 4, 4. 6, and 4. 7 into Eq. 4. 2 and making use of the fact that X(t, tl) = I, we obtain T T~ T, -1 V (t1,tl)c T dD(tl) db(t) 4 + cTD(t )A(tl)x(tl) + Cc D(tl)B(tl)G l(t1) T + c (tl tl)c- =0 Iv ( 1_tt)1 which can be rearranged and simplified to give,~T T T dD(tl) T db(tl ) o = -C IT(t, t) I - c D(tl)A(tl)x(tl) - c dt1 (tl) + c(4. dC Thus, 4o is directly proportional to d —, with positive proportionality constant. This dC fact leads to the conclusion that a minimum peak amplitude problem for which d- is positive corresponds to a trajectory which satisfies all the time-optimal necessary positive corresponds to a trajectory which satisfies all the time-optimal necessary

79 conditions given in Section 4. 2 for the inverse problem, except the condition that 4o be non-positive. The failure of this condition implies that the solution is not time-optimal, and hence that the inverse relationship does not exist in this case. dC When dt- is negative or zero, the po for the corresponding time optimal problem is also negative or zero, but the inverse relationship may or may not exist. (We refer again to problems with final times tlb and tlf in Fig. 4. la) to illustrate this point. ) 4. 5 Summary and Conclusions It has been shown in this chapter that an intimate relationship (called here the "inverse relationship") exists between the original minimum peak amplitude problem and a certain class of time optimal problems, and that in many cases the answer to one problem can be obtained by solving the other, and vice versa. This would seem to indicate that we have obtained two distinct methods of solving the original linear minimum peak amplitude problem. The methods are actually the same, however, since both involve the solution of essentially the same set of equations; that is, one must find a control of the form given by Thm. 3. 2 (or Eq. 4. 7) which causes the boundary conditions to be satisfied. For nonlinear systems, on the other hand, there is no well-developed theory for the solution of minimum peak amplitude problems to correspond to that developed in Chapter III for linear systems. In such nonlinear problems, there is therefore some hope that the inverse problem could be useful. This is touched upon in Chapter VI. In the next chapter (Chapter V) a method of solution of the linear problem is discussed which is in fact distinct from any of the methods discussed thus far, and which may have computational advantages in some cases. As will be seen, this method has certain things in common with the direct methods discussed in Section 1. 24.

CHAPTER V THE RELATED LINEAR PROBLEM 5. 1 Motivation It was noted above that the usual variational techniques do not apply to minimum peak amplitude problems because the cost functional C = ess. sup. IG(t)u(t)l (5.1) tE T 43 is not expressible as an integral. However, for G(t) and u(t) as described4 in Sec. 2. 1, it is true that functional44 lGuWlp = [ 1 TIG(t)u(t) IPdt]P (5.2) can be made to approximate the essential supremum of I G(t)u(t) I to any arbitrary degree of accuracy by choosing the constant p sufficiently large [Taylor, Ref. 81, p. 91]. This fact suggests a possible alternative approach to the minimum peak amplitude problem: Suppose we set up a problem (to be referred to as the related problem) with the same system characterization (Eqs. 2. 1 and 2. 2) and the same boundary conditions as in the original problem, but with JIGu |p as the quantity to be minimized, instead of the quantity given by Eq. 5. 1. (This is a problem that can be treated by 43 Since u(t) and G(t) consist of bounded measurable functions (by assumption) the quantities I G(t)u(t) I and I G(t)u(t) IP are also bounded measurably functions, and the integral shown here exists for all p> 1. Thus IGullp also exists for all p> 1. The symbol T used as an algebraic quantity denotes the length of the interval T, i. e., t1 - to. The norm of a function is defined in the way shown here, rather than without the factor 1/ T, because this definition allows considerable simplification in the presentation of certain later results. 80

81 either functional analysis or classical variational techniques. ) Assume that this related problem could be solved, and that expressions could be obtained for x(t), u(t), etc., having p as a parameter. Since the cost functional for the related problem approaches that of the original problem as p approaches infinity, it is not unreasonable to hope that the solution(s) to the related problem will likewise approach (in some meaningful and useful sense) the solution(s) to the original problem. Should this indeed prove to be the case, it would open up two new possibilities: a) It can happen in the minimum peak amplitude problem that there is no unique optimal control. (See, for instance, Example 1 of Chapter III.) This can give rise to computational difficulties, since Thm. 3. 2 does not completely specify the optimal control in such cases. If it could be shown that the related linear problem always had a solution which could be obtained from some simple expression (such as the first line of Eq. 3.12), then there might be computational advantages inherent in a procedure which approximated the desired solution by a solution of the related problem for large p. This would be especially true if means were available for estimating how far from optimal the result was. b) The functional analysis techniques used in Chapter III to solve the original linear problem do not apply directly to nonlinear problems. However, the related problem can be solved by classical variational techniques, which apply equally well to linear and nonlinear problems. Therefore, we might hope to be able to solve nonlinear minimum peak amplitude problems by setting up a related nonlinear problem, solving it by classical techniques, and then passing to the limit. This possibility is investigated in Chapter VI. 45 4This procedure is analogous to that used by Kirillova (Ref. 35) for proper systems with a single control variable.

82 The proposed approach is analogous to the interchanging of an integration and a limiting process in the calculus: Here we wish to interchange an optimization process and a limiting process; that is, instead of solving an optimization problem involving a limiting form of JIG ulp, we wish to solve an optimization problem involving IGullp and then pass to the limit. And, as in the analogous interchanging of integration and limit, it is by no means certain that the procedure is valid. Its validity will be established for the linear problem. For the nonlinear problems considered in the next chapter, however, corresponding results are not available, so that for such problems this approach must be considered as a heuristic or practical tool, rather than a rigorous procedure for determining optimal controls. 5. 2 Statement and Solution of the Related Linear Problem In the original problem, as formulated in Chapter II, the optimal control was to be sought among the set of essentially bounded measurable controls which caused the system to satisfy the desired boundary conditions. For the related linear problem, 46 it will be convenient to choose as the class of admissible controls not the class of essentially bounded controls but the class of controls for which the integral S IG(t)u(t) P dt T exists. (This is, of course, a different class for each different choice of p, but in every case it includes the class of essentially bounded measurable controls as a subclass.) This broadening of the class of admissible controls would seem to open up the possibility that the optimal control for the related linear problem could turn out to be not essentially bounded, a result which would offer computational difficulties if nothing else. However, we shall show that for the kinds of systems considered here (see Sec. 2. 1), the optimal controls for the related linear problem are essentially bounded, so that this difficulty does not occur. 6The results obtained here rest on the fact that the space of admissible controls is a Banach space. This choice of the class of admissible controls simplifies the proof that the control space is a Banach space. See Appendix E.

83 We now proceed to a statement of the related linear problem: Let Lr be the space of r-component column vector functions of time p z(t) defined on the interval T which are integrable in the above sense, and with norm defined by ZIp = [4 If z(t) Pdt]P (5.3) T ~r ~ 47 Then Lr is a Banach space. p Let U (a, b, T) be the set of all controls u(t) for which G(t)u(t) is in L p- p and which cause the system defined in Sec. 2. 1 to satisfy the boundary conditions x(to) = a, y(t1) = b. Choose as an optimal control any element of U (a, b, T) for which |Gu ip takes on its minimum value over all u(t) in Up(a, b, T). In order to simplify the notation, we proceed as in Chapter III and define z(t) = G(t)u(t). The related linear problem is then equivalent to the problem of choosing a z(t) in Lr satisfying Eq. 3. 7 for which Ilzlp is a minimum. The existence of an optimal control for this problem follows at once from Neustadt's Theorem 1 (Ref. 77), or can be proven by steps analogous and in many cases identical to those used in the proof of Theorem 3. 1 above (See Sec. 3. 2). The form of the optimal control(s) can also be obtained from Neustadt's Theorem 1 (Ref. 77), or by steps analogous to those used in Section 3. 4 for the minimum peak amplitude problem. The details of these proofs are omitted here. We present instead only a lemma analogous to the lemma of Sec. 3. 4, which can be regarded as a generalization of Holder's inequality to the Banach spaces considered here. Lemma: Let z(t) and v(t) be r-component vector functions of time defined on the interval T and belonging to the spaces48 and L a respectively, P 47 See Appendix E for a discussion of this space. 48 The vectors v(t) that we are actually concerned with in these problems are bounded in Euclidean norm, and hence belong to Lr. The fact that L& contains other functions which are not bounded or even essentially bounded is of no importance here.

84 where 1 < p < o and q = pP —. Let T1 be the subset of T on which Iv(t)l i O. Then 1 1 f v(t) z(t)dt < [1 I z(t) IP dt]P[ I v(t) Iqdt]q T T T Furthermore, equality is taken on in this inequality if and only if z(t) satisfies K I v(t) q2 yT(t) for almost all t e T z(t) = 0 for almost all t e [T - T1] where K is some nonnegative constant. Proof: f v z dt < If v zdt] < 1 f v zl dt <1 f Iv(t)l Iz(t) dt T T T T T T T T 1 1 < [T f Iz(t)lp dt ]P [ f Iv(t)lq]q = |zlpl|V|q T T T The last inequality is the usual form of Holder's inequality for integrals. The first inequality becomes an equality if and only if f v z dt is nonnegative. The second T becomes an equality if and only if the real scalar quantity v(t) z(t) is of the same sign almost everywhere on T (excluding points at which it is zero). The third becomed an T equality if and only if v(t) and z(t) are colinear a. e. on T. The last inequality becomes an equality if and only if Iz(t) I = Klv(t) l 1 for almost all t e T, where K is a positive constant (See, for example, Appendix A of Ref. 36). These four conditions combine to give the stated result. Q. E. D. Using this lemma and steps analogous or identical to those used in the proof of Theorem 3. 2, Section 3. 4, we conclude that a control z (t) of the form -p

85 C(1T 1 {C IvTc p- 1 Tc for almost all t e T () z (t) = p pp (5.4) -P 0 for almost all t e [T - T1(Cp)] is an optimal control for the related linear problem, where C is a positive constant, -p cp is a constant k-component column vector of nonvanishing Euclidean length, V stands for V(t, t), T1(cp) is the subset of T on which IVTc I O, and where C andc satisfy the boundary condition equation T -1 + 1 T g =C f IVTc VV c dt (5.5) Here the subscripted p in C and c serve as a reminder that these quantities are p -p different, in general, for different choices of p. For convenience in later sections, we normalize c to unit Euclidean length, making the appropriate change in C in the process. We note in passing that nothing has been said about the uniqueness of the optimal control given by Eq. 5. 4. This control turns out to be unique, but this fact is not important to the use to which these results are to be put, and therefore is not investigated here. The following theorem is needed in the next section: Theorem 5. 1: The optimal control z (t) for the related linear problem defined by -p Eqs. 5. 4 and 5. 5 is essentially bounded in Euclidean norm; i. e. 1ii,= ess. sup. I z(t)/<. llp - t e T - Proof: Since z (t) is by definition in Lr, the quantity |Lz | is a finite number. From -Eqs. 5. 3 and 5. Eqs. 5.3 and 5.4,

86 1 1 Iz ip = C [ 1 f T T IP -ldt] Since V is bounded in norm for all t e T (due to the assumption of Chapter II), since c is a unit vector, and since and - are positive numbers for all p allowed here -p P- i p (i. e., 1 < p < oo), the bracketed quantity and its ()th power are finite numbers, which implies that C is finite also. P Since z (t) can differ from the expressions given in Eq. 5. 4 on a set of -p measure zero, at most, it follows that 1 l ess. sup. [C I VTc p - ] pll'- tET p -p As n a C a are boted above, C and is i anite both bounded, andpositive number, p -p p - which implies that z pl is finite also. Q.E. D. 5. 3 The Limiting Process It is shown in this section that the solution z of the related linear problem -p can be made to approach the minimum peak amplitude solution in cost to any desired degree of accuracy, by choice of p sufficiently large. Furthermore, an upper bound on the difference between the peak amplitude of z and the minimum peak amplitude is -p obtained, so that this becomes a useful computational technique for obtaining nearlyoptimal controls. Before presenting the main theorem of this section, we prove certain lemmas that are needed for the proof of the theorem. Lemma 5.1: For all essentially bounded measurable functions z(t) defined onT = [t0,t1], andfor 1 < m < n < oo 11Zll < z 11ln where lzllZ_ is defined49 as [ess. sup. I z(t) ]. (Note that m and n are pt.E t ] 49 Note that Izlooz is the same as the quantity Illz defined by Eq. 3. 6. "- oo -

87 not necessarily integers, and that the n used here has no connection whatsoever with the n used in Chapters I through IV to denote the number of components in the system state vector. ) Proof: First of all, we note that the quantities appearing in this lemma are defined for all m and n in the given range. Also, for m = n the lemma is trivially true. Thus, we confine attention to cases where m < n. We proceed by first proving a similar result for real scalar functions v(t); namely, that 1 1 a) [ / Ivv(t) mdt]m < [T f Iv(t)dt] n< oo T T T 1 b) [ f v(t) Imdt < ess. sup. Iv(t)l T tT where v(t) is any essentially bounded measurable real scalar function defined on T. For part a), we apply Hblder's inequality (in the usual scalar form) to 1 Iv(t) mdt= 4,f 1I i Iv(t)lmdt T T using p n and q = n (We note that, for the given ranges of m and n - m p- n, p is greater than one and less than infinity, so that Holder's inequality is indeed applicable. ) Thus 1 1 41 I v(t)Imdt [1 f lIPdt]p T f Iv(tdt T T T m [1 f Iv(t) ndt] n T which yields 1 1 [1 f v(t)Imdt]m < [1 f Iv(t)lndt]n as claimed. T T For part b) we proceed as follows: By definition, Iv(t)l < ess. sup. Iv(t)I for almost all t e T, so that Iv(t) Im< [ess. sup. v(t)I]" for almost all t T. Therefore for almost all t e T. Therefore

88 J^ ilv(t)|dt S< [ess. sup. Iv(t)I] mdt= [ess. sup.iv( I]m 1 1 v(t)Imdt < T T Taking the mth root of both sides, we have finally that 1 [1 f Iv(t) mdt]m < t e T Iv) as claimed. T Now, since the Euclidean length of a vector z(t) is just a real scalar function of t, we can replace Iv(t) by Iz(t) I in the above expressions, and thereby obtain the inequalities which were to be established. Q. E. D. Comment: This inequality is not true, in general, if the definition of the norm does not include the factor. (To prove this, take v(t) = 1, T = 2, m = 1, n = 2.) It was to simplify the statement of this and the following lemmas that this factor was included in the original definition of IZp. Lemma 5. 2: a) ||zmllm < z 1< m < o b) zlfloo <_ lZ oo where z (t) is the optimal control of the related linear problem with p set equal to m, z(t) is a minimum peak amplitude control, and z(t) is any control satisfying Eq. 3. 7 (i. e., causing the system to satisfy the desired boundary conditions). Proof: In the related problem, the cost associated with a control z(t) is j|z $m; similarly, in the minimum peak amplitude problem the cost5 associated with a control z(t) is [z[ 1. These two inequalities are thus merely statements of the facts that z (t) and z(t) are optimal controls for the corresponding problems. 50 precisely true, since the quantity involves the essential This statement is not precisely true, since the quantity [Izlloo involves the essential supremum of I z(t) I, whereas the peak amplitude was originally defined in terms of the supremum of z(t) I. However, as discussed in Section 2. 2, we here consider the controls which result from these two cost functionals to be equivalent, and hence make no distinction between the two cost functionals.

89 Lemma 5.3: a) ]lz mll < Iz nllm < lz nll < lzl 1 < m < n < oo / m Izn~m ~ I <_ <m < -mn b) Ilz _ < Izilm < ||z||I < ||MIz _ 1 < m < co Proof: The first inequality in each line follows from Lemma 5. 2, the second from Lemma 5. 1, and the third from Lemma 5. 2. n 1 Lemma 5.4: lim [ IVTc I dt = 1 n-co T where c is the unit vector associated with the optimal control z (t) of -n -n the related problem, and where the usual absolute value metric is understood in connection with the limiting process. Proof: Define |IVI| = ess. sup I sup IVTc (Since c is a unit vector and since tE T Ic =1 every element of V is a bounded measurable function of t, IIVII is a finite number.) Either IIV I > 1 or |VII <<. If IIV I < 1, then n n IvTc n- 1 < vnl - 1 < 1 for all n > 2, for all c, and for almost all t e T. If IVII > 1, then n n IVTc n- 1 < lVlln - 1 < llV12 - n vl for all n > 2, for all c, and for almost all t e T, since for such values of n the exponent n is less than or equal to 2. Thus n-1n n 4 f IVc n- 1dt < max[i f dt, 1 f IV 2dt] = max[l,l IV2] < oc T _T T T n On the other hand, this quantity 4 f IVTc In dt is bounded away from zero. We prove this as follows: The complete controllability assumption (see Appendix D) states that the matrix [ S VVTdt] is positive definite. Its minimum eigenvalue is thus a T positive number, which we denote by q7. Then for all c such that I c = 1,

90 cT[ VVTdt]c = - f IVTdc2dt > 1 Il12 = T > 0 T T T T i T T Applying Holder's inequality to this, with p = 1 and q = oo, gives T f IVTc 2 dt < [ T f IvTc dt] [ces sup. IVTcI] < [ f IVcl dt] IIVII =IIV VII T T T T t T T T T T T so that [V cll T> l. Now, using this result in Lemma 5. 1, and using VT c for z, 1 for m, and n 1 for n in this lemma, gives n-i n n- lyvT n _= f Jo n-ldt] n T_ _ T f _VT] n- 1 T Raising both sides to the n - 1 power gives finally n n [1 f iv In - I d7r/ ]n- 1 > 0 [~ f IvTc[ dt] >[ n 1 TIIT n as claimed. Therefore, since the quantity [' f IVTcnl n - dt] is uniformly bounded T and uniformly bounded away from zero for all n > 2 and for all c n' and since the exponent e approached zero as n approaches infinity, the statement of the lemma follows immediately. bounds provided by Lemma 5. 3). Let the limiting value be denoted by C for the n-0 [2h i l Z n/l exists and is equal to C, the optimal costfor the corresponding minimum peak amplitude problem. (Here, as above, the metric involved in the limiting process is understood to be the usual absolute value metric. ) Proof: First note that the limit exists, since, from Lemma 5. 3, IZn n is a monotone nondecreasing function of n bounded above (by 11_z11 = C, to name one of the many upper bounds provided by Lemma 5. 3). Let the limiting value be denoted by C for the co present. From Lemma 5. 3, we have that ll Inlln < llzIIoo which can be rewritten as n 1 I-nln = Cn [ T VTc n- 1 dt]n < C TVcn n i n -

91 Since, by definition, lim |z = C, and since, from Lemma 5. 4 n- oo lz-nln oo00 n 1 n [ vc I VT " In- dt] =1 n-oo T T - n it follows at once that lim C = C and that C < C. n- o o o00 00 Lemma 5. 3 also states that lZIz ~o ~ lnlloo= C [ess. sup. IvTc n- ] -= n t E T -n It is clear from the definition of |IVII that IVT CI < |IVI| for all unit vectors c and for almost all t e T. Since c is a unit vector, it follows that, for n > 2, -n 1 1 IVTc n 1 -I< IV I 1 for almost all t e T - n 1 1 1 Therefore [ess. sup. IVTc In-1 < [ess. sup. I - ] = V - Thus t E T -n t E T 1 lizll = C < lznlI < Cn lIVIln - 1 But, as n approaches oo, C approaches C (from [-zl= - oo - n 1 n 00 the first part of this proof), and l[VIin - 1 approaches j[V|]0 = 1, because IVI is a bounded and nonzero number. Thus C < C. Combining these two inequalities involving C and Cc gives C < C < C, which of course implies that C = C. Q. E. D. With little additional effort we can also obtain the following useful result, which implies that the sequence of controls z (t), n = 1, 2,... come arbitrarily close to C in peak amplitude, and hence that this sequence is a minimizing sequence in the sense of Section 1. 2. 4. Corollary: For every e > 0 there exists a number N(e) such that | Iznloo - C < e for all n > N(e). That is, we can obtain a control satisfying Eq. 3. 7 which has a peak amplitude as close as we like to the minimum peak amplitude, simply by choosing n large enough and using the optimal control z (t) for that problem. 1 Proof: As stated in the proof of Theorem 5. 2, lzn ll lies between C and Cn lV n- But Cn IVII 1 approaches the limit C as n approaches infinity, as is also shown in the

92 proof of Theorem 5. 2. Therefore \Iz nll also approaches the limit C as n approaches infinity. The statement of this corollary is simply another way of phrasing this fact. Q. E. D. These results, taken together, lead to the following conclusions: The minimum peak amplitude control can be approximated (in the sense that the costs are close) to any desired degree of accuracy by solutions of the related linear problem. Furthermore, for any choice of n, Lemma 5. 3 provides a bound on the amount that the actual peak amplitude of z (t) can exceed C. To illustrate this point, suppose we pick a value of n, compute zn(t), and determine IZ nlln and LIZ n| O. By Lemma 5. 3b), C must lie between z nin and itlz nl. Thus, the peak amplitude for the control z (t) that we propose to use in place of z(t) is |znll, and we know that C can not be less than n.1'00 IZnl ln We are thus within [ Iz nll - lZnin] of the minimum cost C. If this quantity is sufficiently small to suit us, we say that z (t) is a sufficiently good approximation to -n the optimal control. All this does not imply that the control z (t) converges [ in the metric — n defined by the norm given in Eq. 3. 6] to z(t) as n approaches oo. This is a point that is not resolved one way or the other by the above discussions and proofs. In fact, the examples of the next section show that, in general, one can not expect z (t) to converge in this metric to z(t). This point will not be investigated further here, except to the extent that the examples of the next section give insight into the matter. 5. 4 Examples of the Limiting Process: The limiting process discussed above will be illustrated by its application to some examples. Example 9: Consider the same system used in Example 1, with initial conditions x(O) = 0 and desired final conditions x(2) = [ 2, ]. Example 1 shows that this problem does not have a unique minimum peak amplitude control, and that any control satisfying

93 f4 0 < t < 1 ul(t) = 0 0 1 < t < 2 u2(t)= 0 O t< 1 (5.6) lu2(t)l < 4 1 < t < 2 2 f (t-1)u2(t)dt = is a minimum peak amplitude control. The optimal control for the related linear problem [as given by Eq. 5. 4, with u(t) = z(t)] is u(t) = CplVT(2,t)c p- 1 VT(2, t)c -P -p Upon denoting the two components of c by c1 and cp and making use of the expressions for V(2,t) given in Example 1, this becomes C sgnc I(l- t)c1lP-1 < t 0 1 < t < 2 ~ O< t < 1 u2(t) = 1 C sgn c2p I(t - l)c2p 1 < t <2 Use of Eq. 5. 5 to evaluate Cp, cp, and c 2p and normalization of c to unit Euclidean length yields C = [2+ 1 ][jlglj2- 2+a g2I -2]2p- 2 pP-1 ] - 2clp = [gngl][+ - 2] - 2 c2p= [sgng2] [1 + g 2p- 2] 2p- 2 Finally, setting g1 = 2 and g2 = 1 and substituting these results into the above expressions for ul(t) and u2(t), we have

94 1 2[ 2 + 1 - itpt] < t < 0 1 t < 2 u2(t) = ( O< < 1 _ [2+ I-][t- 1]P-~ 1 < t < 2 Fig. 5. 1 shows plots of these functions for various values of p. As p approaches co, they approach the functions ul(t) 6- p=2 p=3 p=21 p p=oo 3 0 1 2 u2(t) p=2 p=3 3 p=4 p=2c P =CC 2 ~~~~~~1~~~~~~~~~~~~~~~~~~~~ t 0 2 Fig. 5. 1 The limiting process illustrated for Example 9.

95 4 < t< 1t ul(t) = 0 1 < t < 2 (O O<t< 1 u2(t) = 2 1 t< 2 which can easily be shown to satisfy Eqs. 5. 6. Furthermore, they approach these limiting functions (in the sense that the area between these curves and the limiting curves approaches zero, and in the sense that the peak amplitude of these curves approaches that of the limiting curve) with reasonable rapidity. For example, half of the total reduction in peak amplitude that can be achieved in going from p = 2 to p = oo is already accomplished for p = 3. Also, for p = 21, the control function ul(t) already looks very much like the optimal rectangular function, and the peak amplitude is within 2. 5 percent of the minimum. It is instructive to consider the bounds on the minimum peak amplitude provided by the solution of the related linear problem for various values of p, as discussed in the last section. Fig. 5. 2 shows the upper bound |u pllo and the lower bound lUp |p for this example for various values of p. The actual value of C is = 4. Two other points are worthy of mention here: 1) If instead of looking at the functions u (t) as p increases, we look instead at the vector c, which completely determines the shape of u (t) for each p < oo, we see that c P -p -p approaches [ 1, 0] T as p approaches oo. Substitution of this result into Eq. 3.12 does not completely determine an optimal control. Thus, in this example the functions u (t) -p 51 converge to a particular function, but the corresponding limiting process in c does -p not uniquely determine this or any other function. 2) Any control involving u1(t) = 4, 0 < t < 1, u1(t) = 0, 1 < t < 2 along with any one of the functions u2(t) shown in Fig. 5. 1 is an optimal control, since it has peak amplitude equal to 4 (the minimum) and it satisfies the desired boundary conditions. However, the limiting process tends toward a particular function for u2(t), which we note to be of the The convergence is not uniform, of course, because of the behavior of up(t) at t = 1. However, the sequence up(t), p = 2, 3,..., converges to u(t) in the metric defined by any of the norms 1 lip, for 1 < p < oo.

96 same rectangular shape as the uniquely-determined control component ul(t). This is both an advantage and a disadvantage: an advantage in that the process does generate a specific and unambiguous control function; a disadvantage in that the possibility of secondary optimizations, as discussed in Sec. 3. 5, is lost. cost 6 upper bound Ilup 110 (both examples) 5+ 4. 3. 21 10 lower bound l1u 11 -P Example 9 asymptote p 0 2 3 45 10 20 50 100 Fig. 5. 2 Upper and lower bounds on minimum peak amplitude provided by related problems for Examples 9 and 10. Example 10: Consider the same system used in Example 3 (the double integrator) with initial state x(0) = 0 and desired final state x(1) = [ 1, 0] T. The minimum peak amplitude control for this example is

97 u(t) =1 -4 < t <1 The solution to the related linear problem is 1 up(t) = 2(2 + p ) 11- 2tP 1 sgn[1- 2t] Cp = [-.8, iT 1 Cp = 2(2+-! )(.2)2P 2 |upIL| = 2(2- C = 2(2+ 1) "lp p - (All of these results can be verified by means of Theorems 3. 2 and 5. 1 and the results of Example 3.) Fig. 5. 3 shows u (t) for various values of p. Because the bounds on the p minimum peak amplitude provided by |lu p| and |u p100 are (by coincidence) the same as pp or similar to the corresponding results for the previous example, these bounds for this example are also plotted in Fig. 5. 2. Note that in this example c is always the same, independent of p, and -p that the functions u (t) resemble the minimum peak amplitude control more and more as p grows larger. Example 11: Consider the same system used in Example 5, namely x = -x + u, with fixed final time t1= 1 and with boundary conditions x(O) = 0, x(1) = 1. The minimum peak amplitude control for this problem, as derived in Example 5, is i1 e u(t) = = C 1 - e The solution to the related linear problem is

98 u(t) 6. p p= p=2 p0 p== Fig. 5. 3 The limiting process illustrated for Example 10. 1~-1 ~~ t- 1 - p=2 p=3 -2 — p=4 -4 -5 -6 Fig. 5.3 The limiting process illustrated for Example 10. t- 1 u (t) = C ep- 1 p p iupp = C Suplloo= cp where C =q q= q- and q= 1+ P 1- e- eqp_- 1

99 -t +t These results follow from Eq. 5. 4, using V(tl, t) = e. Fig. 5. 4a) shows u (t) for various values of p, and Fig. 5. 4b) shows the behavior of the upper and lower bounds on C as a function of p. Since in this example u(t) does not involve any "switchings" (i. e., step discontinuities), the convergence of u (t) to u(t) is uniform on the open interval (0,1). ul(t) f p=c~ fp=17.p= 5 1.0 Fig. 5. 4a) u (t) for various values of p for Example 11. p=5 p =9 p=17 -=3 0.5 1. O Fig. 5.4a) Up(t) for various values of p for Example 11. cost 2 1 upper bound Ilu 11 _, P L~ asymptote C lower bound Ilu 11 P P 0 2 3 45 10 p 20 30 40 50 100 Fig. 5. 4b) Upper and lower bounds on C, as a function of p.

CHAPTER VI NONLINEAR PROBLEMS 6. 1 Introduction There is no rigorously established technique presently available for the solution of minimum peak amplitude problems for nonlinear systems. Variational techniques are not readily applicable because the cost functional is not expressible as an integral, dynamic programming fails for the same reason. Functional analysis methods for nonlinear systems are not sufficiently well developed to allow the solution of minimum peak amplitude problems. The method to be presented, while it is also lacking in rigor, may be of engineering value in some problems, in that it provides a "likely candidate" for the optimal control, which in some cases can be shown to be optimal by other tests. The method is essentially as outlined in the previous chapter: Set up and solve the related nonlinear problem for various values of the parameter p (using variational techniques, in this case), and then either use the optimal solution to the related problem for large p as an approximation (who knows how good an approximation) to the minimum peak amplitude control, or else take the limiting form of this solution (if it exists) as p approaches infinity as the "candidate" for the optimal control. As has been suggested above, the difficulties that could be encountered in this approach are many. To name a few: There may be no minimum peak amplitude control (only a minimizing sequence); the related problem may not converge to the minimum, or it may converge to a local but not an absolute minimum; or the related problem may have no solution. The many powerful theorems that were available in the linear case to eliminate such possibilities are not available here. Even such basic assumptions of the linear problem as the complete controllability assumption have no convenient analytic formulation in the nonlinear case. The test of the method must therefore be in its ability to obtain results. It is not the intention of this work to investigate this problem in detail. The example given 100

101 later in this chapter serves to indicate that the method has some validity, nonetheless. 6. 2 Statement of the Nonlinear Problem Since the proposed limiting method involves the use of variational techniques, we restrict attention to a class of systems to which these variational techniques (in their present state of development; see Pontryagin, et al., Ref. 13, p. 79) are applicable; that is, we assume that the physical system to be controlled can be characterized by the system of differential equations x = f(x,ut) (6.1) with the output y(t) being given by y(t) = D x(t) (6.2) Here f (x,u, t) is a vector function of the vectors x and u and the scalar t, with components f1(x,_u,t),.,fn (x,u,t) which are defined for all x En, u e E, and t e T = [t0,t1]. Furthermore, each component f.Yx,u, t) of f (x,u, t) is assumed to be continuous in all its variables and continuously differentiable with respect to t and to the components x,...,xn of x. The matrix D is as defined in Section 2. 1. The problem is then to choose, from among all the bounded measurable controls u(t) which cause the system to transfer from the initial state x(t0) = a (6.3) at initial time t0 to any state for which y(tl) = b (6. 4) at final time tl, any one for which ess. sup G(t) u(t) tE T [a(t) u(t)I is minimum. The matrix G(t) is as defined in Section 2. 1.

102 6. 3 Statement and Partial Solution of the Related Problem In exact analogy to the related linear problem of Chapter V, we define the related nonlinear problem as the problem identical to the problem of Section 6. 2 except that the functional to be minimized is IIGulil = [ IG(t)u(t)lPdtP 1 < p < o0 p T T instead of the essential supremum of IG(t) u(t)j. The functional |IGullp is still not of a form to which the usual variational techniques apply directly, but this presents no problem, because the minimization of [IGull is completely equivalent2 to the minimization of - P J(u) = f G(t) (t)| dt 1 < p < oo T a functional to which variational techniques do apply. We now apply the maximum principle of Pontryagin to obtain the following necessary condition for optimality: A necessary condition that a bounded measurable control u(t) and the corresponding x(t) and y(t) satisfying Eqs. 6. 1, 6. 2, 6. 3, and 6. 4 be optimal is that there exist a nonpositive constant,0' a constant k-component column vector 0, and an absolutely continuous nonzero vector function _(t) with components il(t),...., n(t) satisfying 52To convince oneself of the equivalence of these two functionals, consider that a cost functional merely provides an ordering of the controls u(t). Suppose that, in the ordering provided by ILGullp, the control u1(t) is lower in cost than u2(t); i. e., ILGuillp < lGu211|p. But this implies that [ |LGuljp]P < [l]Gu2l|p]P also, and hence that J(u1) < J(u2). By similar arguments, one sees that J(u1) < J(u2) implies IIGullp < IIGu211p, and that J(u1) = J(u2) implies and is implied by IIGul lip = IIGu2 lp. Thus, the ordering imposed by one cost functional is identical to the ordering imposed by the other, and the two cost functionals are equivalent, in the sense that controls which are optimal for one are optimal for the other, and vice versa.

103 a H(x(t), 9(t), u(t), t) i - axi(t) t e T 4'i(t1) = Ja i= 1,..n (6. 5) 1 ~tl)ax1(t1) such that, for the stated x(t) and g/(t), and for almost all t e T, the quantity H(x(t), (t ) (), v, t)=IGt) + Tt)x(t (6. 6) regarded as a function of the variable v, attains its maximum (over all finite v) at the value v = u(t). Here the terminal constraint function Q is defined as Q = T[D x(t)-b] (6.7) Carrying out the operations indicated in Eq. 6. 5 gives [af afn Qi = a- x,** * ax. (t) L 1 4i(t) d= i'd2 *dni] 0 where dij is the element of the matrix D in the ith row and jth column. In vector matrix form, these equations can be written as = - Jx(x,, t) (t) (6. 8) (t1) = D T (6.9) where Jx(x, u, t) is the Jacobian matrix of f with respect to x, i. e., af1(x,, t) af1(x,_u,t) axl ax a(fl, f 2..,fn) Jx(xut) = a( x2. x n) = af(X, u, t) afn(x,u, t) axl ax n.

104 Thus far, the problem has been transformed from one involving n + k + r unknown functions of time x1,... *,xn, Y1,..', u1,.. I,ur, n differential equations (Eq. 6. 1), k finite equations (Eq. 6. 2), and n + k boundary condition (Eqs. 6. 3 and 6. 4), plus the requirement that a functional of u be minimized, into a problem involving 2n + k + r unknown functions X1,... x, 4'1...'n Y1'...' k' u...*,ur, 2n differential equations (Eqs. 6. 1 and 6. 8), k finite equations (Eq. 6. 2), 2n + k boundary conditions (Eqs. 6. 3, 6. 4, and 6. 9), k + 1 unknown constants /0, 019...' 0k Iand the condition that u be chosen to maximum H at almost every t e T. This last condition serves to determine r conditions that u must satisfy a. e. on T. When we take into account the fact that the equations determining 4(t) are homogeneous in 4(t), so that the length of ik(t) may be arbitrarily specified at any one moment in time, we see that the transformed problem is a "well specified" problem, in that, in principle at least, there are enough differential equations, finite equations, conditions, etc., to specify all variables and constants of the problem. Whether this transformed problem does indeed have a solution is a question that can not be answered until more information about the system is given. So that we may proceed to a more detailed discussion of the nature of the limiting process, we now restrict attention to systems for which the components of f (x, u, t) are not only continuous in u but continuously differentiable in u also: Under this assumption, the stationary points of H with respect to u can be found by setting the partial derivatives of H with respect to u equal to zero. Proceeding in this manner, we have that u(t) must satisfy a T 2 T p O (0 G Gu) + f t) = 0 j = 1,...,r or au. - ~~~/ (U g ~u II. r1_(xu t)0j1, p TT021 + ]Gu) 2 0(ug ]G j=1,...,r af n(x,u, t) au. _ - _L

105 which can be written in vector-matrix form as P-2 P o0 |Gu G TGu + J(x,u,t)j 0 (6.10) where Ju(xu, t) is the Jacobian matrix of f (x,u, t) with respect to u, i. e., af (x,u,t) afl(x,u,t) au.''' au Ju(X,,t) = afn(x,u, t) afn(x,, t) au''' au _ auk Ou 1 r By solving for Gu, taking the Euclidean norm of the result, and using this result to eliminate the quantity GuP2 in Eq. 6. 10, we can rewrite Eq. 6. 10 as -__[1 ]P |-1+ p-l T lp- T T IT T Gu -TJ LJJ G Ju j (6. 11) a result which can be recognized as similar to the expression for the optimal control for the related linear problem, as given by Eq. 5. 4. In the nonlinear case, this is not an explicit expression for u, since J (x,u, t) involves u, in general. (The exception here is of course when f(x, u, t) is linear in u; that is, when f(x,u, t) is expressible asf(x, t) + B(t)u, in which case Eq. 6. 11 is an explicit expression for u, and resembles the results for the related linear problem even more. ) Thus, in general, if u can not be solved for explicity, an iterative procedure must be used in order to determine the optimal u(t) (if it exists) from Eq. 6.11. Also, since this is only a necessary condition that H have a stationary point at u, one must check to see that this u does indeed correspond to a maximum of H, and that no other u gives a higher maximum —a possibility common to all maximizations involving nonlinear functions. 6. 4 Two Examples The examples considered here are admittedly very simple ones, and the behavior of these examples can not be assumed to parallel that of nonlinear problems in their full generality. Simple examples were chosen so that the solutions could be obtained in analytic form, thus facilitating the study of their behavior under the limiting process.

106 Example 12: The system for this example is the same double integrator used in Examples 3 and 10, but with a nonlinear input; namely, X1 = X2 x(O) = o X2 = 3 x(1) = [2,0]T In the related problem we wish to choose u as a function of time so that these boundary conditions are satisfied and so that 1 1 ]p 1 o pr P II Ullp = T |G(t) u(t) dt = u(t) dt is minimized, or equivalently, so that 1 J(u) = f lu(t)Pdt 0 is minimized. Applying the maximum principle to this problem gives H = %0 1uIp + lX2 + 42 u3 Q = 01[x(1) - 2] + 02x2(1) 1 = 0 ~1(1) = 0 ~2 = -41 /2(1) = 02 choose u(t) to maximize H as necessary conditions for optimality. The 4 equations are easily solved to give 41(t) = 01, 42(t) = 01 + 02 - 01t, where 01 and 02 are constants. Equation 6. 11 could be used to give the form of the optimal control u(t), but in this case it is just as easy to repeat the steps used to obtain Eq. 6. 11, solving for u(t) explicitly in the process: aH = 0 = plp-2 2 au - 0 =4pu u + 3u +3 2 8-''

107 When u 0, and for p > 3, this can be solved to give 1 1 0-~P-3 Up(t) = bp p]2(t)l sgn 2(t) (The restriction of p to values greater than 3 is not a problem, since we plan to use large values of p anyway. For p < 3, different techniques must be used to find the value of u 3 1/p-3 which maximizes H, but these will not be discussed here. ) Letting the quantity L[ be denoted by Cp, and making use of the above solution for 4,2(t), then gives 1 u(t) = Cp 01+ 02- 01t P -3 sgn[ 1 + 02- 0t] Since the unforced system is linear in this case, the response of x for a given u(t) can be determined by taking the convolution of the forcing term with the fundamental matrix of the 53 linear system;53 namely t x(t) = X(t,t0) x(t0) + f X(t,s) [forcing term] ds to which in this case becomes 3 t t -st rt-s -t-s p-3 x(t) = f ts u 3(s)ds = C t tj 10 + 0- 2 01s sgn[01 + 2 - 01s]ds OJ _1+a,-BS sgn[1+ 02- 1s 1ds At t = t1 =1, this becomes 3 2 1 i-s p-3 x(t ) = L = C f 1 + i02 01s s sgn[01 + 02 01s]ds x(tl) = 0 P 0 1In order to force x2(1) to be zero, it turns out that we must choose 01 and 02 so that 01 + 02 - 1 01t switches sign at t = 2, and in order that x1(l) be positive, this switching must be from positive to negative. Thus, we may set 0 1 = 2, 0 2 = -1, and obtain finally that 5See Coddington and Levinson, Ref. 57.

108 3 1 p-3 x1(l) = 2 = C3 [-s] [1-s] sgn[-2s]ds P 0 which can be integrated and solved for C to give 1 Cp =[4(2 + p33) P L Using the above-stated values of 01 and 02, the following results are also obtained: 1 p-3 Up(t) = Cp 1-2t | sgn[l-2t] 1 p-3 IlU =: C ess. sup. 1-2tl = C Ip i p t e T p ||upII= C[ [2 + p-3 P These results are plotted in Figs. 6. l(a) and 6. l(b), from which it can be seen that u p(t) approaches the function 2 0 < t < u(t) = -2 - < t < 1 as p approaches infinity. That this is indeed the optimal control for the nonlinear problem can be shown by working the inverse problem (i. e., time optimal problem) using the constraint |u(t)I < 2- and noting that for any E > 0, the goal [2, 0] can not be achieved in a time t1 < 1. We have thus shown a control with peak amplitude 2 which satisfies the desired conditions, and have shown that no control with smaller peak amplitude can accomplish this, which constitutes a proof of the optimality of u(t).

109 up(t) 3 p=23. 1 0 -1 -2 -3 t Fig. 6. l(a). The optimal control up(t) for the related nonlinear problem for various values of p, for Example 12. cost \upper bound flu IT p00 asymptote 2 1, ~ ~ ~ ~ ~ ~ ~ ~ ~ lower bound flu 11 P P 0 2 4 8 10 20 40 100 P Fig. 6. l(b). Bounds on the optimal peak amplitude provided by Ilu |loo and Ilu lp for Example 12. Pi Pilo i ip Note that this use of the inverse (i. e., time-optimal) problem to prove optimality may or may not work in a given case. For example, if the final time, tl, had been

110 specified at a time for which the minimum peak amplitude problem had no minimum time inverse [i. e., a time such as tld or tle or tlf in Fig. 4. l(a)] then the minimum time problem would indicate that the desired result could be achieved with the given limit on |u|, and in fact at a final time considerably less than the given final time. This would not disprove the optimality of the limiting u(t); it would merely be an inconclusive test. Example 13: In this example also, a system has been chosen for which the state x(t) can be written out as a known functional of u(t). The system in this case is -X x = -1 + e u x(O) = 0 x(l) = loge2 for which it can be shown that, for the given initial value, x(t) = log [e t + f et u(s)ds (6.12) so long as the expression in parentheses remains positive. Proceeding as above with the related nonlinear problem, we have as necessary conditions for optimality that u(t), x(t), and 4(t) must satisfy -x x = -1 + e u x(O) = loge2 4 = / e u u(t) must maximize H = 10 julp- 4/ + 4 e u where V0 is a nonpositive constant and V/ must be an absolutely continuous function of t which is never equal to zero on the interval [0, 1]. (This is merely a statement of the necessary conditions for optimality given in Section 6. 3, appropriately specialized to this problem.) aH The maximization of H with respect to u can be carried out by setting - = 0, and then solving for u, which gives 1 1 1 p- 1 i x(t eX(t)i p-1 u(t) = _ —p |I(t) ex sgn 4/(t) 0 Or

111 1 xp- Cp|j(t)e eX(t)L sgn i(t) All of these conditions are satisfied by t-1 u,(t) C eP- C = q(2- e-) up(t) = lo ep- C1e - (q 1) x(t) =log e I + -C e-t - (e -(t) = e1 + (2- e ) (e ) \ eq - where, as always, q = p For this control u (t), it is also easily shown that where, as always, p |lu u | = C I1- e 1p op p p q These results are plotted in Figs. 6. 2(a) and 6. 2(b). As p approaches oo, up(t) approaches the function 2 -e' 2e -1 u(t) 2- 0 < t 1 1- e which, upon substitution into Eq. 6. 12, is easily shown to satisfy the desired final condition. Furthermore, it can be shown by means of the inverse problem (see Chapter IV) that this u(t) is indeed the minimum peak amplitude control, by the same reasoning as was used in Example 12. Alternatively, it could be argued that since the function v = loge(w) has a unique inverse w = eV over the ranges involved here, the value x(1) = loge2 can be achieved only if the bracketed expression in Eq. 6. 12 equals 2 at t = 1. Linear systems optimization techniques can then be applied to the bracketed expression to show that u(t) is indeed optimal.

112 Fig. 6. 2(a). The optimal control u (t) for the related nonlinear problem for various values of p, for Example 12. p cost 51 4 I u 11 P 0 3 2 1 asymptote I u II PP 0 2 3 45 8 10 Fig. 6. 2(b). Bounds on the opti I ON 20 30 40 50 80 100 p Imal peak amp~litude Dro~viddc~ by Iluplloo and I|up|[p for Example 13.

CHAPTER VII COMPUTATIONAL ALGORITHMS AND EXAMPLES 7. 1 Preliminary Discussion For proper linear systems (defined in Section 3. 6) the problem of actually determining the numerical values of the minimum peak amplitude control in a given case is greatly simplified by Theorem 3. 2, since, according to this theorem we need determine only the direction of the constant k-component vector c and the value of the cost C (which is easily determined once c is known) in order to specify completely the optimal control almost everywhere on T, by means of Eq. 3.12. The first computational algorithm discussed here involves the determination of c and C for proper linear systems by steepest-descent techniques in k- dimensional Euclidean space. Before discussing this algorithm in detail, we make some preliminary observations and definitions to lay the theoretical groundwork for the algorithm: Let V T(tl, t)c h(c) = f V(t t) -- dt (7.1) T IT(t, t)cI cos = (g (c) (7.2) E = g g - C(c) h(c) (7. 3) E iT grad E = [ a E T (7.4) ac' ck where C(c) is chosen so as to minimize the error E for the given g and h(c). [This turns out to be C(c) = h((7. 5) - Ih(c)12 113

114 aE as can be shown by setting -E = 0 and solving for C(c). ] Using this value of C(c), Eq. 7. 3 can then be reduced to E = [1- cos 0] (7.6) It is assumed throughout this section that Ig I > 0. The set of points obtained by evaluating Eq. 7. 1 for all possible unit vectors c in Ek is the boundary aS1 of the set S1 defined in the proof of Theorem 3. 1, since these points are precisely those obtained by substituting into Eq. 3. 7 all controls of the form given by Eq. 3. 12 with C set equal to unity. (We need consider only unit vectors c because h(c) is independent of the magnitude of c, so long as c A 0. ) We now state certain results that are useful in the discussion of the convergence of the computational algorithms: The set of points aS1 with the k- dimensional Euclidean metric, and the surface aUk of the unit hypersphere (centered on the origin) of k E with the same metric, are metric spaces. Concerning these spaces we have the following Lemma: Eq. 7. 1 defines a continuous mapping of auk onto as1 for systems which are proper on the interval T. Proof: The mapping is onto because every point in aS1 is attained by means of an optimal control with cost C equal to unity [see part b) of the proof of Theorem 3. 1], and for proper systems every optimal control corresponds to a unit vector c (see Theorem 3. 3). The continuity of this mapping will have been shown if we can show that for every c and the corresponding h(c), and for every neighborhood NE = [h: h e aS1, Ih - h(c) I < e ] of h(c), there is a neighborhood N6 = [d: d aUk, Id - cI < 6 ] of c such that every point of N6 maps into N. That this is so can best be shown graphically. Consider a typical set S1 for a two-dimensional problem, as shown in Fig. 7. la). Consider the unit vector c and its image h(c). By Theorem 3. 2, c is an outward normal to aS1 at h(c). The neighborhood N of h(c) contains other points h of aS1, each with a corresponding normal direction. Because the system is proper on the interval T, the set S1 has no "flats" (see Theorem 3. 3). Thus these normals are not colinear, but lie in some (nondegenerate) cone, the limits of which are shown by c' and c" in Fig. 7. la). To find a satisfactory neighborhood N6 of c we therefore need only choose a 6 small enough so

115 that N lies completely in the interior of the stated cone; i. e., such that neither c' nor c" lie in N6. Such a neighborhood will always map inside N. This argument goes through whether or not the point h(c) is at or near a "corner" of S1, as is shown by Fig. 7. lb). Furthermore, the same arguments can be generalized to k- dimensional space. Q. E. D. Because h(c) is continuous in c, and because Ih(c)| > 0 for all c [see part b) of the proof of Theorem 3. 1], the scalar functions cos 0, C(c), and E are also continuous functions of c (with the appropriate Euclidean metric being understood in each case). This follows because continuous functions of continuous functions are continuous. It can readily be seen from Eq. 7. 1 that for c = c, where c is the unit normal corresponding to the goal g, h(c) will be colinear with g, which implies that cos 0 = 1, E = 0, and C(c) I Ih( On the basis of all this we can conclude that if the computer program generates a sequence of unit vectors c which converge to the unit vector c (in the Euclidean metric on Ek), then the corresponding sequences in C(c)h(c), cos 0, C(c), and E will converge to g, 1, C, and zero, respectively (with the Euclidean metric understood in each case). The convergence of all these sequences follows from the continuity of the various mappings involved (see Theorem 13. B, Simmons, Ref. 80, p. 76). Conversely, it is easy to show, by arguments similar to those used above, that if we have a sequence of values of E which converges to zero, the corresponding C(c), cos 0, and C(c)h(c) converge to C, 1, and g respectively. This fact underlies the computer algorithms to be discussed, which approximate the optimal control by choosing a sequence of vectors c. such that the error E is successively reduced at each step. We note in passing that an alternative approach would be to implement Eq. 3.13 by seeking to maximize the expression shown there. This approach has much in common with the one used here, when the details of the two methods are compared, but it has

116, / c" X 9/ N6 (a) image of \ elN, \ 5^ima ge c c2c —C2\ I'' aUk 2 /C (b) Fig. 7. 1 Constructions to show the continuity of the mapping of aUk onto aS1. the disadvantage that the maximum value of the expression is not known, so that there is no convenient way to tell how close one is to the maximum at any given stage of the process. In contrast, when we seek to minimize E, we of course know the absolute minimum value to be zero, so that we can readily supply an error criterion to the computer to terminate the iteration when E is sufficiently small.

117 However, this advantage of the method used here is more apparent than real, since there is no simple relationship between the size of E and the amount that C(c) deviates from C. If g happened to be in a direction such that in the vicinity of this direction a small change in the direction of h(c) produced a relatively large change in the magnitude of h(c), it could happen that an error of, for example, 2 percent could result in a C(c) which differed from C by 5 percent, or even more, in either direction. To illustrate this point, we note that C(c), as determined from Eq. 7. 5, is the ratio of the magnitude of projection of g in the direction of h(c) to the magnitude of h(c). A small change in the direction of h(c) will not change the projection very much, but if it changes the magnitude of h(c) considerably, the stated result will take place. This effect comes about because of the departure of the set S1 from a perfect hypersphere, since if S1 were a hypersphere there would be no change in the magnitude of h(c) as its direction changed. Unless the shape of as1 is known in the vicinity of g (and it usually is not, in a typical problem) it is not even possible to estimate the "error amplification factor" that can result. There is no simple way to avoid this difficulty (whatever algorithm is used), and the usual approach is to set the allowable error limit rather small, as a "safety factor" against such occurrences. 7. 2 An Algorithm for Proper Systems We wishto approximate, by an iterative process, that value c of c (or, more precisely, any value c of c) for which h(c), as defined by Eq. 7. 1, is colinear with the goal g. Since for all c (c, h(c)) = IVT(tl,t)cldt > 0 T we know that c must have a positive inner product with g, and hence must lie in the hyperhemisphere of auk of vectors having positive inner products with g. A good first guess for c is simply the unit vector colinear with g; i. e., -. This is the initial Igl value used in the algorithm. Having made a choice of c, we compute h(c) from Eq. 7. 1 and then determine cos 0, C(c), and E from Eqs. 7. 2, 7. 5, and 7. 6. If E is sufficiently small (i. e., less than the preset limit), computation stops and the results are printed out. Otherwise,

118 the value of c is changed to a new value c' and the whole process is repeated. The key issue is how to select the new value c'. This decision can be broken down into two steps: a) determine the angular direction in which c should be shifted, and b) decide how large an angular shift to carry out. This algorithm shifts c in the direction which tends to reduce E the fastest, as indicated by the gradient of E evaluated at h(c). Where this gradient is zero (such as at a "corner" of S1, where a change in the normal vector c may result in no change in h(c), because of the nonuniqueness of the normal at such points), the angular shift in c is carried out in that direction which, if carried out on h(c), would rotate h(c) into g. (This, by the way, is the direction which would be called for if the process implied by Eq. 3. 13 were implemented using a gradient technique.) As to the amount of the shift, there is no simple best answer. Too small a shift may result in an excessive number of iterations being required, while too large a shift may cause oscillatory behavior and nonconvergence of the algorithm. The approach used here is as follows: The initial angular shift is chosen as KO, where K is a constant set initially by the user of the algorithm and 0 is defined from Eq. 7. 2 as 0 = cos1 (cos 0); 0 < 0 < X The h(c') corresponding to this new c = c' is computed, and from it the new value of E. If the new value of E is less than the previous one, then computation proceeds as indicated above. If it is not, then this h(c') and c' are discarded and a c' with half of the rotation used previously is used instead. Also, the value K is cut in half, for use in succeeding iterations. If the resulting error is less than the original error, then the next value of c is chosen in the negative gradient direction (or, for systems with "corners", in the direction required to shift h(c) into g), as indicated above. Otherwise, the angular shift and the value of K are cut in half again, and so on. Eventually, of course, the process of successively halving the angular shift must end up with a shift small enough that the gradient dominates the higher order curvatures, so that the process of successive halving eventually terminates and normal computation resumes.

119 In this algorithm there is no guarantee that there will be convergence to within an acceptable error in any given number of steps, but the results on all problems tried thus far have been good, convergence to within an error E of 0. 1 percent having been obtained in 10 iterations or less (often much less). A simplified diagram of this algorithm is shown in Fig. 7. 2, and a complete listing in the MAD language is given on pages 120-123. We offer a few notes of explanation in connection with this algorithm: It is possible to derive analytic expressions for grad E, but these expressions involve improper integrals (unbounded integrands) if the set S1 has "corners". Therefore, the approach used here is to approximate grad E by E - E 100 where E is the error corresponding to the unit vector c = [cl,.., ci 1' ci c +'...ck]T and E, i= l,..,k, is the error corresponding to the unit vector c i[c,..., ci. c.+. 01, ci + l'.' ck] T, normalized to unit length. External functions are used to compute the matrices X(t1,t), B(t), and G(t). This allows the user great flexibility in the use of the program. Depending on the problem, these external functions may be used to insert constant matrices, compute the matrices from given functions, or, in the case of X(tl, t), generate them by solving a system of differential equations. Some of the subroutines that have been used store all the values computed during the first entry to the subroutine, so that on succeeding iterations no computations are needed. In this algorithm, the matrix D is required to be of the form 1 0 0...0 0... 0 0 1 0...0 0... 0 krows 0 0 1... 0... 0 0 0 0... 0 0... k columns n- k columns

PROGRAM FOR PROPER SYSTEMS $ COMPILE MAD, EXECUTE, DUMP, PUNCH OBJECT, PRINT OBJECT START READ DATA PRINT FORMAT HEADA VECTOR VALUES HEADA = $ 1 44H1PROBLEM. CHOOSE THE INPUTS U(1)...U(R) AS F, 2 44HUNCTIONS OF TIME SO AS TO FORCE THE GIVEN SY, 3 44HSTEM FROM ITS INITIAL STATE X(1)...X(N) = / 4 44H ALPHA(1)...ALPHA(N) AT INITIAL TIME TZERO T, 5 44HO THE DESIRED GOAL X(1)...X(K) = BETA(1)...B, 6 44HETA(K) AT TIME T1 AND SO THAT THE P-NORM OF / 7 44H THE AMPLITUDE OF THE VECTOR GU IS MINIMUM. 8 44HG IS ANY INVERTIBLE MATRIX FUNCTION OF TIME./*$ PRINT FORMAT HEADB VECTOR VALUES HEADB = $ 1 44HOINPUT DATA REQUIRED. SYSTEM ORDER N, NUMBER, 2 44H OF STATE VARIABLES TO BE CONTROLLED K, NUMB, 3 44HER OF CONTROL INPUTS R, NUMBER OF INTEGRA- / 4 44H TION INTERVALS S, INITIAL TIME TZERO, FINAL, 5 44H TIME T1, ERROR CRITERION ERB, ITERATION GAI, 6 44HN CONSTANT CAPK, MAXIMUM NUMBER OF ITER- / 7 44H ATIONS ITR. PRINTU (0 RESULTS IN NO PRINTOU, 8 44HT OF U. J RESULTS IN U(1)...U(R) BEING PRINT, 9 44HED OUT EVERY J-TH INTEGRATION INTERVAL.), /*$ PRINT FORMAT HEADC VECTOR VALUES HEADC = $ 1 44H INITIAL STATE ALPHA(1)...ALPHA(N), GOAL BET, 2 44HA(1)...BETA(K), AND NORM EXPONENT Pe P.L-, 3 44H2 IMPLIES P = INFINITY. EXTERNAL FUNCTIONS / 4 44H FND., CON., AND WGT. MUST ALSO BE PROVIDED, 5 44H FOR COMPUTATION OF THE FUNDAMENTAL* CONTROL, 6 44H, AND WEIGHTING MATRICES. /*$ PRINT FORMAT HEADD VECTOR VALUES HEADD = $ 1 44HOINTERPRETATION OF OUTPUT DATA. A, BEE, G, A, 2 44HND GINV ARE THE SYSTEM, CONTROL, WEIGHTING,, 3 44HAND INVERSE WEIGHTING MATRICES, RESPECTIVELY/ 4 44H IF U IS PRINTED OUT, THE COLUMNS BELOW INDI, 5 44HCATE THE INTEGRATION INDEX, THE CORRESPONDIN, 6 44HG TIME, AND THE ELEMENTS OF U/CAPC, RESP. / 7 44H CAPC = PEAK AMPLITUDE. IC = NO. OF ITERATIO, 8 44HNS. FLAG = 1, CONVERGENCE. FLAG = 2, TOO MAN, 9 44HY ITERATIONS. FLAG = 3, BOTH. /*$ WHENEVER P.GE. 2.0 PWR =-(P - 2.)/(2.*P-2.) OTHERWISE PWR =-.5 END OF CONDITIONAL NN=N+N KK=K*K KN = K*N KR=K*R KRS = KR*S DT=(T1-TZERO)/S WO=DT*(2.+25.*DT)/48. W1=DT*(70.-14.*DT)/48. W2=DT*(48.-l1.*DT)/48. INDEX=PRINTU IC = 0 FLAG=0 THROUGH BEN, FOR Q=O01,Q.G.S T=T1-Q*DT EXECUTE FND.(T1,T,DT,PHIA,N,Q) EXECUTE CON.(T,BEE,N,R,~QBN) EXECUTE WGT.(T,G,GINV,R,IC,Q,GN,SFLAGPRINTU) WHENEVER FLAG.G. 0 TRANSFER TO BEN WHENEVER BN.G. 0 THROUGH DOLLY, FOR I=l,1,I.G.K M = I*N - N THROUGH DOLLY, FOR J=1l1,J.G.R Y = KRQ + M + J PSI(Y) = 0. THROUGH DOLLY, FOR L = 1, 1, L.G. N DOLLY PSI(Y) = PSI(Y) + PHI(M+L)*BEE(L*R-R+J) OTHERWISE THROUGH EVE, FOR I = 1, 1, I.G. K M = I*N - N Y = KRQ + I*R - R THROUGH EVE, FOR J=1,1,J.G.R EVE PSI(Y+J) = PHI(M+J) END OF CONDITIONAL WHENEVER GN.G. 0 THROUGH ADA, FOR I=1, 1, I.G. K Y = KRQ + I*R - R THROUGH FRAN, FOR J=l1 1 J.G. R STOR(J)=0. THROUGH FRAN, FOR L=I, 1 L.G. R FRAN STOR(J) = STOR(J) + PSI(Y+L)*GINV(L*R-R+J) THROUGH ADA, FOR J=l1 1, J.G. R ADA PSI(Y+J) = STOR(J) END OF CONDITIONAL BEN CONTINUE WHENEVER FLAG.G. O, TRANSFER TO START ALPHA(O) = 0. BETA(0)=O. GAMMA(0)=0. THROUGH ALEX, FOR I=1,1,I.G.N ALEX ALPHA(O) = ALPHA(O) + ALPHA(I)*ALPHA(I) THROUGH GREG, FOR I=1,1*I.G.K GAMMA(I)=BETA(I) M=I*N-N THROUGH HARRY, FOR J = 1, 19 J.GeN HARRY GAMMA(I)=GAMMA(I)-PHI(M+J)*ALPHA(J) BETA(0)=BETA(O)+BETA(I)*BETA(I) GREG GAMMA(O)=GAMMA(O)+GAMMA(I)*GAMMA(I) ALPHA(O) = SORT.(ALPHA(O)) BETA(O) = SQRT.(BETA(O)) GAMMA(O) = SQRT.(3AMMA(0)) 0

PRINT RESULTS ALPHA(O),BETA(O) GAMMA(O),ALPHA(1)*.. 1ALPHA(N) BETA(1).**BETA(K) GAMMA(1)***GAMMA(K) WHENEVER (GAMMA(O).L..0001*BETA(O)).OR.(GAMMA(O).L. 1.0001*ALPHA(O)) PRINT COMMENT $ DEGENERATE PROBLEMS TRANSFER TO START END OF CONDITIONAL THROUGH CLARA, FOR I=1, 1l I *G. K CLARA C(I) = GAMMA(I) REPEAT EXECUTE ZERO.(H(O)*..H(K) PARTL(1)**.PARTL(KK), 1STOR(1)*.*STOR(K)) THROUGH WALLY, FOR I = 1, 1, I *.G K WALLY C(O)=C(O)+C(I)*C(I) C(O)=SQRT.(C(O)) THROUGH LARRY, FOR I=1lI.*G.K LARRY C(I)=C(I)/C(O) THROUGH AL, FOR Q=O, 1, Q.G. S W=DT WHENEVER (Q.E.S)*ORo(Q.E.O), W=WO WHENEVER (Q.E.S-1)*OR.(Q.E.1)t W=W1 WHENEVER (Q.E.S-2).OR.(Qo.E2), W=W2 T = Tl - Q*DT KRQ = KR*Q U(O) = 0. THROUGH BOB, FOR I = 1, l, I *Go R U(I) = 0O M a KRQ + I - R THROUGH CAL, FOR J = l, 1l J.G. K CAL U(I) = U(I) + C(J)*PSI(M+J*R) BOB U(O) = U(O) + U(I)*U(I) THROUGH DAN, FOR I=1 1 I *G. K M = I*R - R Y= KRQ + M THETA(I) = 0. THROUGH ED, FOR J=l1 1, J *G. R USTOR(M+J) = U(J) + 0*1*PSI(Y+J) ED THETA(I) = THETA(I) + USTOR(M+J)*USTOR(M+J) THETA(I) = THETA(I).P.PWR THROUGH DAN, FOR J=l, 1, J *G. R DAN USTOR(M+J) = THETA(I)*USTOR(M+J) U(O) = U(O).P.PWR THROUGH FRANK, FOR I=1, 1, I *.G R FRANK U(I) = U(O)*U(I) WHENEVER FLAG.GOO.AND.PRINTU.G.O WHENEVER INDEX.E. PRINTU INDEX=1 WHENEVER GN G.E. 1 EXECUTE WGT (TG,GINV,RICQGNSFLAGPRINTU) THROUGH CARL, FOR I=1,1,I.G.R STOR(I)=0. M=I*R-R THROUGH CARL, FOR J=1,l1J.G.R CARL STOR(I)=STOR(I)+GINV(M+J)*U(J) THROUGH DOUG, FOR I=1l,1IIG.R DOUG U(I)=STOR(I) END OF CONDITIONAL PRINT FORMAT OUTPUTQTU(1)o.)U(R) OTHERWISE INDEX=INDEX+1 END OF CONDITIONAL END OF CONDITIONAL H(O) = 0O THROUGH IRA, FOR I=1, 1, I.G. K M = KRQ + I*R -R THETA(I) = 0O THROUGH IRA, FOR J=l 1 J *G. R IRA THETA(I) = THETA(I) + PSI(M+J)*U(J) THROUGH JIM, FOR I=1 1, I *Go K H(I) = H(I) + W*THETA(I) JIM H(O) = H(O) + H(I)*H(I) H(O) = SQRT.(H(O)) THROUGH JACK, FOR I=1, l* I *G. K Y = I*K - K Z = I*R - R THROUGH KEN, FOR J=1l 1, J *Go K M = KRQ + J*R - R THETA(J) = 0. THROUGH KEN, FOR L19, 1, L *G. R KEN THETA(J) = THETA(J) + PSI(M+L)*USTOR(Z+L) THROUGH JACK, FOR J=l, 1l J *G. K JACK PARTL(Y+J) = PARTL(Y+J) + W*THETA(J) THROUGH AL, FOR I=l 1, I.G. K M = I*K - K THROUGH LEN, FOR J=l, 1l J *G. K Y = M+J LEN STOR(I) = STOR(I) + PARTL(Y)*PARTL(Y) AL STOR(I) = SQRT.(STOR(I)) EXECUTE ZERO.(GAMH,THETA(1)...THETA(K)) THROUGH MAX, FOR I=1, 1 I.G. K M = I*R - R THROUGH MAX, FOR J=1, 1 J *Go K GAMH = GAMH + GAMMA(I)*H(I) CAPC = GAMH/(H(O)*H(O)) HCOS = GAMH/(GAMMA(O)*H(0)) ERR = SQRT.(1. - HCOS*HCOS) MAX THETA(I) = THETA(I) + GAMMA(J)*PARTL(M+J) THROUGH OTTO, FOR I=1* 1, I *G. K COS = THETA(I)/(GAMMA(O)*STOR(I)) OTTO GRAD(I) = 100.*(SQRT.(1. - COS*COS) - ERR) GRAD(O) = 0. THROUGH PAUL, FOR I=1, 1, I *G. K PAUL GRAD(O) = GRAD(O) + GRAD(I)*GRAD(I) IC = IC + 1 WHENEVER FLAGoG.O PRINT RESULTS CAPC, ERRIC, FLAG. CAPK PRINT FORMAT GAMM,GAMMA(1)..oGAMMA(K) PRINT FORMAT FINXH(1)*e.H(K) PRINT FORMAT FINCC(1)*.lC(K)

TRANSFER TO START END OF CONDITIONAL WHENEVER ERR.LE.ERB.FLAG=1 WHENEVER IC.G. ITR* FLAG = FLAG + 2 WHENEVER IC = 1 THROUGH BETSY, FOR I=l, 1 I.G. K BETSY CP(I) = C(I) EP = ERR TRANSFER TO JUMP END OF CONDITIONAL WHENEVER ERR *G. EP THROUGH CHRIS. FOR I=1, 1, I.G. K CHRIS C(II = C(II + CP(I) CAPK = 0.5* CAPK TRANSFER TO REPEAT END OF CONDITIONAL EP = ERR THROUGH DOTTIE, FOR I=1 1, IG..K DOTTIE CP(I) = C(I) JUMP WHENEVER GRAD(O) *LE..001 CG = 0. CH = 0. THROUGH EDIE, FOR I1l, 1l I.G.K CG = CG + GAMMA(I)*C(I) EDIE CH = CH + GAMMA(I)*H(I) HCON = CG/CH D(O) = 0. THROUGH FERN, FOR Il1, 1l I.G.K D(I) = GAMMA(I) - HCON*H(II FERN D(O) = D(0) + D(I)*D(I) HCON = CAPK*ERR/(HCOS*SQRT.(D( 0)) THROUGH GAIL, FOR I=1, 1* I.G.K GAIL C(I) = C(I) + HCON*D(I) OTHERWISE HCON = CAPK*ERR/(HCOS*SQRT.(GRAD(O))) THROUGH JOAN, FOR I=1~1I *G. K JOAN C(I) = C(I) - HCON*GRAD(I) END OF CONDITIONAL TRANSFER TO REPEAT DIMENSION ALPHA(10),BETA(10)tGAMMA(10)OPSI(10100)I 1H(10)OPHI(100)tTHETA(10)tBEE(100i)U(10),PARTL(100), 2GRAD(10)tGINV(100 )STOR( 10 ),G(100),A(10),C(10) 4CP(10)oUSTOR(100), D(10) EQUIVALENCE (PHI,PARTL), (BEE.USTOR) INTEGER KNR,IIJLMQ YZ,FLAGITRBNKRS, NN, 1PRINTUINDEX,IC,KKKNGNKRQKRS VECTOR VALUES GAMM=$7HOGAMMA,10E12.4/(S710E12.4)*S VECTOR VALUES FINX=$7HOH l10E12.4/(S7l10E1224)*S VECTOR VALUES FINC=$7HOC,10E12.4/(S7t10E12.41*$ VECTOR VALUES OUTPUT=$1H,I6,10E12.4/(S19,9E12.4)*S END OF PROGRAM TYPICAL DATA SET s DATA N=10,K=4,R=3,S=100,TZERO=O0.T=l.,ERB=.001,CAPK=2., ITR=10,G(1)=1.,0. t0.,0.,1.-1.,0. 1.,2. BEE(1)0*,90*0..* 0.,0.,0.1.,0. t.0.,0.,0.,O.,O,0,0.,l.,1.t0.,0eOe, 0.,0.,0O9,0.0.,1.,1.,1.,1. P=l, PRINTU=10,ALPHA(1).0., 0.,0.0.,.00,g1l.~1,l1.l. P=l, PRINTU=l10ALPHA(1)=0e,2~~ 2.90.,1,1,0., 0.,4.C 0.2O.25..BETA(1)=10.,10.,20.,0O* TYPICAL EXTERNAL FUNCTION FOR COMPUTING PHI $ COMPILE MAD, PUNCH OBJECT, PRINT OBJECT EXTERNAL FUNCTION (T1XTXDTX,PHIXAXNX.QX) ENTRY TO FNDo INTEGER IXJXMX,LXYXZXQXNNX DIMENSION EXPA(100)STORX(10) NNX = NX*NX WHENEVER QX.E. 0 PRINT COMMENT SOPHI IS COMPUTED FROM THE DIFFERENTIAL 1 EQUATION USING FND41001 AND $ PRINT RESULTS AX(1)o..AX(NNX) EXECUTE ZERO.(EXPA(1)...EXPA(NNX) STORX(1)*e. 1STORX(NX),PHIX(1)...PHIX(NNX)) THROUGH FIDO, FOR IX = It NX+1I IX *.G NNX PHIX(IX) = 1. FIDO EXPA(IX) = 1. THROUGH SPOT, FOR IX=1l1, IX.G. 12 WX =-DTX/IX THROUGH LASSY. FOR JX1l, 1, JX.G. NX YX = JX*NX - NX THROUGH LAD, FOR MX =1 1i, MX *G. NX ZX = MX*NX - NX THROUGH LAD, FOR LX=1, 1, LX.G. NX LAD STORX(MX)= STORX(MX)+WX*AX(ZX+LX)*PHIX(YX+LX) THROUGH LASSY, FOR MX=1,1, MX.*G NX PHIX(YX+MX)=STORX(MX) LASSY STORX(MX) = 0. THROUGH SPOT, FOR JX = 1l 1, JX.*G NNX SPOT EXPA(JX)= EXPA(JX) + PHIX(JX) EXECUTE ZERO.(PHIX(1)...PHIX(NNX)) THROUGH SNOOPY, FOR IX=1, NX+1. IX.G. NNX SNOOPY PHIX(IX) = 1. OTHERWISE THROUGH PLUTO, FOR IX=1, 1, IX.G. NX YX= IX*NX - NX THROUGH ROVER, FOR JX= 1, 1, JX.G. NX STORX(JX) = 0O THROUGH ROVER, FOR LX=1,1, LX.G. NX ROVER STORX(JX) = STORX(JX)+PHIX(YX+LX)*EXPA(LX*NX-NX+JX) THROUGH PLUTO, FOR JX = 1, 1, JX.G. NX PLUTO PHIX(YX+JX) = STORX(JX) END OF CONDITIONAL FUNCTION RETURN END OF FUNCTION tO3 w3

TYPICAL EXTERNAL FUNCTION FOR COMPUTING G (IN THIS CASE THE CONSTANTS OF G ARE FED IN AS DATA) $ COMPILE MAD, PUNCH OBJECT, PRINT OBJECT EXTERNAL FUNCTION (TZ,GZ,GINVZRZ, ICZQZGNZ SZ 1 FLAZ,PRINTU) ENTRY TO WGT. INTEGER RZ,QZICZ,GNZ,SZRRZ,IZ,JZLZFLAZtPRINTU, 1 MZ,YZ WHENEVER ICZ = 0 WHENEVER QZ = 0 RRZ = RZ * RZ THROUGH TABBYFOR IZ=1, 1, IZ *G. RZ YZ = IZ * RZ - RZ THROUGH TABBY,FOR JZ=1, 1, JZ.G. RZ TABBY GINVZ(YZ+JZ)= (GZ(YZ+JZ) + GZ(JZ*RZ - RZ + IZ))/2. MZ = BORDS.(RZRZGINVZ(1),DZ) PRINT RESULTS GZ(1)...GZ(RRZ) WHENEVER MZ.L. 1 PRINT COMMENT $OG IS NOT INVERTIBLE. $ QZ = SZ + 1 FLAZ = 4 OTHERWISE PRINT COMMENT $OG-INVERSE, COMPUTED USING WGT400009 1 IS THE CONSTANT R X R MATRIX $ PRINT RESULTS GINVZ(1)*o.GINVZ(RRZ) GNZ= 4 END OF CONDITIONAL END OF CONDITIONAL END OF CONDITIONAL FUNCTION RETURN END OF FUNCTION TYPICAL EXTERNAL FUNCTION FOR COMPUTING B (IN THIS CASE THE CONSTANTS OF B ARE FED IN AS DATA) $ COMPILE MAD, PUNCH OBJECT, PRINT OBJECT EXTERNAL FUNCTION (TYBEEYNYRYQY,BNY) ENTRY TO CON. INTEGER NY,RYQYBNYNRY WHENEVER QY.E. 0 BNY = 3 NRY = NY*RY PRINT COMMENT $OBEE IS THE CONSTANT N X R MATRIX $ PRINT RESULTS BEEY(1).e.BEEY(NRY) END OF CONDITIONAL FUNCTION RETURN END OF FUNCTION

124 Internal subroutine to compute g and I gl. Computes and stores all values of V(t1,t) in the process, calling on external functions for the computation of X(t1,t), B(t), and G (t). Includes tests for singular G(t) and for "degenerate" problems; i. e., problems in which I gl << I or I gl < ia Set c = g PRINT C,EIC= TRFLANSF g Ch, S TRANSFER TO START1 Fig. 7. 2 Simplified diagram of computer program for proper systems.

125 That is, the first k components of the state vector are taken as the output vector, where k may take any value from 1 to n, inclusive. The time interval T is divided into 100 subintervals for purposes of numerical integration, and a modified version of Simpson's rule is used to increase the accuracy of the integrations. Because it required the addition of only one step for the program, the capability to solve the related linear problem (see Chapter V) was written into the program. Any value of p > 2 may be used. Finally, we note that in the IBM 7090, for which the program was written, there is no provision for distinguishing between upper and lower case letters. This has necessitated certain changes in notation. A listing of the notation used throughout this work and the corresponding notation used in the program listing follows: a........ ALPHA g........GAMMA K........ CAPK b........ BETA (g,h)....GAMX u(t)....... U B(t)...... BEE G(t)...... G V(tlt).... PSI c........C G- (t).... GINV x(t).......X C(c)...... CAPC grad E... GRAD X(tt).... PHI E........ ERR h(c)...... H Fig. 7. 3 shows an example of the convergence of this algorithm in a twodimensional problem, with the initial value of K chosen as 2. From this figure we see that the first two iterations, while producing no change in h(c), nonetheless shift c toward the edge of the cone of normals corresponding to the corner. The third iteration not only shifts c out of this cone, but goes well beyond the goal --- so far beyond that the error is actually increased over that associated with h(c 2) Thus, the value of K is cut in half and the normal c 4 results. This gives an h(c) rather close to g in direction. The next iteration (with K now equal to 1) gives an even better result. Further iterations bring h(c) even closer to the direction of g, but these can not readily be shown graphically. In this example, C(c) is within 2. 5 percent of C after the fifth iteration, whereas the initial error in C(c) was 50 percent [both expressed as a percentage of C(c)]. The value of E after the fifth iteration is about 0. 015.

126 c (colinear with g) c1 c47. 3 Another Algorithm has been used with good results on systems for which the matrix V(tl, t) V (tl, t) (c (tt) dt auk asi -4 Fig. 7. 3 Successive iterations of the computational algorithm for a 2-dimensional example. 7. 3 Another Algorithm A second algorithm, a simplified diagram of which is shown in Fig. 7. 4, has been used with good results on systems for which the matrix V(W, t) VT411 t) W(C) f T dt T IV ftjt)cj

127 54 k is bounded in norm5 (uniformly in c) and is invertible for all unit vectors c Ek. Lemma: A sufficient condition that W(c) be bounded in norm and invertible is that there exist an e > 0 such that VT(t1, t) c > e for all unit vectors c e Ek and for almost all t e T. Here V(tl, t) must satisfy the boundedness conditions imposed in Chapters II and III. Proof: That this is a sufficient condition, as claimed, is easily shown by noting that under this condition and the boundedness of V(tl, t), the quadratic form T I=vT(t1,t)dl2 d W(c) d = f-dt T IT(t,t)cI is positive and bounded for all bounded nonzero d, which implies that W(c) is not only invertible but positive definite. Q. E. D. This lemma points directly to the relaxed sufficient condition given in the following: Lemma: The matrix W(c) is bounded in norm and invertible if a) v(tt, t) c > 0 for all unit vectors c e E and for almost all t E T, and b) the integral f dt T VT(tl,t)cl exists for all such c. (Here, as above, V(tl,t) is assumed to satisfy the conditions of Chapters II and III. ) Proof: The proof follows that the preceeding lemma in every respect except that in order to show that W(c) is bounded in norm we must make use of the condition in part b) of this lemma to show that the quadratic form is bounded. Q. E. D. We require boundedness because the computer can not operate with infinitely-large numbers. The norm of a matrix is defined as in Section 3. 2 (footnote).

128 3START EAD DATA I PRINT HEADINGS AND DATA SET FLAG = 0 IC =0 — I~ Internal subroutine to compute g and [ g. Computes and stores all values of V(t, t) in the process, calling on external functions for the computation of X(tl,t), B(t), and G (t). Includes tests for singular G(t) and for "degenerate" problems; i. e., problems in which glj << J|b or gl << |al. Set c = g REPEA~ T Normalize c to unit length Internal subroutine to compute W(c), h(c), C, and Eo Prints out u(t) on final iteration if desired. _ T I IC = IC + 1 I (IC = no. of iterations made) ( Is FLAG > 0? \ Yes r 1 No If E < (allowable error), set FLAG = 1 If IC (allowed no. of iterations), set FLAG = 2 If both, set FLAG = 3:PRINT C,E, IC, ]: FLACG, g, Ch, c Computes W-l(c) using an external function. Stops program by transferring to START if W(c) is not invertible I TRANSFER TO START SET c = W ()g TRANSFER TO REPEAT Fig. 7. 4 Simplified diagram for second algorithm.

129 The basic idea of this algorithm is very simple, and follows the pattern used 55 in contraction mappings: We choose a unit vector c (the initial choice, as before, is -l ), and then compute the next value of c as the unit vector W 1(c)g c' (7. 7) Iw (c)_g Using this value c', we compute W(c') and h(c') = W(c')c'. The error E is evaluated as before, and if it is sufficiently small, the results are printed out and iteration ceases. Otherwise, a new value of c is computed by the same rule, and so on. Although the convergence of this algorithm has not been proved theoretically, the algorithm has in fact converged for the half-dozen or so cases to which it was applied. Indeed, in the second-and third-order problems for which comparisons were made, it has proven to be somewhat faster than the first algorithm. Any advantage it has over the other might be expected to diminish or disappear for higher order systems, however, because of the difficulties involved in the inversion of large matrices. Fig. 7. 5 illustrates the convergence of the algorithm for the simple system used in Example 4, Section 3. 7, with boundary conditions x(O) = 0, x(1) = [1, 1]. For this system |12 1 _ 2 log 1 O 2c2 C 1 Ic I Io g 1e c2i 1+ IC1I Ic I e IC I for c1 0 and Wi 1 W(c) = for c1 = 0. 0 1 It has not been proven to be a contraction mapping, however.

130 y2(1) c (colinear with g) -o Yl(1) -.5. 5 -1. 0 Fig. 7. 5 Convergence of the second algorithm for a 2-dimensional problem. This figure shows that after only one iteration the error has been greatly reduced and after two iterations the error is too small to be shown graphically to the scale used in Fig. 7.5. 7. 4 Some Additional Examples In this section we present some additional examples, solved with the aid of one of the other of the computational algorithms mentioned above. These examples have been chosen primarily to illustrate the various kinds of behavior that can result in minimum peak amplitude problems. Example 14: An undamped oscillatory system. x = x + U System Characterization: x2 = -X + u2

131 Boundary Conditions: x(O) = [] x(t1)= [b2] Curves of C vs t1 for various values of b2 are shown in Fig. 7. 6. We note that each of these curves exhibits some ripple (except the one with b2 = 0), and hence that (except for the case b2 = 0) the inverse relationship is not complete for these cases (see Sections 4. 3 and 4. 5). Example 15: A problem having an absolute minimum cost. x1 = -X1 + 2 + u1 System Characterization: 2 = - 2x2 + 2 Boundary Conditions: x(0) = L4 xl(tl) = 1. 1477 x2(tl) unspecified The curve of C vs. t1 is shown for this example in Fig. 7. 7 from which we see that C takes on its absolute minimum value at a value of t1 near 1. 0. Example 16: A system with a stable focus. System Characterization: x1 = -x1+ 2 x2+u1 X2 = 2iT x1 - x2 + u2 Boundary Conditions: x(O) = L~J xl(t1) = b x2(tl) unspecified The resulting C vs. t1 curve is shown in Fig. 7. 8. For the curve with b1 = 1. 0, note the succession of minima, all having the same value. Example 17: A time-varying system. System Characterization: x = 5 7 x+

b = 12 2 b2= 1 b = 1 2 1 2 2 a, = 1 2 \ \ \ - a2 0 for all curves b2 =0.=i 0-b 0 7T 2r 37r 47T Fig. 7. 6 C -vs- t1 for an undamped oscillatory system, for various final conditions.

I a =0 ar corm n. -f. a 1. a2 =4 4oj-1MI — \ b - 1.1477 b2 unspecified 0. 5 0.1.0 2.0 3.0 Fig. 7. 7 C -vs- t1 for a system with a stable node, for a particular set of boundary conditions.

a = 1.0 a2 =0.0 b2 unspecified 4.0 3.0 =0.9 bl= ~. 9 ~~~rb1.0 3.0.' = I Fig. 7. 8 C - vs- t for a system with a stable focus, for various final conditions.

135 0.14 Boundary ( L 1 Conditions: t = 1.0 t1 > t0 The first computational algorithm was used solve this problem for a series of values of t1 from 1. 2 to 2. 0. The values of C obtained are plotted vs. t1 in Fig. 7. 9a). Fig. 7. 9b) shows the shape of the control functions u1(t) and u2(t) for the value t1 = 1. 8. (These results are typical of all the various values of t1 investigated.) The number of iterations required for convergence to within an error E of. 001 ranged from 10 for t =1. 2 to 2 for t = 2. 0, with the average number of iterations for the twelve chosen values of t1 being slightly over three and the most common number being two. Execution time on an IBM 7090 was 41. 5 seconds for all twelve problems (i. e., all twelve values of tl). Example 18: A sixth-order time-varying system. This example involves an object moving in three-dimensional Euclidean space subject to velocity damping. The mass of the object decreases linearly with time. Initial position and velocity are specified, and the control is to be chosen so as to force the object to a given destination at a given time while requiring the minimum peak amplitude. As noted in Chapter I, this problem can be identified with that of determining the minimum-thrust steerable rocket engine that will accomplish the given task. 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 System 0 0 0 0 0 1 0 0 0 System 1 Characterization: Kd + M(t) 1 0 0 0 0 0 1 0 M~t)Kd 0 0 0 0 (t) 0 0 1 o 0 1 0 0 O0 where M(t) = M + K (t1 - t) p m -tl

136 Boundary Conditions: x(0) = al a2 a3 a4 a5 a6 b1 Y(tl) = b2 b3 C 4 3t 2t 1.. 1.0 a=[1 b = -14 t =1.0 O ti 1. 5 2.0 Fig. 7.9(a) u (t), u2(t) u (t) 1. 5. 1. 0 0. 5 0 -0. 5 -1. 0 -1. 5. C vs. tl for a time-varying system. I - 1 \ / u2(t)! 1 t 1.0.1 1:2 1 3 1.4 1.5 1.6 1.7 1.8 t = L0 t = 1.8 a = b = 45 (b) Fig. 7. 9(b) The optimal control for a particular choice of tl.

137 For this system, it can be shown that 1 0 0 fl(t,s) 0 0 0 1 0 0 fl(t, s) 0 0 0 1 0 0 fl(t, s) X(t, s) = 0 0 0 f2(t,s) 0 0 0 0 0 0 f2(t, s) 0 0 0 0 0 f2(t, s) where dr K 1 Kd Mts)1- [ M(t) Km + 1 f1(t,s) Kd + K m () and Kd f2(t, s) = MS Km A series of problems of this type were run on the computer, using the second algorithm. Numerical values used were M = 1000 Kd = 5x 10 4 t = 5 x 105 p d 1 a = [0, 0, 0, 10000, 500, 1000]T b = [1010 0, 0]T and K ranging from 0. 002 to 0. 02 in 9 steps. (Note that variations in Km correspond to variations in initial mass, since the final mass is fixed at 1000.) The resulting values of C are plotted vs. Km in Fig. 7. 10. The algorithm required two iterations in each case to converge to within an error E of 0. 001, and the total execution time for the 9 problems on an IBM 7090 computer was 44. 1 seconds.

138 C 40 30(0 200" 100. initial 3500 6000 8500 11000 mass.L~~~ ~ ~I I I I 0.005.010.015.020 K m Fig. 7. 10 The variation of the minimum peak amplitude as a function of the constant K (and the initial mass) for Example 18.

CHAPTER VIII SUMMARY AND CONCLUSIONS 8. 1 Introduction Certain optimal control problems, designated here as minimum peak amplitude control problems, are studied in detail in this work. Stated briefly, these problems involve the determination of a set of control variables (i. e., the control vector) as a function of time so as to accomplish a specified goal with the minimum cost, where the cost is defined as the essential supremum over the given time interval of the Euclidean length of the control vector, considered as a vector function of time in Euclidean space. The significant feature of these problems is that this cost functional can not be expressed as an integral over time of some function of the control and state variables, so that classical variational techniques are not applicable. This feature is responsible for both the interest and the difficulties held by this class of problems. 8. 2 Conclusions After a brief statement of the problem and an identification of problems of this type with certain system design problems in electronics and rocketry, Chapter I provides a discussion of the history and present status of optimal control theory. Four major approaches to optimization problems are discussed in some detail: 1) Classical calculus of variations and its modern extensions, particularly Pontryagin's maximum principle: This approach offers rigorous proofs of optimality, but is not very well adapted to numerical computations (because of the two-point boundary value problems involved). These techniques apply equally well to linear and nonlinear problems, but in practice the nonlinear problems are, of course, much more difficult to solve, in general. 139

140 2) Dynamic programming: This approach does not have the mathematical rigor of the variational methods, but it does not involve two-point boundary value problems and it can be used to obtain answers (without assurance of optimality, unfortunately) for a larger class of problems than any of the other techniques discussed. This approach also applies to both linear and nonlinear problems. 3) Functional analysis: The considerable body of theory available in functional analysis has been applied to certain linear optimal control problems, often resulting in proofs, derivations, and solutions which are neat and concise, as compared to those involved in the other techniques discussed. Of particular interest are the existence theorems that this approach provides. Unfortunately, these results can not be extended to nonlinear problems, in general. 4) Direct methods: These methods involve the approximation of the optimal control by an iterative process operating directly on the cost in some way (rather than the solution of a set of auxiliary equations). Drawbacks include the need for proving that the iterative process converges to the optimum. Recent efforts to render the various optimization techniques of more engineering significance are also discussed briefly; to name a few of these: improved computational techniques, optimal control laws, vector-valued cost functions, the inverse problem, and multilevel control. Chapter I concludes with a discussion of the various approaches already discussed as they apply (or fail to apply) to minimum peak amplitude control problems. The applicability of the functional analysis approach is noted. The problem to be considered is formulated in detail in Chapter II, and the various limitations and assumptions are stated and discussed. Various generalizations of the problem are also noted and discussed.

141 The main results of this work are contained in Chapter III. These are: 1) An existence theorem for optimal controls; 2) A theorem given conditions on the form of the optimal control; 3) A uniqueness theorem applicable to a restricted class of systems (here called proper systems); 4) Extensive discussions concerning the relationship between the shape of the reachable set and the form of the optimal controls; 5) Various generalizations of the problem, 6) A new approach to the selection of cost functionals for optimal control problems, called the "hierarchical" approach. Theorems and ideas from modern analysis and topology are used in the proofs of most of the mathematical results of this chapter. The relationship between the original minimum peak amplitude problem and a certain time-optimal problem, here called the inverse problem, is investigated in Chapter IV. A close connection between the two problems is established; namely that under certain conditions (discussed in considerable detail) the control which is optimal for one is optimal for the other. However, it is shown that not all minimum peak amplitude problems have inverses. In Chapter V another kind of problem, here called the related linear problem, is introduced and discussed. The basic result here is that the optimal control problem involving a limiting form of a certain integral-type cost functional has the same optimal cost as the limiting cost of a sequence of problems using the stated integral-type cost functional; i. e., that the processes of solution of an optimization problem and passing to the limit on the cost functional can be interchanged in this case. The approach of Chapter V is applied to certain nonlinear problems in Chapter VI. Steps analogous to those used in solving the related linear problem and passing to the limit can be used in nonlinear problems. Rigorous proofs of convergence are not attempted, but examples are presented to show the convergence in particular cases. Chapter VII presents computational algorithms applicable to proper linear systems and to a special class of systems for which a certain matrix is invertible. The convergence of these algorithms is discussed both from a theoretical standpoint and by means of

142 examples. The role of the shape of the reachable set in the convergence of the algorithms is especially emphasized. Finally, additional examples are presented to illustrate the variety of behavior that can be obtained even in simple problems and to demonstrate the capabilities and speed of the computational algorithms. 8. 3 Suggestions for Future Research Three areas touched upon in this work appear to this writer to merit further investigation. The first of these involves the extension of these results to nonlinear systems in a rigorous manner. The most promising area for initial work involves nonlinear systems with linear control, since in such problems the limiting process indicated in Chapter VI can be used to obtain an explicit and simple expression for the "candidate for optimality. " The convergence (or lack of convergence) of this limiting process for more general types of nonlinear systems might also be a fruitful area for future work. A second area of possible research is that of nonlinear mappings between linear spaces. In the problem considered here the original mapping from control space to "final condition" space was linear, but the problem was later reduced to one involving a nonlinear mapping between two finite dimensional spaces. Such mappings might exhibit very interesting properties under further study. Finally, rigorous proofs of the convergence of the computational algorithms would be desirable. In particular, it would be interesting to know under what conditions (if any) the mapping defined by Eq. 7.7 is indeed a contraction mapping (as it appears to be on the basis of the various numerical examples to which this algorithm was applied).

143 APPENDIX A OPTIMAL CONTROL PROBLEMS AS PROBLEMS IN THE CALCULUS OF VARIATIONS This appendix demonstrates the equivalence of a certain class of optimal control problems to the Lagrange problem of the calculus of variations, gives a derivation of a simplified version of the Lagrange multiplier rule, states the more general form of this rule applicable to problems with separated but otherwise general boundary conditions, and discusses extensions of these results to more general problems. A. 1 Types of Solutions Before proceeding with a discussion of variational methods, we shall define what is meant by the term "solution, " as applied to optimal control problems: a) A transformation of the original problem (which involves certain differential equations, boundary values, and constraint equations, plus a statement that a certain functional is to be minimized) into a well-specified boundary value problem solvable (in principle, at least) by conventional techniques is sometimes called a solution to the optimal control problem. b) A complete analytical or numerical solution of the problem, resulting in a specification of the controlling variables as functions of time, is also called a solution. c) A determination of the control law, — i. e., an expression which yields the optimal control as a function of the state of the system (rather than as a function of time) —is referred to by some (Pontryagin, Ref. 13) as a solution to the synthesis problem of optimal control. Variational methods typically yield a solution in the sense of definition a). Solutions in sense b) and/or sense c), if desired, are then obtained by other techniques. In practice, of course, the two-point boundary value problems which result from a) may be extremely difficult to solve.

144 A. 2 Problem Formulations A. 2.1 The Problems of Bolza, Lagrange, and Mayer. An important problem in the calculus of variations is the problem of Bolza with separated end-conditions, which is the problem (as formulated by Bliss, Ref. 8, p. 193) of finding, in the class of arcs y(t) = [yi(t),... Ym(t )]T t e T lying in (m+1)-dimensional Euclidean space56 and satisfying differential equations and end conditions of the form j(y,y,t) =0 O j = 1,...,n; n < m hk((to),to) = 0 k = 1,...,p; p < m+1 hk(y(tl),tl) = 0 k = p+l,...,p+q; q < m+1 one which minimizes a sum of the form J = F0(Y(t0),t0) - Fl(y(tl),tl) + f fo(y,',t)dt where T is the closed interval [t0,tl] of the time axis. The point (y,y,t) is required by Bliss to lie in an open region R in (2m+1)-dimensional Euclidean space, the functions fg and gj are assumed to have continuous partial derivatives of at least third-order in this region, and restrictions are placed on gj and hk in order to insure independence. Not all the variables need be present in every case. This problem has equivalent formulations as a problem of Mayer (in which f0 is missing) or as a problem of Lagrange (in which F0 and F1 are missing), (Bliss, Ref. 8, p. 189, and Meile, Ref. 62, p. 109). For purposes of this discussion, the Lagrange formulation is more convenient. Hence, without losing any generality, we will henceforth restrict attention to this formulation of the problem. 56 The values of the variables yl(t),..., ym(t), and t are represented as distances along the coordinate axes of the Euclidean space. This representation of a vector function of time as a moving point in a Euclidean space is used throughout this section.

145 A. 2. 2 A Class of Optimal Control Problem as Problems of Lagrange. Optimal control problems are usually stated in a form different from that just given. For example, a certain class of optimal control problems is typically formulated as follows: Assume a system satisfying the independent equations x. = f(x,ut) = fi(x,...,xn U,...U,t); i = 1,...n; t T where (x,_x,t) is constrained to lie in the open set R of (2n+l)-dimensional Euclidean space and where fi (and also the f0 defined below) are continuous and have continuous partial derivatives of at least third order in R. In the class of bounded measurable control functions u(t), x t e T lying in the open set R of r-dimensional Euclidean space Er(r > 1) which cause x(t) to satisfy the independent conditions satisfy the independent conditions hk(x(to),to) = 0 hk(X(tl),tl) = 0 p < n+1 k = p+l,...,p+q; q < n+1 find one which minimizes J = J f o(xut)dt T We will now recast this problem into a form identical to that given above for the problem of Lagrange: Define Yi = x. =i u i = 1,...,n Yn+ = u.1 m =n+r m = n+r y(t) = [y Y(t),...,Yn+l(t),...Ym(t)] gj(y,It) = yj- fj(Yl*.. Yn' Yn+l'y'..,mt) hk(Y(to),to)= hk(x(to),t) k = 1,...,p hk(y(tl),tl) = hk(x(tl),tl) k = p+l,...,p+q fo(y,,t) = fo(Y1,'...**, Yn+,1*... Ym t) R = R xR xE X u

146 Then this optimal control problem can be rewritten in a form identical to that of the abovestated calculus of variations problem, as follows: Find, in the class of arcs y(t), t e T, lying in (m+1)-dimensional Euclidean space and satisfying differential equations and end conditions of the form 9j~,,t) = 0 j = 1,...,n; n < m hk((to),to) = 0 k = l,...,p; p < n+1 hk((tl),tl) = 0 k = p+l,...,p+q; q < n+1 one which minimizes J = J fo(y, t)dt T The point (Yy,,t) must lie in the open set R of (2m+1)-dimensional Euclidean space. (Note that the inclusion of the whole space Er in the definition of R implies that the point is restricted in at most only 2n+r+l of its coordinates, the remaining r being completely free.) The functions f0 and gj are continuous and have continuous partial derivatives of at least third-order in R. (Note also that there is no longer any explicit distinction between state and control variables. ) With this equivalence of the two problems in mind, we can proceed to apply the results of the classical calculus of variations to optimal control problems. Furthermore, results from the calculus of variations derived for conditions different from those imposed by Bliss (Ref. 8) in his treatment of the Bolza problem (e. g., the requirement that the set R be open) will apply to optimal control problems with correspondingly different conditions. A. 3 The Lagrange Multiplier Rule The problems posed in the previous section can be solved with the help of the Lagrange multiplier rule. A simplified version of this rule will now be derived. A. 3. 1 Derivation of the Lagrange Multiplier Rule. Problem: Given a system described by x = f(x,_u, t), where the equations xi = fi(x,u, t), i= 1,..., n are consistent and independent. From the set of bounded measurable control functions u(t) lying at each moment of time t e T in the open set R of r-dimensional

147 Euclidean space which cause the system to transfer from the given initial state x(to) = a at initial time t to the desired goal state x(tl) = b at final time tI find the one which minimizes J = S fo(x, u,t)dt T The functions f and f. are continuous in all their arguments and have continuous partial O 1 derivatives of at least third order. Derivation (after Weinstock, Ref. 63, pp. 57-60): Assume that the above problem has a solution. Denote this solution by x(t), u(t). Imbed this problem in the more general problem of minimizing J() = f fo(X, U, t) dt T subject to the constraints gi(X, X, U, t) = 0 X.(t) = a. I0 i=l,..., n Xi(t1) = b. 1l where Xi = Xi(t) = i(t) + j = U(t) = u(t) + gi(X, X, U, t) = Xi - e Yi(t) e vj(t) f.(X, U, t) i=l,..., n j= 1,..., r and where the bounded measurable functions Yl(t),..., y (t)(t)l(t),..., vr(t) are arbitrary except to the extent implied by the above constraints. Note that, because x(t) and X(t) satisfy the same boundary conditions, Yi(to) = Yi(tl) = O, i= 1,..., n.

148 J(e) will be a minimum only if d() = 0. But, by definition of x(t) and(t), this occurs when e = 0. Thus, a necessary condition that J(e) be minimum is that dJ(E)1 = J'(0) =. de e =0 Performing the indicated differentiation gives n af r f J'(E) = i ax i V. dt. At e = 0, this becomes n af r af J'(O) =f Y axi i+ a vj] dt (A. 1) Now, differentiating gk(X,, XU, t) = 0 with respect to E, we get n ragk a gk [gr r a g i Lax. Yi J+ aku [LUa vj = 0, k=1,..., n. agk o i kl1 Making use of the facts that = L., we have that ai = k agk afk agk afk af aX = ~ aX and = ax. ax. 8 U. au. Setting c = O, we obtain finally n afk r afk "- ax i i Yk Yk au v = 0, k= 1.., n. (A. 2) i=l l 3=1 j v Since each of the expressions on the left-hand sides of Eqs. A. 2 is equal to zero along any system trajectory (and hence also along the optimal trajectory), adding any multiples of them to the integrand of Eq. A. 1 does not change the value of the integral. Thus, we may multiply the left-hand sides of Eqs. A. 2 by the as-yet-undetermined Lagrange multipliers

149 4/l(t),..., Pn/(t), respectively, and add the results to the integrand of Eq. A. 1, yielding n[[.f n afk I l r af n afk n Z [ V'k ax.i Yi +i O(i i] + au. Z 4/k a U. 3 T - k= k f =l - k=l j n Integration by parts of the term f q/i yi dt and application of the boundary conditions T i= 1 Yi(to) = Yi(tl) = 0 gives n n f S i y dt = -f E y'i dt T i=l T i=l Hence, f r'k axa au. f n f Z n 3fk r 3f n 3fk T -i= La k=l 1 i j=1 j k=1 3 (A. 3) Because of the set of n equations gk(X, X, U, t) = 0, k = 1,..., n relating the n + r functions y1(t),..., yn(t), v1(t),..., vr(t), we cannot regard these n + r functions as being free for arbitrary choice. In fact, there is some subset of n of these functions which are dependent on the choice of the remaining r. We now choose the n arbitrary functions, 1(t),..., n(t) so as to make the coefficients of these n dependent functions in Eq. A. 3 equal to zero. Equation A. 3 then reduces to the integral of a linear combination of the remaining r arbitrary functions. But this integral must be zero for any choice of the arbitrary functions, and therefore the coefficients of these functions must also be zero. Thus, all the coefficients of the variables y1,..., y, v1,..., vr in Eq. A. 3 must be zero. In summary, then, if the problem has solutions, these solutions must satisfy the following equations and conditions: x = f(x,u,t); x(t ) = a; x(t1) = b i = aa [ -Vfo(x u t) + 4'k fk(x'-u t; i= 1,.., n k= 1

150 n ai [-fo(-x, + k k k( t = 0; j = 1..., r auj - k= 1 A. 3. 2 Application of the Lagrange Multiplier Rule. The use of the Lagrange Multiplier rule will be illustrated by its application to an optimal control problem less general than that posed in Section A. 2. 2; namely, one in which the set R is the whole (2n+ 1)x dimensional Euclidean space and in which the end conditions are completely specified, as follows: Xi(t ) = a. x(t ) b i i1,..., n 1 0 1 i "'' where ai, bi, to, and tI are all given constants. The Lagrange multiplier rule then gives as a necessary (but not sufficient) condition that a u(t) and the corresponding x(t) satisfying x = f(x, u); x(t ) = a; x(t1) = b be 0 - optimal is that there exists a continuous nonzero vector function (t) = (/ l(t),..., n (t) )T which satisfies the Euler-Lagrange equations: aH(x, u,_, t) x=.- -aHx - i= 1,..., n (A. 4) 4i = -.~x.. a H(x, u,, t) u =0 j= 1,...,r (A. 5) au. where, for convenience of notation (and in order to bring out an analogy with classical mechanics), H(x, u,, t) has been used to denote the expression n -f (x,, t) + M qi(t) f(x,u,t) i= 1 Here n additional variables 41(t),... P, n(t), n additional differential equations (Eq. A. 4), and r additional finite equations (Eq. A. 5) have been introduced. The result is a problem with 2n differential equations, 2n boundary conditions, r finite equations, and 2n+r dependent variables —a "well-specified" problem to which conventional methods for the solution of systems of differential equations can be applied.

151 Two assumptions are crucial to these results: a) In deriving these equations, we assume the existence of an optimal solution to the problem. Thus, the results apply to the solutions, if they exist, but they do not guarantee the existence of a solution or solutions, thus justifying the above statement that the satisfaction of the Euler-Lagrange equations is a necessary but not a sufficient condition that a given set of functions x(t), u(t), Ij(t) be optimal. b) The control u(t) is required to lie in an open set R in r-dimensional Euclidean space E. J always has an infimum over u's in this set, but not necessarily a minimum (Pontryagin, Ref. 13, p. 239; Mortensen, Ref. 22, p. 5). This is,in fact,the major weakness of the Lagrange multiplier rule and of classical calculus of variations — they apply only when u lies in an open set, but the resulting transformed problem may have no solution. In connection with the necessity of these conditions, note that in their derivation the solution which minimizes J is obtained by setting dJe) = 0, where J(c) is a onedE parameter family of functionals and e is the parameter. But obviously a maximum or a zero-slope inflection point of J(e) would also satisfy the same condition. Thus, the EulerLagrange equations alone do not allow one to distinguish between maxima, minima, and zero-slope inflection points. To overcome this difficulty, we add as an additional necessary condition for a minimum the Legendre-Clebsch condition (Mortensen, Ref. 22, p. 7; Meile, Ref. 62, p. 104): a2 H(_,_u,1, t) The matrix with elements Hau,- must be negative semidefinite. au.au. 1 J A point of interest to the discussion of Pontryagin's maximum principle now becomes apparent: Equation A. 5 indicates that the optimal u corresponds to a stationary point of the function H(x,u,_j,t). The Legendre-Clebsch conditions indicates that this stationary point is a maximum. Both conditions can thus be summarized as follows: u(t) with values in R at each moment of time t e T must be chosen so as to maximize H(x, u,,t) at each moment in time. These results can be generalized to cover problems having the separated but otherwise general boundary conditions of the originally posed Bolza problem by noting that,

152 in order to be optimal, _x(t) and j(t) must satisfy the transversality conditions of the calculus of variations. Using the approach of Bryson, (Refs. 41,42), we say that the transversality conditions are satisfied if i and x are such that the additional conditions aQ / (t = i ( axi(t0) i= 1,...,n aQ it ) = i(tl) = axi(t) are satisfied, where Q is defined by the expression P q Q = ihi(x(t),to) + p+i hp+i(x(t)t) i=1 i=l1 and where 01,..., 0 are multiplicative constants to be determined from the boundary conp+q ditions. Furthermore, if t0 and/or t1 are unspecified, the additional conditions needed for the determination of t0 and/or t1 are given by H(t ) + a - 0 H(t) + a 0 A. 4 Extensions of the Classical Calculus of Variations A. 4. 1 Pontryagin's Maximum Principle. As noted in the previous section, the classical calculus of variations, in its usual formulation, is limited to problems in which the point (x, x, u, t) is constrained to lie in an open set in (2n + r + 1)-dimensional Euclidean space. Problems of engineering interest often require some subset of these coordinates to lie in closed sets. Therefore, generalization of the classical results to problems with closed sets would be of great practical value. Note, however, that various degrees of generalization are possible. As a first step, one might consider the case in which the values of u are constrained to lie in the closed set R in r-space but t, x, and x are unconstrained (i. e., the set R is the cartesian product

153 of the whole (2n + r + 1)-space and the closed set R )o Pontryagin and his associates (Refs.) 10, 13) devised a set of necessary conditions for the solution of this problem. Similar results have also been presented by Berkovitz (Ref. 19) and Larsen (Ref. 21) using the method of Valentine (Ref. 9), as well as by Halkin (Refs. 64, 65), Kalman (Ref. 66), Leitman (Ref. 67), Rozoener (Ref. 16), and others. The best known of these results, those of Pontryagin and his associates, are discussed below: In Section A. 3. 1, a set of necessary conditions for the optimality of x and u was given for the case where R is an open set and R is the whole Euclidean space. These conditions can be summarized as follows: A necessary condition that u(t) and the corresponding x(t), satisfying x = f(x, u,t) x(t0) = a, x(t1) = b, be optimal (i. e., minimize J = f fO(x,u, t)dt) is that there exists a T continuous nonzero vector function j_(t) such that aH 4^= —a = 1,...,n ax. and such that u(t) with values in Ru at each moment of time t e T maximizes n H(x,u,, t) = -f0(xu,t) + 3 4ifi(x,u,t) i=1 57 at each moment of time. This can also be taken as a statement of the maximum principle by letting R be a closed set. Pontryagin et al., proved (Ref. 13) that the maximum principle applies to problems with much less restrictive conditions on f (x,u, t) and u than were 57 The maximum principle, as stated by Pontryagin, is more general than this, in that it is necessary only that u maximize H almost everywhere on the time interval, and in that a constant multiplier, 4/o, is also used for fo(x,u,t). Q/O is required to be nonpositive, and the augmented vector (go0, 41.., ~m) is required to be continuous and nonzero. Since 4 is determined only to within a multiplicative constant, however, it follows that the two statements are identical unless Q0 turns out to be zero. In the calculus of variations, solutions in which Q/O = 0 are called "abnormal" solutions (Bliss, Ref. 8, p. 213). Abnormality is often an indication of the fact that there is only one arc in the given neighborhood which satisfies the boundary conditions and constraints, and hence that the optimal solution does not depend on the "cost" function f0(x,u, t).

154 assumed by Bliss (Ref. 8). Specifically, in Pontryagin's treatment f (x,u,t) must be continuous in x, u, and t, but the only partial derivatives that are required to exist are the first partial derivatives of f with respect to t and the xi's (but not the uj's). Bliss required the existence of partial derivatives of at least third order with respect to all the arguments of f. This weakening of restrictions is the principal theoretical contribution of Pontryagin (according to Mortensen, Ref. 22, p. 4). A. 4. 2 Problems with Bounded State Variables. In the problems considered thus far, the state variables x(t) have been restricted only to the extent that they must satisfy certain differential equations. If we further restrict x by means of inequality constraints of the form Zi(Xl...,Xm) 0 i =,...,s the results described thus far no longer apply. However, it has been shown, (Berkovitz, Refs. 19, 20) that such problems can be put into the form of a classical Lagrange problem using the method of Valentine (Refs. 9, 19, 20, 21): Upon definition of additional functions x (t),...,x (t) satisfying m+1 m+s (Xm+i)2 - Zi(Xl1... Xm) = 0; m+i(t0) = 0; i =,...,s, the problem can now be regarded as an ordinary Lagrange problem in the variables x1(t),... x (t), to which all the classical results apply. Any solution to this augmented problem will m+s obviously satisfy the inequality constraints, as well as the usual necessary conditions for an optimal solution. (Of course, additional variables, additional differential equations, and additional Lagrange multipliers have been introduced, so that the augmented problem may be (and usually is) considerably more difficult to solve, in practice, than the corresponding problem without inequality constraints. ) Gamkrelidze (Refs. 13, 14) proves similar results without using Valentine's method. A further generalization is the case in which x and u are jointly constrained to a closed set in (n+r)-dimensional Euclidean space —a set which can not be decomposed into the cartesian product of a constraint set on x (independent of u) in n-space and a constraint set on u (independent of x) in r-space. Berkovitz's treatment (Ref. 19) covers this

155 situation also. The same approach can be used if x, u, and x are jointly constrained to a closed set in (2n+r)-dimensional Euclidean space.

156 APPENDIX B DYNAMIC PROGRAMMING AS A COMPUTATIONAL TECHNIQUE B. 1 Philosophy and Motivation "Dynamic programming" is the name given to a computational technique devised by Bellman (Refs. 17, 18) for use in the solution of a class of optimization problems. The dynamic programming approach, which differs markedly from the usual variational approach, is motivated by the following considerations: (a) Variational methods invariably yield two-point boundary value problems, which can not be solved analytically except in the simplest cases. Furthermore, even the form of the solution is often not determinable using these techniques. Thus, even though variational methods are analytical, in principle, results can only be obtained by numerical methods in many cases. Even here,variational techniques require the use of iterative methods since two-point boundary value problems generally do not lend themselves to straightforward solution, even on a computing machine. Dynamic programming, on the other hand, is from the outset a numerical method. (b) Dynamic programming, because it bypasses the difficulties inherent in two-point boundary value problems, often results in reduced computation time. (c) Recent extensions of the classical calculus of variations involve less restrictive conditions on the system equations than those required in earlier treatments. However, it is easy to find systems (e. g., systems with multiple modes of operation selected by relays), which do not fit even these relaxed conditions. Dynamic programming, because it makes no use of differential properties of system and cost functions, can handle more general problems than can classical methods. (d) The insight into the behavior of the system in question (as a function of boundary conditions, system parameters, etc.) that is provided by the analytic solution (when available) generally can not be obtained from one or a few numerical solutions for specific sets of boundary conditions. Thus, when numerical solutions must be resorted

157 to because of system complexity or nonlinearity, it is really a family of solutions that is 58 needed for intelligent evaluation of the results. Dynamic programming provides such families. (e) Optimization problems must generally be converted to discrete-time form before they can be solved on digital computers. Thus, even though dynamic programming (as a computational technique) is used primarily in discrete-time problems, it does not necessarily suffer any basic loss of accuracy in comparison to continuous time techniques converted to a form suitable for digital computation. (f) Variational techniques offer means of finding local maxima (and minima). In a complicated problem, considerable computation might be required in order to obtain and compare all such maxima (or minima) and thus select an absolute maximum (or minimum). Dynamic programming, however, always finds the absolute maximum (or minimum) in the specified region. (g) Dynamic programming can be extended to apply to stochastic problems, whereas classical techniques are not readily extended in this direction. The above points, while smacking in some cases of post facto justification of certain unavoidable aspects of the dynamic programming approach (in particular, the discrete-time format and the imbedding, which results in families of solutions), nonetheless summarize a valid point of view which has extensive application. Certain weaknesses inherent in the dynamic programming approach have already been noted (see Section 1o 2. 3). B. 2 Formulation of the Problem Assume that the system to be controlled is characterized by a set of difference equations x(ti + = x(ti) + F(x(ti), u(ti)) i = 0,1,..., N 5For example, in practice the boundary conditions are never known with absolute precision. Thus, from an engineering standpoint,it is important to determine the sensitivity of the optimal solution to variations in the boundary conditions. This sensitivity can be determined if families of solutions with the boundary conditions as parameters are provided.

158 where x(ti + 1), the state of the system at time ti + 1, is given as a function of the previous state x(ti) and the control vector u(ti) applied at the previous time t.. The time points to, tl... tN need not be equally spaced, although in practice they usually are. (If the original system description is in terms of differential equations, then a conversion to difference equation form must be made.) For simplicity, we denote the vectors x(ti) and u(ti) by xi and u, respectively. The problem to be solved is then written as follows: Given an initial state xo, choose an input sequence u 1 u.. N (where each vector u. must lie in the given set U) so that the criterion or cost function N ci Fo(Ui,xi) i=O0 59 is minimum. B. 3 Imbedding Let Min N GN U= i Z ci Fo(Ui Xi), i=0 i= 0,...,N where here and in all that follows each u. is understood to be chosen from the set U. (Such u's are said to be "admissible". ) Since uN affects only the last term in this series, (because, by assumption, the system is physically realizable, and hence can not respond to an input before the input is applied) it is obvious that GN can be rewritten6 as No final conditions or bounds on x have been specified here, in order to avoid undue complexity in this exposition of the method. Problems with such constraints can be solved by dynamic programming with no increase in difficulty, however. Variations of the set U with time also pose no difficulties. 60 6This result is a consequence of Bellman's "principle of optimality" (Refs. 17, 18), which states that an optimal trajectory has the property that, no matter what initial decisions (choices of control vectors) are made, the remaining part of the trajectory must be optimal.

159 Min N- I Min. F (u x)M GN = u1 u Z c F (uix) + [cNFo(uN,xN)] i=0,...,N- = -N Continuing this process, we get N-1 [N-1 Fo(U N- ) + U CN N N)'] a series of "nested" minimizations, each with respect to a single control vector u.. We reason as follows: If we knew the state XN, we could compute the cost for the last stage of the process, simply by trying all values of uN and picking the one which gave the lowest cost. We also thereby determine uN. But xN is not known at this point, since it is a function of the as-yet-undetermined control vectors uO,..,uN Therefore, since we can't anticipate which value of xN actually lies on the optimal trajectory, we will tabulate the optimal control and the associated cost for all xN in our range of interest. 6In practice, a grid of points in the region of interest in state space is established, and the optimal control and cost are computed and tabulated for each of these. Now we consider the cost for the last two stages of the process. Again, we don't know xN' so we consider allxN _'s of interest. For each possible xN _ 1 each possible choice of u N 1 determines a cost for that stage, and it also determines 62 x N Since we have already tabulated the optimal cost for each xN of interest, we can compute the total cost for the last two stages for that choice of uN 1 The choice 61It can be seen that bounds on the state variables x only serve to make the problem easier, since they limit the range of values of x that must be investigated. Any u which causes x to exceed the stated bounds is, of course, rejected, and the optimum choice is made from the remaining admissible u's (if any). If the x N thus determined falls between the chosen grid points, we interpolate to get the needed data. If it falls outside the chosen grid, we extend the grid.

160 involving the lowest cost is then the optimal choice for uN 1 for the given xN _ We again tabulate these results for each xN 1 of interest. Now that the optimal cost and control are known for each x N 1 of interest, we can use these results to compute the optimal cost and control for each x N 2 of interest. And so on, until we have completed the table for each x of interest. -o Entering this table at the given value of x, we can follow through and determine the optimal control at each step of the way, and hence our problem is solved. We can also look at the results for other values of x with equal ease, and hence we have -o in effect obtained a family of optimal trajectories, with x as the parameter. Furthermore, if a certain final state must be reached (i. e., if it is required that x N+ 1 = b, where b is some given vector) then the process is unchanged except that in the final stage, only those values of xN which are transformed (by means of the given difference equations) into the required x N + 1 = b by an allowable u need be considered. If more than one allowable u transforms a given xN into the required xN + 1 then the u with the lowest last-stage cost is entered in the table. This process of solving the optimal control problem in question by solving a whole family of problems (a family which includes the given problem) is known as "imbedding", and is characteristic of the dynamic programming approach. B. 4 Drawbacks and Limitations Dynamic programming has certain drawbacks and limitations in addition to those already noted, which must be taken into account in determining its domain of practical applicability: Care must be taken in converting a continuous-time problem to discrete time form. Too coarse a subdivision of the time axis results in excessive errors, and too fine a subdivision causes computing time to increase unreasonably. The same can be said for the choice of grid points in state space (the states for which costs and optimal control are tabulated) and in control space (the various controls which must be tested at each stage and for each state of interest in order to determine the optimal control and cost for that stage and state).

161 Computing time and/or computer memory requirements very rapidly become excessive as system dimension is increased. Other techniques also suffer from this "curse of dimensionality" (Bellman, Ref. 18, p. 94), although in varying degrees, depending on the type of problem. A difficulty peculiar to dynamic programming is that referred to by Bellman (Ref. 18) as "the menace of the expanding grid": At some stage in the solution of a problem, the optimal control may transform the state to a point beyond the originallychosen grid of points. In this event, a new point (or some new points) must be added to the grid and computations must be made at this point (or points) for some or all of the previously-completed columns of the table. These,in turn-may add new points to the grid, and so on. This "doubling-back" to recompute additional points may add considerably to program complexity, computing time, and memory requirements in a particular problem. There is no way to avoid this difficulty, in general.

162 APPENDIX C THE FUNCTIONAL ANALYSIS APPROACH TO LINEAR TIME OPTIMAL PROBLEMS The two principal methods of solution of optimization problems discussed above, dynamic programming and the calculus of variations, apply to problems with involving restrictions on the instantaneous values of the control inputs u(t), or perhaps no restrictions on u(t) at all. This does not begin to exhaust all the kinds of restrictions on u(t) that are of interest, however. For example, one might wish to place limits on the energy required to accomplish a certain task, or on the area under the curve lu(t) I, to < t < tlo In many cases, such problems can be put into a form suitable for the application of one or more of the methods mentioned above.63 However, the resulting nonlinear problems may be difficult to solve. The functional analysis approach offers an alternative method of solution for certain problems of this type. 63For example, consider the minimum time problem with control energy E required to be less t1 than or equal to some limit L, energy being defined here as E = f (u(t), u(t)) dt. Using t 0 the method of Valentine, we can define an additional state variable xn+i(t) by means of the ) 2 L equations (xn+)2 =- t t- ( u(t), u(t)); xn+l(to) = 0. The right-hand side of the differentl tial equation integrates to L - f (u(t), u(t)) dt = L- E. The left-hand side integrates to t a number either positive or zero. Thusany solution to the augmented system of equations must satisfy the condition E < L, as desired. This is a problem to which classical techniques apply.

163 C. 1 The Linear Time-Optimal Problem Krassovski (Ref. 30) has made use of certain of the results bf Krein (Ref. 68) in functional analysis to solve a class of time-optimal control problems. This work was extended by Kulikowski (Refs. 31, 32, 33, 34), Kirillova (Ref. 35), and Kranc and Sarachik (Refs. 36, 37). In its simplest form, the problem considered is that of forcing a system characterized by the equations x = A(t) x + b(t) u; (here b(t) is an n-vector and u = u(t) = the scalar input or control variable), from its given initial state x(t ) to the origin of the coordinate system in minimum time, subject to the constraint t1 1 u(t) = [ / lu(t) IPdt P < 1; p > 1 o 0 The quantity l u(t)II pis called the p-norm of u(t). For p = 2, this is equivalent to an "energy" constraint or an "average power" constraint. As p approaches infinity, this condition approaches the amplitude- constraint condition I u(t) < 1, which is a problem solvable with the aid of Pontryagin's maximum principle. By means of Holder's inequality, an expression for the optimal control is obtained, 64 as well as conditions under which it exists. Kirillova (Refo 35) showed further that, as p approaches infinity, this result approaches a limit which he proved to be the optimal control for the limiting condition lu(t) I < 1. This limiting result is, of course, identical to that obtained using the maximum principle. C. 2 Solution of the Linear Time-Optimal Problem The problem to be solved here can be stated as follows (see Kranc and Sarachik, Ref. 36): Given a completely controllable65 system characterized by the set of equations 64 A more detailed description of this step is given in the next section. 65 Complete controllability is defined in Appendix D. Also,see Kalman (Ref. 25)o

164 x = A(t) x + B(t) u, where x is the n-dimensional state vector, x is its time derivative, u is the r-dimensional control vector, and A(t) and B(t) are continuous n x n and n x r matrices, respectively. The system is in the given state x(t ) = a at initial time to. Choose the control vector u(t), t < t < t1, from the class of bounded measurable functions satisfying 1 1 r 11u(t) Ilp = [ lutl (t) Idt < L where p is a number greater than one, so that the k-dimensional output vector y(t) = D(t) x(t) satisfies the desired final condition y(t1) = b and so that t1, the final time, is minimum. D(t) is a continuous k x n matrix of rank k. The solution of the system of differential equations can be written as t x(t) = X(t,t) a + f X(t,s) B(s) u(s) ds, (C. 1) t O where X(t, s) is the fundamental matrix (see Coddington and Levinson, Ref. 57) of the unforced system; i. e., t X(t,s) = A(t) X(t,s); X(t,t) = I where I is the n x n identity matrix. In these terms, y(t) is given by t y(t) = D(t) X(t, to) a + f D(t) X(t, s) B(s) u(s) ds. t o Define V(t, s) = D(t) X(t, s) B(s) and g(t) = b - D(t) X(t, t ) a. Then the satisfaction of the final condition y(t1) = b, is equivalent to the satisfaction of ti tt g(t1) = f V(t, s) u(s) ds = f v1 (tls)... vr (tls)] u(s) ds (C. 2) t t 0 0 where v. (t1, s) is the ith column of V(t, s).

165 An intermediate step in the solution of this minimum-time constrained-norm problem is the solution of a class of fixed time minimum norm problems. That is, we assume that the final time t1 is fixed, and obtain an expression (with t1 as a parameter) for the control having minimum p-norm, subject to the constraint that the final conditions must be met. It follows from Holder's inequality (see Ref. 36) that this minimum-norm control can be written as I c Iq"1 ucv(tt)= u.(t) = t sgncvci(tlt)] j=l,..., r where c is the constant k-element row vector which minimizes Ilc VII subject to the con_ ~~~~~~~~~~~_ q straint cg(t) = 1. Here q = P 1 t r q IIc V lq = f, E I vc (tl, t) Iq dt t0o j= J and sgn [ Z] is a function which equals +1 if Z > 0, -1 if Z < 0, and is undetermined but less than one in absolute magnitude when Z = 0. The p-norm of this control is I u(t) = 1 P Ile V II q The solution to the original minimum-time constrained-norm problem is now 66 obtained by varying t1 and finding the smallest value of t1 for which the above p-norm of u(t) is less than or equal to L, the given limit. If L is too small, there may be no control that satisfies these requirements. The limiting cases p = 1 and p = infinity can also be solved by this method, if care is taken in interpretation and definition at various points in the derivation. This is stated without proof by Dranc and Sarachik (Ref. 36).

166 C. 3 The Minimum-Time Problem with Multi-Norm Constraints Kranc and Sarachik (Ref. 37) have further extended these results to minimumtime problems in which the individual components of the control vector are constrained to have norms of various types, each norm less than or equal to its respective limit. That is, 1 tl P. i llui(t)llp f S lui( t)I dt L i=1 r P. il L. L to The various pi's and L.'s may be the same or different. These individual constraints are then incorporated into the single constraint equation Ilu(t) p = f L-P [llui(t)ll P dt < 1 t i= I - 0 and p is allowed to approach infinity. The limiting solution then satisfies all of the individual constraints. Kreindler (Ref. 61) has given a comprehensive treatment of constrained linear time-optimal problems, using a functional analysis formulation coupled with a geometrical interpretation. C. 4 Difficulties Arising in the Functional-Analysis Approach This functional-analysis approach to linear time optimal problems offers solutions to a class of problems some of which are not readily solvable by other methods, but the task of carrying out the computations in an actual problem can be formidable. The difficult step is the minimization of the expression -I VIq [ t l r vj(tlt d - q f vjc(t t) I q dt to j= (or some more complicated expression) over all_ satisfying c_ g(t1) = 1. Furthermore,

167 this must be done for a range of values of t1 sufficiently wide either to include the smallest value of t1 for which 11c VIl = L, or else to show that such a case does not exist. 1 - q

168 APPENDIX D THE CONTROLLABILITY ASSUMPTION The concept of controllability, introduced by Kalman (Refs. 25, 69, 70), is defined as follows: Definition 1 Definition 1: An initia.l state x(t ) = a of a given linear system characterized by the system of equations x = A(t) x + B(t) u (D. 1) is said to be controllable at time to if there exists a control function u(t), depending on a and on the initial time t and defined on some closed interval T = [to, tl] such that x(tl) = 0. Definition 2: If Definition 1 is satisfied for every state a, the system is said to be completely controllable at time to. Definition 3: If Definition 2 is satisfied for all to, the system is said to be completely controllable. It has been shown by Kalman (Ref. 25) that Condition 1: a necessary and sufficient condition that the linear system characterized by (D. 1) be completely controllable at time t is that the matrix t 1 T T f X(t, t) B(t) B(t) XT(to, t) dt t O be positive definite for some tl > to. Here, as in Section 3. 1, X(t, s) is the fundamental

169 matrix of the unforced system (Ref. 57), defined as the solution of d X(t, s) = A(t) X(t,s) X(t,t) = I. dt These definitions and the equivalent condition are not in a form suitable for application here, for three reasons: 1) They are not stated in terms of the system considered in this work; that is, they do not include the "cost" weighting matrix G(t) or the output matrix D(t). 2) They are framed in terms of forcing the system to the origin (its only finite equilibrium point), whereas the problems considered in this work involve forcing the system from a general initial state to a general final manifold. 3) They involve an unspecified final time t1 > to, but the problems considered here usually involve a specified final time. Therefore, to simplify the discussion here, we shall make additional controllability definitions in terms appropriate to the problem at hand. Definition 4: A set of boundary conditions (to) = a (t1) = b for a given linear system characterized by the system of equations k = A(t) x + B(t) u Y = D(t) x (D. 2) is said to be controllable on the interval T if there exists a bounded measurable control u(t), depending on a, b, to, and t1 and defined on T, such that the boundary conditions are satisfied. Definition 5: If Definition 4 is satisfied for all a and b, the system is said to be completely controllable on the interval T.

170 We obtain a necessary and sufficient condition for the satisfaction of Definition 5 by following essentially the same steps followed by Kalman (Ref. 25): Condition 2: A necessary and sufficient condition that the linear system characterized by (D. 2) be completely controllable on the interval T is that the matrix f V(tl, s) V (tl, s) ds T 1 1 be positive definite, where, as in Section 3. 1, V(t1, s) = D(tl) X(t1, s) B(s) G1(s) We prove sufficiency by assuming that Definition 2 is satisfied and then exhibiting a control which allows the boundary conditions to be satisfied: (As in Kalman's proof, this control turns out to be the minimum "energy" control, energy being defined as the integral of I G(t) u(t) 2. ) Such a control can be obtained by setting p = 2 in Eq. 5. 4, using Eq. 5. 5 to eliminate C and cp, and applying the definition of z(t) given by Eq. 3. 5. This gives u(t) = G- (t) (t1t) t)f V(t1, s) T(t1, s) ds [b- D(t) X(t, t)a] where, by assumption, the inversion is an allowed operation. This control causes the required final condition y(t1) = b to be satisfied, as can be verified by substitution in the expression for y(t) given by Eq. 3. 2. The proof of the necessity of Condition 2 also parallels Kalman's proof: Assume that Condition 2 is not satisfied but that the system is completely controllable on the interval T. Under the first assumption, there must exist some vector c 0 such that T [ Vt1, s) vT(t1, s) ds c = 0 L_ T

171 Define _(t) = G(t) VT(t, t) c. Then, f V(t, s) VT(t1,s) ds c = f I G(s) (s) 2 ds = 0 t0 which in turn implies that u(t) = 0 a. e. on T, since G(t) is an invertible matrix for all t e T. From the second assumption, we know that there must exist some bounded measurable control which causes any specified set of boundary conditions to be satisfied, and therefore that there must exist a bounded measurable control (call it u*(t)) which causes the particular set of conditions x(to) = O, j(tl) = c to be satisfied. Thus, from Eq. 3. 2, c = f D(t) X(t1, s) B(s) u*(s) ds = f V(t1, s) G(s) u*(s) ds T T CT C= cl2 = -c V(t, s (s s () G(s) u*(s) ds = s) s) G(s)u*(s)ds. T T Since i(t) = 0 a. e. on T, from above, it follows that Ilc2 = O. But this contradicts the initial assumption that c / 0, thus proving the necessity of Condition 2.

172 APPENDIX E CERTAIN PROPERTIES OF THE SPACES Lr AND Lr p oo E. 1 Completeness The results of Chapters III and V depend on the fact that certain function spaces defined there are Banach spaces. The purpose of this appendix is to establish that these spaces are indeed Banach spaces, and to give a representation theorem for bounded linear functionals on the space Z (referred to in this appendix as L ) defined in Chapter III. The spaces considered are: a) the spaces Lr of measurable r-component vector functions of time z(t) defined p on the interval T = [to, t1] for which f Iz(t) ldt < oo l <p< o T with norm given by 1 Ilzllp = f[ T Iz(t)lP dt]P (E. 1) where Iz(t) I is the Euclidean norm of z(t). b) the space Lcr of essentially bounded measurable r- component vector functions of time z(t) defined on the interval T, with norm zl ess sup z(t) I (E. 2) - O tT - Using the usual definitions of vector addition and scalar multiplication of the elements z(t) of L or Loo it is easy to show that these are linear vector spaces. Likewise it p cc'

173 is easy to show from the properties of the Euclidean norm involved in the definition of II z II and 11z l1 that Eqs. E. 1 and E. 2 define norms on the corresponding spaces. To show that Lr and Lr are Banach spaces it only remains to be shown that these spaces are complete. Lemma: The space Lr is complete for 1 < p < co. Proof: First we prove that a function z(t) is in Lr a) if and b) only if each of its components p zi(t) is in Lp = Lp(to, tl) a) Let zi(t) be in L for i = 1,..., r. Then each z.(t) is a measurable function for which the quantity Iz II = [ f Iz.(t)Pdt]p ilp [I T S zi(t) IP dt] P T is defined and bounded. By repeated application of Minkowski's inequality (Natanson, Ref. 79, p. 198), we can show that 1 1 I T S izi(t)lP dt] > [t S [E Izij' dt From the properties of the Euclidean norm r r r 2 i izi(t)I > z (t) = Iz(t), so that r 1 L T S I~z lt]) l lzillp L> T S I z(t)iPdt llzllp iL1 I P T T Since each II z. lI is bounded, li zll is also, and hence z(t) is in Lp. p p p b) Let z(t) be in Lr. Then 1 (t) < -1 Iz(t) dtp < oo L T

174 Since Iz(t)l = zi2(t) > Izi(t)l for i = 1,... r, it follows at once that [+ f Iz.l(t)P dt] = ilz. i < llzllp < z o - T p so that each z.(t) is in Lp. We now complete the proof of the lemma by showing that every Cauchy sequence in Lp converges to an element of L: Consider any Cauchy sequence [z (t)]~~ in Lr. Then P P -n n=1 P for every e > 0 there exists an N = N(e) such that for all m and n greater than N T Im -(t)-Zn(t)lPdiP < Since, by the above-noted property of the Euclidean norm, Izl > I z, i= 1,..., r, it follows that for all m and n greater than N fT zmi(t - zni(t) dt] < where mi(t) and z i(t) are the ith components of z (t) and zn(t), respectively. Thus the original Cauchy sequence in L corresponds to r Cauchy sequences in L. Since L is P P P known to be a Banach space, and hence a complete space (Dunford and Schwartz, Ref. 85), each of the sequences [zni(t)] converges to a point in Lp, which we denote by zoi(t), i= 1,..., r. Let zo(t) = [z01(t),..., Z] T Then, by part a) of this proof, z0(t) is in Lr Q.E.D. P Lemma: The space Lr is complete. Proof: The proof of this lemma is identical to that of the previous lemma, with the appropriate limiting form of Minkowski's inequality being used.

175 E. 2 A Representation Theorem Theorem: Each bounded linear functional z(v) defined on the space Lr of r- component row vector functions of time can be represented in the form z(v) = f v(t) z(t) dt T where z(t) is an essentially bounded r- component column vector function of time defined on the interval T. Furthermore, llzll = ess sup Iz(t)l = ilzll teT - - o0 Proof: Consider elements of Lr of the form v(t) = [0,..., 0, vi(t),,O.., 0]. When restricted to such a subspace of Lr, the bounded linear functional z(v) must be equivalent to some bounded linear functional z.(v.) operating on elements vi(t) of L1, since if v(t) is in L' then vi(t) is in L1. This same argument applies to each of the components of v(t), which leads to the conclusion that z(v) can be written as the summation of r bounded linear functions zi(vi) operating on the components vi(t) of v(t), i. e., r z(v) = E zi(vi) i=1 From the well-known representation theorem for L1 spaces (Ref. 85, p. 289),each of these r functionals zi(v) is representable in the form zi(v) = J v(t) zi(t) dt where zi(t) is an essenT tially bounded measurable function defined on T. Combining these two results then gives r r z(v) = f vi(t) zi(t) dt = E vi(t) zi(t) dt = f v(t) z(t) dt i=1 T T i=1 T where z(t) is the r-component column vector function z(t) = [z (t),..., zr(t)]T Since each component of z(t) is essentially bounded, Iz(t) I is essentially bounded, and hence z(t) is an element of Lr. 00 It remains to be shown that I z II = 11z ll: From the definition of the norm of a - oo functional and from the linearity of z(v),

176 f v() (t)z(t) dt{ 11zIl sup Iz(v) = sup T (E.3) - 1From the extention of Holder's inequality derived in the lemma67 of Section 3. 4, we have that t _v(t)z(t) dt lzl = ess sup Iz(t) I > T (E. 4) - tE T Ilvll with equality if and only if v(t) = K(t) zT a.e. onT2 (E.5) v(t) = 0 a. e. on T- T where K(t) is any function in L1 and T2 is the subset of T on which lz(t) I = ess sup z(t) I. 1 2 - - (The condition for equality stated in the referenced lemma is framed in terms of a given v(t) and an unspecified z(t). In the evaluation of 11 z [I, z(t) is given and v(t) is varied. Therefore, the condition for equality has been reframed to fit this situation. ) Note that v(t), as given by Eq. E. 5, is in L, since Iz(t) I is essentially bounded. The point of all this is that the largest value that can be attained by the right-hand side of Eq. E. 4 over all v(t) in L1can not exceed 11z l o, and that this supremum is indeed taken on by an element of L1, as given by Eq. E. 5. Thus llzll = llzi, as claimed. Q. E.D. 67 No circular reasoning is involved here, even though the representation theorem being proven here is applied in a section prior to the section in which this lemma is derived, because the proof of the lemma does not depend on this representation theorem or even on the fact that the spaces involved are Banach spaceso

REFERENCES 1. A. A. Feldbaum, "Optimal Processes in Automatic Control Systems, " Automatika i Telemekhanika, Vol. 14, No. 5, 1953. 2. A. A. Feldbaum, "On the Question of Synthesizing Optimal Automatic Control Systems," Trans. 2nd All-Union Conf. on Automatic Control Theory, Vol. 2, Izd-vo, Akad. Nauk SSSR, 1955 3. F. A. Mikhailov, "Integral Indicators of Automatic Control Systems Quality, " Chap. 25 of Automatic Control Fundamentals, V. V. Solodovnikov, ed., Mashgiz, 1954. 4. R. Bellman, I. Glicksberg, and O. Gross, "On the'Bang-Bang' Control Problem," Quart. Appl. Math., Vol. 14, pp. 11-18, 1956. 5. Ostwald, Klassider der exakten Wissenschaften, No. 46, 1894. 6. G. A. Bliss, "The Problem of Mayer with Variable Endpoints," Trans. Am. Math. Soc., Vol. XIX, 1918. 7. 0. Bolza, "t'ber den abnormelen Fall biem Lagrangeschen and Mayerschen Problem mit gemischten Bedingungen and variabeln Endpunkten," Math. Ann., Vol. LXXIV, 1913. 8. G. A. Bliss, Lectures on the Calculus of Variations, Univ. of Chicago, 1946. 9. F. A. Valentine, "The Problem of Lagrange with Differential Inequalities as Added Side Conditions," Contributions to Calculus of Variations 1933-1937, Univ. of Chicago, 1937. 10. V. G. Boltyanskii, R. V. Gamkrelidze, L. S. Pontryagin, "On the Theory of Optimum Processes," Dokl. Akad. Nauk SSSR, Ser. Math., Vol. 110, pp. 7-10, 1956. 11. V. G. Boltyanskii, "The Maximum Principle in the Theory of Optimal Processes," Dokl. Akad. Nauk SSSR, Vol. 119, No. 6, 1958. 12. V. G. Boltyanskii, R. V. Gamkrelidze, L. S. Pontryagin, "The Theory of Optimal Processes I. The Maximum Principle," Izvest. Akad. Nauk SSSR, Ser. Math., Vol. 24, pp. 3-42, 1960. 13. L. S. Pontryagin, et al., The Mathematical Theory of Optimal Processes, Interscience, 1962. 177

14. R. V. Gamkrelidze, "Optimal Processes with Bounded Phase Coordinates," Izv. Akad. Nauk SSSR, Ser. Math., Vol. 24, pp. 315-356, 1960. 15. R. V. Gamkrelidze, lectures given in University of Michigan seminar on optimal control, September and October, 1964, and at the Symposium on the Mathematical Theory of Optimal Control, Ann Arbor, Oct. 1964. 16. L. I. Rozonoer, "L. S. Pontryagin's Maximum Principle in the Theory of Optimum Systems," (3 parts), Automatika i Telemekhanika, Vol. 20, Nos. 10,11, 12, 1959. 17. R. Bellman, Dynamic Programming, Princeton, 1957. 18. R. Bellman, Adaptive Control Processes: A Guided Tour, Princeton, 1961. 19. L. D. Berkovitz, "Variational Methods in Problems of Control & Programming," J. Math. Anal. &Appl., Vol. 3, pp. 145-169, August 1961. 20. L. D. Berkovitz, "On Control Problems with Bounded State Variables, " Rand Corp. Memorandum RM-3207-PR, July 1962. 21. A. Larsen, "Optimal Control and the Calculus of Variations, " Ser. 60, Issue 462, Electronics Res. Lab., Univ. of Cal., 1962. 22. R. E. Mortensen,"A Synopsis of Optimal Control Theory, " Ser. 60, Issue 487, Electronics Res. Lab., Univ. of Cal., 1962. 23. R. E. Kopp, "Pontryagin Maximum Principle, " in Optimization Techniques, G. Leitmann,ed., Academic Press, 1962. 24. S. E. Dreyfus, "Dynamic Programming and the Calculus of Variations," J. Math. Anal. & Appl. Vol. 1, No. 2, 1960. 25. R. E. Kalman, "Contributions to the Theory of Optimal Control," Boletin Soc. Math. Mex., Vol. 5, pp. 102-119, 1961. 26. R. E. Kalman, "The Theory of Optimal Control and the Calculus of Variations," Mathematical Optimization Techniques, Univ. of California Press, pp. 309-331, 1963. 27. "Study of Nonlinear Mechanics, " RIAS Final Report, AFOSR Contract AF 49(638)-382, July 1963. 28. E. Roxin, "A Geometric Interpretation of Pontryagin's Maximum Principle, Tech. Report 61-15, RIAS, Dec. 1961. 178

29. I. Fligge-Lotz and H. Halkin, "Pontryagin's Maximum Principle and Optimal Control, Tech. Report 130, Div. of Eng. Mech., Stanford Univ., Sept. 1961. 30. N. N. Krassovski, "On the Theory of Optimum Regulation, " Automation & Remote Control, Vol. 18, pp. 1005-1016, Nov. 1957. 31. R. Kulikowski, "On Optimal Control with Constraints," Bull. Polish Acad. Sci. (Ser. Tech. Sci.), Vol. 7, pp. 285-294, Apr. 1959. 32. R. Kulikowski, "Concerning the Synthesis of the Optimum Nonlinear Control, " Bull. Polish Acad. Sci. (Ser. Tech. Sci.), Vol. 7, pp. 391-399, June 1959. 33. R. Kulikowski, "Synthesis of a Class of Optimum Control Systems," Bull. Polish Acad. Sci. (Ser. Tech. Sci. ), Vol. 7, pp. 663-671, Nov. 1959. 34. R. Kulikowski, "Optimizing Processes and Synthesis of Optimizing Automatic Control Systems with Nonlinear Invariable Elements," Proc. I. F. A. C. Moscow Conf., pp. 473-477, 1960. 35. R. M. Kirillova, "A Limiting Process in the Solution of an Optimal Control Problem," J. Appl. Math. & Mech., Vol. 24, pp. 398-405, 1960, 36. G. M. Kranc and P. E. Sarachik, "An Application of Functional Analysis to the Optimal Control Problem," ASME Paper 62-JACC-4, 1962. 37. G. M. Kranc and P. E. Sarachik, "On Optimal Control of Systems with Multi-Norm Constraints," Paper No. 1-4, JACC, 1963. 38. L. Markus and E. B. Lee, "On the Existence of Optimal Controls, " Paper No. 61-JAC-2, JACC, 1961. 39. J. Hadamard, "Memoire sur le problem d'analyse relatif a lequilibre des plaques elastiques en castrees," Mem. pres. acad. sci. France [2] 33, No. 4, 1908. 40. F. D. Faulkner, "Direct Methods," in Optimization Techniques, G. Leitmann, ed., Academic Press, 1962. 41. A. E. Bryson and W. F. Denham, "A Steepest Ascent Method for Solving Optimum Programming Problems," J. Appl. Mech., Vol. 29, Ser. E, No. 2, pp. 247-257, 1962. 42. A. E. Bryson, and W. F. Denham, "The Solution of Optimal Programming Problems with Inequality Constraints," IAS Paper No. 63-78, IAS Annual Meeting, New York, Jan. 1963. 179

43. H. J. Kelley, "Gradient Theory of Optimal Flight Paths," J. Am. Rocket Soc., Vol. 30, pp. 947-953, 1960. 44. H. J. Kelley, "Method of Gradients, " in Optimization Techniques, G. Leitmann, ed., Academic Press, 1962. 45. R. G. Graham, "A Steepest-Ascent Solution of Multiple-Arc Vehicle Optimization Problems, " Report No. TDR-269(4550-20)-3, Contract No. AF 04(695)-269, Aerospace Corp., El Segundo, Cal., Dec. 1963. 46. L. A. Zadeh, "Optimality and Non-Scalar-Valued Performance Criteria," IEEE Trans. on Automatic Control, Vol. AC-8, No. 1, Jan. 1963. 47. R. Sivan, "The Necessary and Sufficient Conditions for the Optimal Controller to be Linear," Session XI, Paper 2, JACC, 1964. 48. Z. V. Rekasius and T. C. Hsia, "On the Inverse Problem in Optimal Control," Session XI, Paper 4, JACC, 1964. 49. A. A. Feldbaum, "Dual Control Theory," (4 parts), Automatika i Telemekhanika, Vol. 21, Nos. 9 and 11, 1960, Vol. 22, Nos. 1 and 2, 1961. 50. R. E. Kalman and R. S. Bucy, "New Results in Linear Filtering and Prediction Theory," Trans. ASME, J. Basic Eng., Ser. D, pp. 95-108, Mar. 1961. 51. H. J. Kushner, "Near Optimal Control in the Presence of Small Stochastic Perturbations," Session XIV, Paper 4, JACC, 1964. 52. A. M. Letov, "Analytic Controller Design," Automatika i Telemekhanika, Vol. 21, Nos. 4, 5, and 6, 1960. 53. B. Friedland, "The Design of Optimal Controllers for Linear Processes with I1 Energy Constraints, Melpar Technical Note 62/2, Mar. 1962. 54. C. Sprague, "On the Reticulation Problem in Multivariable Control Systems," Session XVII, Paper 4, JACC, 1964. 55. G. J. Coviello, An Organization Approach to the Optimization of Multivariable Systems," Session XVII, Paper 2, JACC, 1964. 56. J. L. Sanders, "Multi-Level Control," Session XVII, Paper 6, JACC, 1964. 57. E. A. Coddington and N. Levinson, Theory of Ordinary Differential Equations, McGraw-Hill, 1955. 180

58. N. I. Akhiezer and I. B. Glazman, Theory of Linear Operators in Hilbert Space, (English transl. ), Ungar, 1961. 59. J. P. LaSalle, "The Time Optimal Control Problem," Contrib. to the Theory of Nonlinear Oscillations, Vol. 5, Princeton Univ. Press, 1959. 60. A. R. Stubberud, "A Controllability Criterion for a Class of Linear Systems," Wescon Paper 12. 1, 1963. 61. E. Kreindler, "Contributions to the Theory of Time-Optimal Control," J. Franklin Inst., Vol. 275, No. 4, Apr. 1963. 62. A. Meile, "The Calculus of Variations in Applied Aerodynamics and Flight Mechanics," in Optimization Techniques, G. Leitmann, ed., Academic Press, 1962. 63. R. Weinstock, Calculus of Variations, McGraw-Hill, 1952. 64. H. Halkin, "Liapunov's Theorem on the Range of a Vector Measure and Pontryagin's Maximum Principle," Tech. Report 109, Appl. Math. and Stat. Lab., Stanford Univ., May 1962. 65. H. Halkin, "On the Necessary Conditions for Optimal Control of Nonlinear Systems," Tech. Report 116, Appl. Math. and Stat. Lab., Stanford Univ., June 1963. 66. R. E. Kalman, "The Theory of Optimal Control and the Calculus of Variations," RIAS Tech. Report 61-3, 1961. 67. G. Leitmann, "An Elementary Derivation of the Optimal Control Conditions," Lockheed Tech. Report 6-90-61-84, Oct. 1961. 68. M. Krein, "The L-Problem in Abstract Linear Normed Spaces," from Akheiser & Krein, On Some Questions of the Theory of Moments (Russian), 1938. 69. R. E. Kalman, Y. C. Ho, and K. S. Narendra, "Controllability of Linear Dynamical Systems," Contributions to Differential Equations, Vol. 1, Aug. 1962. 70. R. E. Kalman, "Mathematical Description of Linear Dynamical Systems," RIAS Tech. Report 62-18, Nov. 1962. 71. R. Bellman, Introduction to Matrix Analysis, McGraw-Hill, 1960. 72. N. I. Akheizer, The Calculus of Variations, (English transl. by A. H. Frink), Blaisdell, 1962. 73. I. M. Gelfand and S. V. Fomin, Calculus of Variations, (English transl. by R. A. Silverman), Prentice-Hall, 1963. 181

74. J. T. Tou, Modern Control Theory, McGraw-Hill, 1964. 75. R. E. Kalman, T. S. Englar, and R. S. Bucy, "Fundamental Study of Adaptive Control Systems," Tech. Report No. ASD-TR-61-27, Vol. 1, Flight Control Lab., Aero. Systems Div., Air Force Systems Command; Apr. 1962. 76. S. S. L. Chang, "Sufficient Conditions for Optimal Control of Linear Systems with Nonlinear Cost Functions", Paper No. XI-1, 1964 JACC; Stanford, Cal. 77. L. W. Neustadt, "Optimization, a Noise Problem, and Nonlinear Programming", SIAM J. on Control, Vol. 2, No. 1, 1964, pp. 33-53. 78. N. N. Krasovski, "On the Theory of Optimum Control", J. Appl. Math. & Mech., Vol. 23, No. 4, 1959, pp. 899-919 (in English transl. ). 79. I. P. Natanson, Theory of Functions of a Real Variable, v. I, (English transl. by Boron and Hewitt), Ungar, 1961. 80. G. F. Simmons, Introduction to Topology and Modern Analysis, McGraw-Hill, 1963. 81. A. E. Taylor, Introduction to Functional Analysis, Wiley, 1958. 82. W. Rudin, Principles of Mathematical Analysis, McGraw-Hill, 1953. 83. W. A. Porter & J. P. Williams, "Minimum Effort Control of Linear Dynamic Systems," Inst. Sci. & Tech. Univ. of Mich., Aug. 1964. 84. E. Hille & R. S. Phillips, Functional Analysis and Semi-Groups, Am. Math. Soc. Colloq. Pub. Vol. 31, Am. Math. Soc., 1957. 85. N. Dunford and J. T. Schwartz, Linear Operators I, Interscience (2nd printing, 1964). 182

DISTRIBUTION LIST No. of Copies 2 Commanding Officer, U. S. Army Electronics Command, U. S. Army Electronics Laboratories, Fort Monmouth, New Jersey, Attn: Senior Scientist, Electronic Warfare Division 1 Commanding General, U. S. Army Electronic Proving Ground, Fort Huachuca, Arizona, Attn: Director, Electronic Warfare Department 1 Commanding General, U. S. Army Materiel Command, Bldg. T-7, Washington 25, D. C., Attn: AMCRD-DE-E-R 1 Commanding Officer, Signal Corps Electronics Research Unit, 9560th USASRU, P. 0. Box 205, Mountain View, California 1 U. S. Atomic Energy Commission, 1901 Constitution Avenue, N.W., Washington 25, D. C., Attn: Chief Librarian 1 Director, Central Intelligence Agency, 2430 E Street, N.W., Washington 25, D. C., Attn: OCD 1 U. S. Army Research Liaison Officer, MIT-Lincoln Laboratory, Lexington 73, Massachusetts 1 Commander, Air Force Systems Command, Andrews Air Force Base, Washington 25, D. C., Attn: SCSE 1 Headquarters, USAF, Washington 25, D. C., Attn: AFRDR 1 Commander, Aeronautical Systems Division, Wright-Patterson Air Force Base, Ohio, Attn: ASRNCC-1 1 Commander, Aeronautical Systems Division, Wright-Patterson Air Force Base, Ohio, Attn: ASAPRD 1 Commander, Aeronautical Systems Division, Wright-Patterson Air Force Base, Ohio, Attn: ASRN-CS 1 Commander, Aeronautical Systems Division, Wright-Patterson Air Force Base, Ohio, Attn: ASNP 1 Commander, Electronic Systems Division, L. G. Hanscom Field, Bedford, Massachusetts 1 Commander, Rome Air Development Center, Griffiss Air Force Base, New York, Attn: RAYLD 1 Commander, Air Proving Ground Center, Eglin Air Force Base, Florida, Attn: ADJ/Technical Report Branch 183

DISTRIBUTION LIST (Cont.) No. of Copies 1 Chief of Naval Operations, EW Systems Branch, OP-35, Department of the Navy, Washington 25, D. C. 1 Chief, Bureau of Ships, Code 691C, Department of the Navy, Washington 25, D. C. 1 Commander, Bu Naval Weapons, Code RRRE-20, Department of the Navy, Washington 25, D. C. 1 Commander, Naval Ordnance Test Station, Inyokern, China Lake, California, Attn: Test Director - Code 30 1 Commander, Naval Air Missile Test Center, Point Mugu, California 1 Director, Naval Research Laboratory, Countermeasures Branch, Code 5430, Washington 25, D. C. 1 Director, Naval Research Laboratory, Washington 25, D. C., Attn: Code 2021 1 Director, Air University Library, Maxwell Air Force Base, Alabama, Attn: CR-4987 1 Commanding Officer-Director, U. S. Navy Electronic Laboratory San Diego 52, California 1 Commanding Officer, U. S. Naval Ordnance Laboratory, Silver Spring 19, Maryland 3 Chief, U. S. Army Security Agency, Arlington Hall Station, Arlington 12, Virginia, 22212 Attn: 2 Cpys - IADEV 1 Copy - EW Div. IATOP 1 President, U. S. Army Defense Board, Headquarters, Fort Bliss, Texas 1 President, U. S. Army Airborne and Electronics Board, Fort Bragg, North Carolina 1 U. S. Army Anti-Aircraft Artillery and Guided Missile School, Fort Bliss, Texas, Attn: ORL 1 Commander, USAF Security Service, San Antonio, Texas, Attn: CLR 1 Chief of Naval Research, Department of the Navy, Washington 25, D. C., Attn: Code 427 1 Commanding Officer, 52d U. S. Army Security Agency, Special Operations Command, Fort Huachuca, Arizona 1 President, U. S. Army Security Agency Board, Arlington Hall Station, Arlington 12, Virginia 1 The Research Analysis Corporation, McLean, Virginia, 22101 Attn: Document Control Officer 184

DISTRIBUTION LIST (Cont.) No. of Copies 10 Headquarters, Defense Documentation Center, Cameron Station, Alexandria, Virginia 1 Commanding Officer, U. S. Army Electronics Research and Development Laboratory, Fort Monmouth, New Jersey, Attn: U. S. Marine Corps Liaison Office, Code: SIGRA/SL-LNR 1 Director, Fort Monmouth Office, Communications-Electronics Combat Developments Agency, Building 410, Fort Monmouth, New Jersey 7 Commanding Officer, U. S. Army Electronics Command, U. S. Army Electronics Laboratories, Fort Monmouth, New Jersey, Attn: 1 Copy - Director of Research 1 Copy - Technical Documents Center - ADT/E 1 Copy - Chief, Special Devices Branch, Electronic Warfare Division 1 Copy - Chief, Advanced Techniques Branch, Electronic Warfare Division I Copy - Chief, Jamming and Deception Branch, Electronic Warfare Division 1 Copy - File Unit No. 2, Mail and Records, Electronic Warfare Division 1 Copy - Chief, Vulnerability Br., Electromagnetic Environment Division 1 Commanding Officer, U. S. Army Signal Missile Support Agency, White Sands Missile Range, New Mexico, Attn: SIGWS-MEW 1 Commanding Officer, U. S. Naval Air Development Center, Johnsville, Pennsylvania, Attn: Naval Air Development Center Library 1 Headquarters, Aeronautical Systems Division, Wright-Patterson Air Force Base, Ohio, Attn: ASRNCC-10 1 U. S. A. F. Project Rand, The Rand Corporation, 1700 Main Street, Santa Monica, California 1 Stanford Electronic Laboratories, Stanford University, Stanford, California 1 Director, National Security Agency, Fort George G. Meade, Maryland, Attn: RADE-1 1 Bureau of Naval Weapons Representative, Lockheed Missiles and Space Company, P. 0. Box 504, Sunnyvale, California 1 Dr. B. F. Barton, Director, Cooley Electronics Laboratory, The University of Michigan, Ann Arbor, Michigan 55 Cooley Electronics Laboratory, The University of Michigan, Ann Arbor, Michigan Above distribution is effected by Electronic Warfare Division, Surveillance Department, USAEL, Evans Area, Belmar, New Jersey. For further information contact Mr. I. O. Myers, Senior Scientist, Telephone 59-61252. 185

Unclassified Security Classification. m DOCUMENT CONTROL DATA - R&D (Security classification of title, body of abstract and indexing annotation must be entered when the overall report is classified) 1 ORIGINATIN G ACTIVITY (Corporate author) 2a. REPORT SECURITY C LASSIFICATION Cooley Electronics Laboratory Unclassified University of Michigan 2b GROUP Ann Arbor, Michigan --- 3. REPORT TITLE Minimum Peak Amplitude Control 4 DESCRIPTIVE NOTES (Typo of report and inclusive dates) Technical Report No. 164 5. AUTHOR(S) (Lost name, first name, initial) Waltz, F. M. 6. REPORT DATE 7a. TOTAL NO. OF PAGES j 7b. NO. OF REFS May 1965 201 85 8a. CONTRACT OR GRANT NO. 9a. ORIGINATOR'S REPORT NUMBER(S) DA36 039 AMC-03733(E) b. PROJECT NO. 06137-7-T 1PO 21101 AO42 01 Task No. -01 c. 9b. OTHER REPORT NQ($) (Any other numbers that may be assigned this report) d.Sub Task No. -02 10. A VA ILABILITY/LIMITATION NOTICES Qualified requesters may obtain copies of this report from DDC. This report has been released to CFSTI. 11. SUPPLEMENTARY NOTES 12. SPONSORING MILITARY ACTIVITY U. S. Army Electronics Command Fort Monmouth, N. J. __Attnu AMSFTP D-SE 13. ABSTRACT This report considers a class of optimal control problems in which the input signal to a given system is to be chosen so as to cause the output of the system to satisfy specified conditions and so that the peak value of the input over the operatinginterval is a minimum. For cases in which the given system is linear, a theorem guaranteeing the existence of an optimal input and a theorem giving the form of this optimal input are presented, as well as computational algorithms for obtaining numerical values of optimal inputs. Two approaches to minimum peak amplitude problems in which the given system in nonlinear are presented: the first makes use of a certain time-optimal problem, and the second involves a limiting process using an L -space norm in place of the originally-specified cost functional. P DD JAN4 1473 Unclassified Security Classification

Unclassified I Security Classification 14. LINK A LINK B LINK C KEY WORDS ROLE WT ROLE WT ROLE WT Optimal Control Functional Analysis Linear-Non Linear Problem Computational Algorithms Pontryagin's Maximum Principle Dynamic Programming _..^- -.-_. — ~ ~ ~- ~ ~~ ^ ~ ~ ~ ~ ~~ ~ -,- ~ ~- - ~ ~l- ^ l — l^ llI INSTRUCTIONS 1. ORIGINATING ACTIVITY: Enter the name and address of the contractor, subcontractor, grantee, Department of Defense activity or other organization (corporate author) issuing the report. 2a. REPORT SECURITY CLASSIFICATION: Enter the overall security classification of the report. Indicate whether "Restricted Data" is included, Marking is to be in accordance with appropriate security regulations. 2b. GROUP: Automatic downgrading is specified in DoD Directive 5200.10 and Armed Forces Industrial Manual. Enter the group number. Also, when applicable, show that optional markings have been used for Group 3 and Group 4 as authorized. 3. REPORT TITLE: Enter the complete report title in all capital letters. Titles in all cases should be unclassified. If a meaningful title cannot be selected without classification, show title classification in all capitals in parenthesis immediately following the title. 4. DESCRIPTIVE NOTES: If appropriate, enter the type of report, e.g., interim, progress, summary, annual, or final. Give the inclusive dates when a specific reporting period is covered. 5. AUTHOR(S): Enter the name(s) of author(s) as shown on or in the report. Enter last name, first name, middle initial. If military, show rank and branch of service. The name of the principal author is an absolute minimum requirement. 6. REPORT DATE: Enter the date of the report as day, month, year; or month, year. If more than one date appears on the report, use date of publication. 7a. TOTAL NUMBER OF PAGES: The total page count should follow normal pagination procedures, i.e., enter the number of pages containing information. 7b. NUMBER OF REFERENCES: Enter the total number of references cited in the report. 8a. CONTRACT OR GRANT NUMBER: If appropriate, enter the applicable number of the contract or grant under which the report was written. 8b, 8c, & 8d. PROJECT NUMBER: Enter the appropriate military department identification, such as project number, subproject number, system numbers, task number, etc. 9a. ORIGINATOR'S REPORT NUMBER(S): Enter the official report number by which the document will be identified and controlled by the originating activity. This number must be unique to this report. 9b. OTHER REPORT NUMBER(S): If the report has been assigned any other report numbers (either by the originator or by the sponsor), also enter this number(s). 10. AVAILABILITY/LIMITATION NOTICES: Enter any limitations on further dissemination of the report, other than those imposed by security classification, using standard statements such as: (1) "Qualified requesters may obtain copies of this report from DDC." (2) "Foreign announcement and dissemination of this report by DDC is not authorized. " (3) "U. S. Government agencies may obtain copies of this report directly from DDC. Other qualified DDC users shall request through t, (4) "U. S. military agencies may obtain copies of this report directly from DDC. Other qualified users shall request through (5) "All distribution of this report is controlled. Qualified DDC users shall request through If the report has been furnished to the Office of Technical Services, Department of Commerce, for sale to the public, indicate this fact and enter the price, if known. 11. SUPPLEMENTARY NOTES: Use for additional explanatory notes. 12. SPONSORING MILITARY ACTIVITY: Enter the name of the departmental project office or laboratory sponsoring (paying for) the research and development. Include address. 13. ABSTRACT: Enter an abstract giving a brief and factual summary of the document indicative of the report, even though it may also appear elsewhere in the body of the technical report. If additional space is required, a continuation sheet shall be attached. It is highly desirable that the abstract of classified reports be unclassified. Each paragraph of the abstract shall end with an indication of the military security classification of the information in the paragraph, represented as (TS), (S), (C), or (U). There is no limitation on the length of the abstract. However, the suggested length is from 150 to 225 words. 14. KEY WORDS: Key words are technically meaningful terms or short phrases that characterize a report and may be used as index entries for cataloging the report. Key words must be selected- so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location, may be used as key words but will be followed by an indication of technical context. The assignment of links, rules, and weights is optional. 6N Unclassified Security Classification