Language Study Note LSN-032-DR Operator Visibility Robert Dewar 23 Mar 91 The remainder of this message is a rather extended (about 500 lines) note on the operator visibility issue. I would appreciate help in the first instance in refining this note by correcting: o Technical errors o Errors in representation of the MRT proposals o Typographical errors etc o Improperly stated arguments Second, I would be interested in hearing other people add their views on the issue of which of the six possible approaches that I identify is the most appropriate (assuming of course that we do *something* in this area, it is certainly not a super high priority issue requiring attention. On the other hand, if we can find something simple and straightforward to solve the problem, why not?) --------------------------------------------------------------------- Language Study Note on Operator Visibility Robert B. K. Dewar 23 Mar 1991 The Problem, A Background Discussion ------------------------------------ In view of comments from the UI teams that seem to indicate that the current proposed mapping for operator visibility is more complex than it initially appeared, I would like to revisit this issue. The following is a review of the situation (as much to test my understanding as anything else). If you declare a type T in a package P with some operators O and some other subprograms S, then a client package that with's P without using P can use neither O nor S without qualification, and of course the qualification is particularly annoying in the O case. Interestingly, if the client package derives a type T1 from P.T, then the operators O, as well as the procedures P for T1 are available without qualification, since they are considered implicitly declared at the point of derivation. The most surprising case of this phenomenon occurs in the situation: with P; package Q is subtype T2 is T; end Q; with Q; package R is type T3 is new T2; -- operators and subprograms on T3 available -- without qualification ... end R; Here we have a case of operators and subprograms being available without qualification in a situation where the corresponding definitions do not occur in the package containing the applied occurence, or in any package named on the with list, and there are no use clauses in sight. Remember that this is Ada-83 we are describing (programmers allergic to the sight of an unqualified name might ponder the gap in their coding standards, forbidding the use of "use" is not enough to stop this phenomenon!) Now back to the original problem. It seems strange that the client package, say in this case R, can see the operators on T3, but not those on T2. To get visibility on the operators for T2, it is necessary to with P and use P. One could argue that having to with P is not necessarily a bad thing (and indeed normally one is withing the package containing the definitions in question in any case). However, for the programming style that wants to require qualification (except for operators) is unhappy with the use of use clauses. Note that this is an essentially pragmatic position. The objection to lack of qualification applies of course equally well to operators and subprograms. However, the difference in the required syntactic payment: MY_PACKAGE."+" (A,B); -- could be A + B and MY_PACKAGE.P (A,B) -- could be P(A,B) is considered significant enough that in the operator case it is considered to be the lesser of two evils to use the nice notation and suffer the presence of an unqualified entity, whereas in the second case the syntactic overhead is not considered objectionable. One can of course reduce the pain in the first case by introducing renaming declarations, and some would argue that this should be good enough. However, it is clearly the case that there is a substantial constituency which doesn't buy this argument, which is why we are willing to consider the possibility of a change in Ada 9X specifically to accomodate this segment of the Ada community. Note, and this is significant, that other segments of the Ada community: o Those willing to introduce renames clauses (and preferring to do so, because at least the required qualification appears ONCE in the client module). o Those willing to make general use of use clauses are not affected by this issue, and thus any proposed solution should desirably not intefere with these approaches. Besides the pragmatic argument, there is a uniformity argument (i.e. one based on the idea of trying to minimize surprises) that finds it odd that the types T2 and T3 are treated so differently in our example, and would like to remove this surprising difference. Finally, we note that the allergy to the use of qualification (on the part of those who normally *prefer* to qualify) may extend to character literals: A := MY_PACKAGE.'A'; and possibly even to other enumeration literals, although the case is less clear here, since these are simply names. In the remainder of this note, we use the term "operator" to include character literals and any other entities that are to be specially treated in the same manner. Other Issues ------------ There are two related issues. First we have the problem of visibility of operators in generic templates. In Ada-83, the predefined operators are always the ones that are obtained in generics for generic formals. This note is NOT addressing this issue, which seems to be clearly separable from the issue of normal direct visibility. Second, the MRT proposal involves changes to the overloading algorithm. Although there may be some possible interactions, again it seems to be the case that these two issues can be separated, so in this note we do *not* consider the possibility of changes to the overloading algorithm, except in a very restricted sense. Possible Approaches ------------------- In this note we consider a space of six possible approaches to providing a solution to this problem. First, there are two possible sources for the operators to be made visible: 1. Type-based Visibility In this approach, the operators that are made visible are those that are derivable from types that are visible. In other words if the type is visible (with or without qualification), then the derivable operators of this type are also visible. 2. Package-based Visibility In this approach, the operators that are made visible are those that would be made visible if use clauses were provided for the packages that are with'ed by the current package. Note that the set of operators involved is quite different in these two approaches. The type-based visibility may make derivable operators visible from packages that are not with'ed, and the package-based approach may make non-derivable operators visible from packages that are with'ed. Consider the following example: package P1 is type T1 is new INTEGER; function "+" (A,B : T1) return T1; end P1; with P1; package P2 is subtype T2 is T1; end P2; with P1; package P3 is function "-" (A,B : P1.T1) return P1.T1; end P3; with P2; with P3; package P4 is ... end P4; In P4, the defined "+" operator is a derivable operator of the type T1 which is visible via the subtype P1.T2, and is thus potentially derivable using the type-based visibility approach. The package-based approach cannot make this "+" visible, because the package in which it is defined (P1) is not with'ed by P4. On the other hand, in P4, the defined "-" operator is defined in the package P3 which is with'ed by P4, and is thus potentially derivable using the package-based visibility approach. The type-based visibility approach cannot make "-" visible because it is not a derivable operator. Notes: A more extensive version of the type-based approach, one which made *all* operators on the type available, whether or not they were derivable, could make the "-" visible, but this approach appears to imply unacceptable implementation requirements of scanning definitions that would not normally be accessed. In the case of the package-based approach, the "+" operator could be potentially made available by adding a with clause for package P1 to P4. In Ada-83, additional with clauses could always be provided in this manner. In Ada-9X, if the concept of private packages is introduced, it is possible that there are situations where operators in private packages could only be made available via the type-based approach. The second dimension of the possible solution space involves the mechanism by which the operators are made available. There are three possibilities: 1. Require explicit syntax In this approach, the additional operators are made available only in the presence of some additional syntax added for this purpose. Clearly this approach is 100% upwards copmpatible, since in the absense of this (presumably illegal Ada-83) syntax, the additional operators are not made available. As an example of possible syntax in this area, we could add a new kind of use clause: use type_name; For the type-based approach. The type name would normally be qualified to indicate the package it comes from, perhaps such qualification is required (which would ensure that the form of this use clause was syntactically unambiguous). This form of the use would indicate that derivable operators for the designated type are to be made available. use (package_name); For the package-based approach. The parentheses around the package name would indicate that operators, but not other entities, in the designated package are to be made available. 2. Make the operators visible without any syntax In this approach, the relevant operators are made visible without the need for any additional syntax. This of course reduces the amount of additional syntax, but at the expense of introducing instances of non-upwards compatibility. For example, the following program, which compiles in Ada-83, would not compile under this approach: package P1 is type C is new CHARACTER; function "+" (A,B : C) return C; end P1; with P1; package P2 is function "+" (A,B : P1.C) return P1.C; end P2; with P1; package P3 is function "+" (A,B : P1.C) return P1.C; end P3; with P1, P2, P3; use P3; package P4 is X : P1.C := 'A' + 'B'; end In Ada-83, the "+" in P4 must be the one defined in P3, since the operators defined in P1,P2 are not directly visible. However, if the automatic visibility approach is adopted then an ambiguity results (package based visibility would make all three versions of "+" available, and type-based visibility would make the "+" in P1 visible). 3. Make the operators automatically visible at a lower preference Here we are talking about a lower preference of visibility. The idea is that the extra operators are automatically visible, but this visibility only applies if an expression is otherwise unresolvable. The intention behind this idea is on the one hand to avoid the need for unnecessary syntax, and on the other hand to avoid introducing instances of non-upwards compatibility. Note that making previous illegal programs legal is *not* considered to be an instance of non-upwards compatibility. This is the approach that certainly does interact with the overloading algorithm. Presumably the formal statement of the algorithm is that an attempt is made to resolve an expression without taking into account any of the potential additional operator visibility. If this attempt fails, then a second attempt is made to resolve the expression with all the additional operators being made visible. It may be possible in practice to somehow merge these two stages to optimize this process. Any of the three approaches to making the operators visible can be applied to either of the two basic methods for selecting the operators to be made visible, giving six possible approaches in all. A Comparison of the Approaches ------------------------------ In this section we compare the various approaches from several different points of view. o Meeting user needs o Ease of implementation o Upwards compatibility o Ease of description o Contribution to uniformity 1. Meeting User Needs In terms of meeting user needs, either the type-based or package-based approach will usually be adequate, since in most cases the operators in question are derivable (character literals are of course always derivable). However, there are cases in which operators are defined which are *not* derivable, and hence cannot be made available by the type based approach. For example, if we have: package MATRIX_OPS is type MATRIX is private; ... end; package VECTOR_OPS is type VECTOR is private; ... end; with MATRIX_OPS; with VECTOR_OPS; package MATRIX_VECTOR_OPS is function "*" (M : MATRIX_OPS.MATRIX; V : VECTOR_OPS.VECTOR) return MATRIX_OPS.MATRIX; end; A package that with's MATRIX_VECTOR_OPS cannot get visibility to the mixed type "*" defined in this package with the type-based approach since this operator is not derivable. On the other hand the package based approach can provide visibility to this operator. The issue is how often this situation arises in practice. Mixed type operators are certainly not that unusual, although they will often be defined in a single package which contains definitions of the two types involved, and therefore be derivable (note that this is the case where derivation does not work so nicely in any case because the cross cases cannot be derived as desired). Certainly it *does* occur, so user need considerations tend to lean in the direction of the package based approach. At least in the case of Ada-83, the package based approach can make all required operators available. This may possibly involve the addition of use clauses, but it can probably be argued that the introduction of these use clauses is in any case desirable. For example: with P; package Q is subtype T2 is T; end Q; with Q; package R is ... end R; The type-based approach potentially makes operators defined in P available to package R for operations on objects of type Q.T2, even though package R does not directly with package P. However, it can certainly be argued that adding a "with P" to package R is helpful in this case, since it warns the reader that declarations in P are used within package R. Indeed the great majority of Ada programmers (essentially all in my survey so far) are surprised to find out that it is possible to write a statement: MY_FUNCTION (4); in a package under conditions where MY_FUNCTION is not explicitly declared in either the current package or in any package that it with's. It can safely be said that this is not considered to be a useful feature whose applicability should be extended, even in the case where we write: X := 'Q' + 6; we would prefer that the definition of both 'Q' and "+" occur either in the current package, or in a package that is with'ed by the current package. Regarding the choice among methods of making the additional operators available, the main issue is to reduce surprises. At the moment in Ada-83, users are continually surprised that the operators are *not* available, they just don't think of operators as being like other entities that need to be qualified, and find it surprising and annoying that the operators are not available. The requirement of using "use" clauses to get visibility is upsetting to the segment of Ada programmers who dislike use clauses. This consideration argues in the direction of one of the automatic methods which does not require additional syntax. Of course the extra syntax method at least provides an easy work around to deal with the surprise. Note of course that programmers who routinely use "use" clauses, at least on packages that define types and operators, do not get surprised in the first place, and are not affected. It is also important to remember that user needs are not in practice restricted to ease of programming. Users are also concerned with the ease of implementation (affects cost and reliability of the compilers they use), upwards compatibility (affects ease of Ada-9X transition), ease of description and uniformity (affects ease of learning Ada-9X). 2. Ease of Implementation Implementations are likely in practice to have the basic mechanisms available for both the package based and type based approaches. The package based approach is simply a matter of applying the normal "use" type processing on a selective basis. Two approaches seem possible for normal use clause handling. Either the individual entities involved are marked as visible as a result of the use clause or a test is made during the overloading algorithm to determine whether or not the package from which an entity is obtained has a use clause. Either approach is easily, and locally, modifiable to accomodate the possibility of selective use. In the case of the type based approach, the compiler must have a mechanims for making the operators (and other entities) available in the case where types are derived from the types in question, so presumably this same mechanism can be made available for the type itself. Many compilers are quite prepared to "find" these operators, as can be determined from the error messages they produce. It is possible that in some compilers, the effort to implement the type based visibility may be slightly complicated by the need to explicitly add entries to the symbol table which are not necessarily present. For example, if the approach of the compiler is to "go and get" these operators only in response to a type derivation, and now such compilers would have to go and get them anyway. A survey of implementors would resolve this question quite quickly (the results from the UI teams will be of some relevance, but this is clearly an area where the difficulty of implementation is very dependent on compiler organization choices, so the UI input is a little limited). With regard to the method of making these operators visible, it is clearly the case that the third approach (introducing a new level of preference) is the most complex. This appears to involve possibly significant modifications to the overloading algorithm, an exceedingly delicate part of most compilers. Either of the first two approaches are fairly straightfoward. The automatic visibility is marginally simpler, since it avoids extra syntactic processing, but in practice this kind of additional front end work is very straightforward. 3. Upwards Compatibility The only possible upwards compatibility issues arise in connection with the second approach to making the operators visible, since this definitely does introduce non-upwards compatible cases. Solutions that introduce instances of non-upwards compatibility, resulting in legal Ada-83 programs that do not compile in Ada-9X, are clearly undesirable, other things being equal. In particular, for the segment of Ada programmers who use "use" clauses in these situations, the introduction of non-upwards compatibilities is wholly negative. However, the situations which cause these non-upwards compatible cases are very forced, so in practice this situation is very unlikely to arise. This consideration argues for avoiding the second approach, in which the operators are simply made available all the time with no special handling. However, if this is considered to be a sufficiently marginal case, the argument may be very weak. 4. Ease of Description The package based approach seems to be much easier to describe. This is because the mechanism of making entities visible is one that is already present in the language and described currently. For example, if we take the approach of automatically making operator declarations visible, then something like the following is sufficient: In 8.3(6): "A declaration, other than an operator declaration, is visible by selection ..... and then a note could be added noting that operators are always directly visible. If the first, approach, explicit syntax is used, then the modification would be to 8.4, and would be something like the following. First we change the syntax in 8.4(2). use_clause ::= use package_name_reference [, package_name_reference]; package_name_reference ::= package_name | ( package_name ) then we make some fairly small changes to the text in 8.4(4) to distinguish the two cases. Something like the following: "except in the following three cases:" o The declaration is of an entity other than an operator and the corresponding package name appears in parentheses in the use clause. I don't pretend to be an expert in visibilty rules, so I can't be sure these changes are accurate, and I am sure they are not as elegant as they could be. However, the fact that I can make a reasonable shot at the changes indicates that they are fairly straightforward. On the other hand, the type based approach seems more problematic from a description point of view. Introduced for the first time is the concept of visibility of declared entities outside the scope of their declarations, and this seems to me as though it would cause a number of subtle changes to the text. The approach is to draw a parallel between the visibility of operators and of basic operations such as assignment. It is possible that this can be done in a clean and neat way without affecting many sections of the RM, but it seems potentially complex. Perhaps someone could indicate the exact wording changes. It would then be easier to make a more informed comparison, With regard to the method of making the operators visible, the only approach that significantly complicates the description is the introducation of an additional preference in the visibility rules. This would require some delicate wording changes in the overloading rules. Right now, the preference rule that favors non-used entities (8.4(5)) is simply stated in terms of whether or not the used entity is visible in a scope. This new preference rule depends on the resolvability of a specific declaration, and thus interacts with overloading. I suspect that this may not be easy to achieve with a small change to the wording, but again one cannot be sure without actually going through the excercise. It is of course easier to describe the automatic approach than the approach requiring additional syntax, since if nothing else the syntax must be described. In the case of the type-based approach, the modification to the syntax of use is a little more extensive, and the description consequently more complex. Finally, it should be noted that there is an important interaction between non-upwards compatibility and ease of description. An obvious requirement for the Ada-9X documentation will be careful documentation of any instances of non-upwards compatibility. Even in cases where these non-upwards compatibilities are extremely marginal (i.e. extremely unlikely to affect any real programs), they will still have to be carefully written up. Furthermore one may expect that current Ada-83 programmers will read these descriptions carefully to determine whether or not they represent a problem. Thus ease of description considerations argue against the adoption of the non-upwards compatible second approach of automatic visibility, even if it is the case that very few (or even no) existing programs are affected. 5. Contribution to Uniformity It seems unlikely that any of the considered approaches can be argued to be contributing to uniformity. After all, at the heart of the issue is the desire to treat certain declared entities in a different manner from other declared entities. From an informal point of view, it does seem strange that entities which are implicitly declared by a derived type declaration are not visible for the parent type: type T1 is new T2; -- operators on T1 *are* visible -- but operators on T2 *not* visible The type-based approach may be argued to be addressing this apparent non-uniformity. On the other hand, if a distinction is made between operators and procedures, we will still have this effect: type T1 is new T2; -- procedures with parameters of type T1 *are* visible -- but procedures with parameters of type T2 *not* visible Furthermore, treating derivable operators in a special manner does introduce other potential non-uniformities. As mentioned above, it means that the general uniform rule that entities cannot be referenced outside their scope no longer holds. Current Proposals ----------------- MRT Proposal As far as I can determine, the current MRT proposal is the type-based approach with the extra level of preference. The reason that I am a little confused here is that the IMD does not mention the extra level of preference (the only statement is that the operators are made available with the same visibility as basic operations, which would tend to imply no extra preference level -- i.e. our second approach). On the other hand, the discussion with the UI teams seems to clearly imply that an extra level of preference is intended. One would expect this to guarantee upwards compatibility. There is some confusion on this point (a recent note from the MRT indicated some uncertainty as to exactly what the non-upwards compatible implications were). An additional confusion here is that the operator visibility issue is considered to be strongly connected to the proposal for modification of the overloading rules (they are for instance included in the same UI implementation module). The proposed overloading rules in any case introduce instances of non-upwards compatibility, independent of the operator visibility issues. In a subsequent discussion, Tucker proposed the possibility of adopting the explicit syntax approach, still in the context of the type-based approach (i.e. the introduction of a use typename syntax). This does not appear to have been further persued. Presumably it would obviate the need for an extra level of preference, and also the need for modifying the overloading algorithm (as previously mentioned, a separate proposal in any case suggests modifying the overloading algorithm, but here we are being careful to consider only the aspects directly related to the operator visibility issue). Dewar's "low bid" ----------------- I have previously proposed the package based approach with explicit syntax (the use with parentheses). This of course is completely upwards compatible and does not interact with the operator overloading algorithm. Other Possibilities ------------------- Note: in this section, more than in previous sections, I introduce my own judgments on the issues! The issue of whether we are willing to introduce the non-upwards compatible approach of making the operators directly visible automatically with no special preference rule. How serious are these non-upwards compatibilities? It is possible that very few existing programs are affected. As previously discussed, there is still the problem that the incompatibility must be described, and in general a very good argument must be made to justify violating the upwards compatibility rule. It is certainly the case that (apart from the compatibility consideration), the automatic approach seems superior. It is the easiest to describe, requires no extra syntax, and avoids any surprises of operators being invisible. My judgment is that we should seriously consider this approach. It can of course be applied to either the type-based or the package-based approach, but it is certainly easier to explain the incompatibility on the basis of the package-based approach, since it can be explained directly in terms of Ada-83 use clauses. This is because the additional ambiguities introduced are simply a subset of those that would be introduced by adding Ada-83 use clauses. In particular, this upwards compatibility would definitely *not* affect any Ada users who write the use clauses anyway. This is not true of the type based approach, and the differences, though unlikely to occur in practice, are subtle and non-trivial to describe. I would therefore be quite willing to consider a variation of the package-based approach which made the operators available automatically. Of course, I am of the school that always uses use clauses anyway, and so this choice cannot possibly introduce any problems for me (it also doesn't particularly help, but then the whole issue is an unimportant one for users of "use" clauses). For users who *do* use use clauses, there is often a perception that the operators are always wanted anyway, so they may be perfectly willing to accept the additional ambiguities that can theoretically be introduced. After all, these are cases where there are operators around that can only be accessed with selected notation in any case, which is generally considered to be undesirable. A brief look at this problem several weeks ago lead me to conclude that the package based approach is preferable to the type based approach, and this more detailed examination certainly has not changed my mind. The primary reasons that I prefer the package based approach is that it seems simpler to describe and understand, and it accomodates cases (of operators defined outside the package defining the type) which the type-based approach is incapable of supporting. In any case I certainly dislike the addition of another visibility preference rule. This seems to me to introduce far too much descriptional and implementational complexity for very little gain. If upwards compatibility is deemed to be of over-riding importance, then I would definitely prefer the explicit syntax approach. Summary ------- This note has identified six possible approaches for solving the operator visibility problem: Type-based approach with explicit syntax Suggested by Tucker during email discussions some weeks ago, but not further persued. Type-based approach with automatic visibility Seems to have been displaced in MRT thinking by the additional preference approach. Would introduce marginal cases of non- upwards compatibility. Type-based approach with additional preference rule This is the current approach being worked on by the UI teams. It reduces or possibly eliminates problems of non-upwards compatibility. None of the typed-based approaches can address the case of operators defined outside the packages that define the relevant types. Package-based approach with explicit syntax This is the original Dewar "low bid". Like all the package based approaches, it can deal with the case of operators defined outside the packages that define the relevant types. In email notes, I argued that this is simpler to describe, implement and understand, arguments which have been further expanded in this note. Package-based approach with automatic visibility In this note I argue that this approach should be given further consideration. It is marginally non-upwards compatible, but the instances of incompatibility seem rare and relatively easy to explain. It reduces programmer surprise compared to the explicit syntax approach. Package-based approach with additional preference rule Not currently being suggested by anyone, but still a perfectly coherent choice. Robert B. K. Dewar