Project Athena - X Window System Users and Developers Conference, Day 2 [3/6]

[MUSIC PLAYING]

HALL: I'm Frank Hall. I'm the project manager for the x-ray project at Hewlett-Packard. We produced the XR lib user interface library, which we've contributed to Project Athena. I'll talk a little bit about that and about what the x-ray project is attempting to do. our commitment to standards, and sponsoring, identifying and sponsoring standards for user interface within the industry.

There are a couple of handouts. One of them, this is a short paper, which summarizes sort of what I'll say. And a copy of my slides are in the back.

There is an excerpt from our programmers manual. It's called Programming with the X Windows System. This is a prototype of it. It's becoming a product. And we'll be shipping supposedly first of February if the shipping department is in line.

This is an excerpt. It contains the table of contents from the manual, which shows you, the second chapter is the XR lib, excuse me, Xlib Documentation Standard from MIT Unchanged. And it just goes through and shows you the basic structure of the manual.

What you have here is the first chapter of the manual that steps you through four examples, Programming in X, then a little bit higher, Programming, adding a little bit of title bars, pieces out of the XR library, moving you up until you have a simple hello world program that has dialog boxes and menus and so forth. And I'd, if any of you like to misspell words, I would suggest that you do it right and misspell it in big, bold letters on a yellow cover--

[LAUGHTER]

--and pass it out to a bunch of people. Okay. Okay, let me tell you a little bit about our background. That'll be helpful in understanding what it is we're trying to do.

Hewlett-Packard has a commitment to standards. And we also, and that means Unix in the workstation market. We are committed to Unix as a standard. We want to stimulate software development for Unix workstations.

And we feel that it's going slowly. And we want to stimulate that and get it moving. We see the MS Dos world that has a lot of advanced user interface software. We want to see that flowing into Unix. We want to see a lot of it on Unix, more of it on Unix than MS Dos, want to get it going.

How do we do that? Well, let me start with a quote from Dave [? Rosenthal, ?] now of Sun. This came up in the Unix Review. I don't know. Can you see that?

AUDIENCE: --spotlights? No, spots.

HALL: Thank you. Okay, this is a direct quote. "Does X offer the capabilities needed to become a de facto standard?" He breaks it up into three things that would be needed.

"The lowest generically describes something like X. On top of it, there's a user interface library that adds components like menus and so forth. And you stick pieces together to, you program at that level by sticking pieces together. Obviously, this library should be closely matched to the layer beneath it, such as X. On top of that, you want a level that lets you generate user interface easily."

We had been, before we discovered X we had been thinking along these very lines for about two years, gradually getting more and more of a vision of where user interface needed to go. When we discovered X, we loved it. We wanted to promote it as a standard.

We asked ourselves, what could we do? Customers were telling us that we needed to have a user interface library for Hewlett-Packard. What were we planning to do? Put pieces together and came up with this x-ray user interface library.

And the way that we were instructed to proceed, I think, is very telling in the need for Unix to get user interface standards going. We started two years ago. We first were going to do a series of advanced user interface Unix applications and discovered immediately we needed a toolbox.

And you take your prototype application, typically the way toolboxes start. You take your [? Whizzy ?] application. You say, what can I take out of this? And turn it into a little toolkit, and we've seen some examples of that this morning, okay.

Then you say, well, now. I'm missing some features. Or there's some rough edges I. Want to re-architect it. You want to throw it away, come back, do it right.

And then if you proceed like we did, somebody else inside the big corporation that you're in says, now, wait a minute. Who are you guys? We have all this list of credentials for user interface. We're going to do the right user interface.

You guys, cancel your project. Just wait six months. We'll come out. It'll be great. It'll be a standard, new standard.

So you get canceled. You sit around. And you secretly keep programming in the background. And finally, those people are about to come out with it.

And then somebody says, it's not object oriented. That's the future. And what we really need as a standard is object oriented. And who are you guys? You don't know about object oriented programming.

Cancel your project. So that gets canceled. We're sitting around programming in the background. And we were revived, canceled, revised, canceled. And we were getting a little tired of that.

[LAUGHTER]

And decided that if we were given a chance, we were going to just go for it. And we had planted seeds in people's minds that we knew what we wanted to do. We had a plan. Let us do it.

And when a very big company came to HP management and said, you don't have a user interface library. What are your plans for them? If you don't have plans, you can tell us. Then forget it.

Okay, that got John Young's personal attention. He's the CEO of HP. Immediately, ramifications running down the chain of command, okay, we need to fire up a user interface library project immediately. Okay, bam, we're back in business, we've got, okay.

[LAUGHTER]

[APPLAUSE]

All right, okay, so we're ready to go. And the question was, what do we do? And first, we had to do a lot of time thinking about what to do at this level. X came along. That was great.

We changed our name to X-ray. We saw what we had to do, we needed to be closely matched to X. We wanted to move on. And we gradually planted seeds that X is in the public domain. What we should be doing, should be in the public domain.

And gradually that built up. And what we now are trying to do is stimulate public domain standards for user interface. And XR lib is the first step in that.

Let me just show you a little slide quickly. Those of you who are trying to sell the concept of X and client server architecture, you know how hard that is to convey. I suggest a slide sort of like.

The only significance of this for our talk is that we use this to show how X-ray fits on top of X. The application uses both X and X-ray. They work together. And that's basically the way it all fits.

Let me show you what X-ray has. Our design focus was X version 10, Looking ahead to version 11, we wanted, our strategy is first to address current customer needs.

We have very real customers with very real needs. They want to be selling products. They want to have products on the market this year. They want to be developing now. They were asking us for our next phase now.

We simply can't wait for a future standard. We see that as another example of possibly the leapfrogging that we've been encountering within HP. So we are moving ahead towards a standard.

But we want to start with addressing current customer needs. It's not, the current customer is programming in C right now, standard languages. They're looking ahead towards objective languages like C++. So we have to have a plan for how to get there. But our first strategy is first to address the current customer needs.

Our research told us that we need to have an architecture that allowed layout of arbitrary objects that the user can add, a thermometer, whatever the user wants to add in. These interactive gadgets have to be done in a way that a panel of them can accept new ones, just coming in, and still operate on them, send them messages like, make yourself sensitive. Make yourself insensitive, without even knowing what it's talking to.

So the architecture that evolved is at the highest layer, you have the interactive dialogue components like menus. We provide pop up, cascading, or walking menus. The items in the menu can have keyboard equivalents so that for Power users, they can see the menu. They realize, if I want to hit Control B, I'll get this function without popping up the menu.

It allows you to serve both initial users as well as Power users. Items in the menu can force direct functions or events to pop into the X queue or execute functions directly. And the user has control over configuring how the access to the menu is, what button should bring up the menu.

When the menu comes up, am I in a drag model. Or am I, does the menu come up and then I go and select and click again? Those things are left to the user. The purpose of X-ray was not to dictate a particular user interface but allow people, if they're configuring a system, to choose what fits for them.

Message boxes, this is the dialog box. We called it Message Box because we didn't want to confuse the dialog layer with a dialog box. We went for ease of programming. Like you can set up a menu, a simple menu just by initializing a constant that has a set of ASCII strings in it for your menu items.

That's your data structure that you need for the simplest menu, very easy to do, similar for dialog box. And then you feed that structure into the call to create a dialog or message box or a menu. And you've got it.

Panels are built up of field editors, arbitrary field editors, which is what we're calling these interactive gadgets. I'll explain that. It operates as a whole. Hewlett-Packard's work in user interface for many years, we've been active in it, had been mainly based on terminals.

First we started out with soft keys and made those important. And we've been working in block mode, working in, by terminal. When the terminal takes over, moving you around the forum and so on.

What we needed to do, we felt, was to extend that forum concept. And this worked well to sell the idea within HP. We're extending the forum concept into a panel. And the panel manager allows things like, helps you traverse between the fields.

If you want to have just a simple form that has text fields in it, you can do it. And you can hit your Return key and just skip right through it. It looks like a form. That was the basic idea at that level.

The field editors, the interactive gadgets, we see them as objects. They really are objects. That's bitmapped, graphical user interface just cries out for objects. And they control the appearance of a region, take input from it. They can contain data that the program can set and read back if it wishes.

And its appearance, it depends upon the state that it's in, sensitive, insensitive, users, many other states that you can put it into. We find a standard set, basically various buttons, titlebar, scroll bar, pixmap functions to either placing or editing them, placing and editing a text, either placing it statically or editing with single or multi-lines. These were built on top of the intrinsic layer, which is an input abstraction on top of X Next Event, which allows you to merge input from various file descriptors, setting time outs.

And you can get your input by either a blocked, you can blocked or unblocked, of course. But you can have a cold read, which is uninterpreted events coming to you. Or you can ask for a hot read, in which the input model then is taking over.

And it is interacting with the user without the application needing to be aware. So when you're in a hot read, if the user brings up a menu and it traverses down the hierarchy, decides, no. I don't want that. Goes over, comes out another, so on, makes an item selection, the application is not involved in any of that until the user finally does make a selection.

And then either the function is executed directly or the event gets popped back in the input queue that says, the user selected this item out of the menu. We found that a useful abstraction. The message oriented interface I should talk about.

We used a very regular way of interacting, our programmatic interface, which is basically, the parameters are three, instance, message, and data. And I work with DEC on the X Toolkit and so forth. One of the questions that's come in on the comment, the proposal that we put forth for the X Toolkit is, why do you want that?

I'd like to address that. We found it first, the previous, simpler versions of X-ray were procedural. Each field editor was represented by a set of procedures. You'd have a cluster of them. And then the next field editor would have another cluster of them and another cluster and another cluster.

There were two problems with that. One, a panel couldn't arbitrarily manipulate one of these field editors without knowing ahead of time what procedure it should call to perform that function on that field editor. It is very difficult to add arbitrary field editors in.

What you wanted was one thing to call or send a message to. It says, do this operation. I don't know anything about you, but just do this operation. And so we found it much better to have one call with a message and an instance to work with.

Secondly, we found that by putting that message oriented interface throughout the library, there was transfer of programmer learning. At first there were some rough edges where we didn't use that style. And programmers using X-ray were making mistakes where they were putting that style in to places where we hadn't put it.

And we realized that just by putting it in and following the programmer's intuition, you get transfer of learning and a much stronger, more comfortable, easier to use programming environment. And then other advantages of that just float on in, that it gradually gets the people that we're, all of our customers are not looking ahead to object oriented programming. But by getting them in, getting used to that style, we're preparing them to phase over into object oriented programming.

Let me quickly show you the, an overview. These are done by hand. These are not taken off the screen. This is from marketing and so forth.

That's basically the visuals, how they look without these boxes. These are not, that's just an artifact of a slide. This is a quick view of a panel with a title bar-- a message box.

Here's a menu showing a title bar. It's optional, cascading down. There's a keyboard equivalent, control-C. It's a pretty nice one to have to get it for getting out.

Where are we going in the future? So the current status is that X-ray is a product. It's going to be supported by HP. It has, like all products of HP, we have to extend it into the future.

We want to bring it over onto whatever standards evolve on X. And we, there is more of our plan here that we have to put in anyway. We've made commitments to our customers.

And so what we're doing is rolling in the good ideas that are coming out of our work on the X Toolkit. And we're offering a migration path to our customers onto the X Toolkit when it can finally stabilize. And we want it to stabilize as quickly as possible.

What's coming in our next phase is reading and writing of panels and so forth, these are into files, taking them back. Then we went to use that for transition tools to getting, to go off into the X Toolkit. So if you've spent a lot of time doing an application. It's got bunches of panels and message boxes and menus and so on and so on.

You don't have to redo all that. In whatever standard emerges, it can be just converted over for you. Our intent with this is to continue adding functionality, make it easier and easier to use, and provide a support path on into the future.

Add native language support for world I/O. That's part of an ongoing effort in just world I/O that we're doing in general with Unix. Our native language support system is being contributed to X Open by Hewlett-Packard.

Language bindings, release 11, will be there in release 11. Cut buffer management, callback functions and event dispatch intrinsics, these are both very strong architectural capabilities that you want to have. The X Toolkit work that we're doing with DEC makes great use of this.

And it solves a lot of difficulties that you run into. And we want to work these in. And this is part of the plan to bridge over. Add more field editors.

We expect that there will be field editors contributed by other people inside HP or people outside HP. If you're programming and you're adding, you want to do your own custom field editor, you can contribute that. The source for X-ray has been given to MIT. As well as the source for this document was also contributed to MIT.

But what we see is needed is for the industry to work together on the platform. So what we call a platform upon which software applications are built and above which different hardware engines run. The platform has the planks in it that allow applications and hardware to run in a heterogeneous environment.

Planks in that platform are obviously the operating system and networking, user interface, windowing. The whole ball of wax, it's at the platform level that we need to work together to forge standards. We want to make Unix successful. And we want to get the software gates opened.

And so my message to you is, if you're a software developer, start developing today. Let's get that software out there and get it on the market. Let's make money, Okay.

[LAUGHTER]

Okay, all right, I think that's really all the time I have. So if there are any questions, maybe we can--

--repeat any questions [INAUDIBLE].

HALL: Okay, Jon, you're on next.

BREWSTER: Do you have the slides?

HALL: Yes, I do. Okay, but let me answer a couple of questions and then-- okay. Excuse me.

AUDIENCE: Do you expect the resource files to have ASCII representations as a programmer could create an X file using a standard editor and use that as the resource file rather than being some, incur a binary representation there?

HALL: OK the question is, should the format of the resource files be pure ASCII that are editable? Or should they be some kind of binary? We see that it needs to be binary. Because there are things besides Ascii data that you want to put in there.

If I write a chess game and I'm using a font, a special font to create the chess pieces and so forth, I want to be able to write my font out to the resource file.

AUDIENCE: But there are other ways of incurring that information. The advantages of an Ascii representation of files gives you a lot of flexibility for creating those, either [INAUDIBLE] and editing it, both in terms of incorporating it within other programs and in [INAUDIBLE]

HALL: Yeah, okay, that's good info. We're in the process of doing that, of defining that format. What we feel is that there are other things that you may want to do.

And maybe you want to put them in a text format that you can convert into binaries. But at the moment your application runs, you want to have that resource available and its icons, its arbitrary resources. We don't want to restrict it to things that have to be converted to be used. Maybe just short, your talk gets the short end, Jon. Okay, who's next?

AUDIENCE: How much interaction do you have with the people doing the [INAUDIBLE] standards at HP and the [INAUDIBLE]

HALL: The series, the Model 320? That's what our product is released on, is the 320. And we work closely with Fort Collins, which is the source word that does Starbase, for example, which is the graphics package that we have as the proprietary one.

And we're working with them to bring that onto X. And one of the things that we want to do with the X community is figure out, how do you do, extend X 11 and add some standard extensions that do high performance graphics in a heterogeneous way? Yes.

AUDIENCE: How do I get it?

HALL: Excuse me?

AUDIENCE: How do I buy it?

HALL: If you want to order it, it's available February 1. And the, yeah, call your HP salesman. But here's the part number on the excerpt there. Yes.

AUDIENCE: Wasn't one important alternative to [INAUDIBLE] for doing graphics is more a programming language kind of approach, say, I like postscript. But you actually write a program that gets into it. Why did you decide not to go that way rather--

HALL: Well, we--

AUDIENCE: It's the messy part [INAUDIBLE] has nicer packaging.

Jon: The best way to distinguish between Frank and I is that he's worried about the application's user interface. And I'm worried about the system's user interface. And most of the graphics, X technologies that has been coming out has been putting together the program for that.

HALL: Right, I think his question is, if you want to do a, some kind of interactive gadget, should you make a call to the library? Or should you just have a little program that you can send down and be interpreted in some way? That's your question.

And we see that, again, we're addressing current customer needs. And we've found that current customers are programming in a language. They don't want to learn yet another language in order to do user interface.

It's not necessary for them to do that. And when X, the X server can handle and interpret it, when the X server becomes an interpretive server, obviously this thing can be extended, or whatever user interface library gets to make use of that. Yes.

AUDIENCE: With the X library at home, obviously you're taking into account what [INAUDIBLE]. You are putting the public domain, I think, into account expensive clients. I'm thinking particuarly [INAUDIBLE] research machines. Even though we run very fast, we're very expensive. It's very unpopular to interrupt an FFT to do well, for somebody to drag a mouse across the street.

HALL: Interrupting, FFT?

AUDIENCE: Fast Fourier transform which a person paid $20 million to do. Are you, have you examined the issue of minimizing the amount of traffic over network, such as HUTPA, for example, would simply be a key click to start it, a key click to end it, not a drag of the mouse. Menus that do not require constantly clocking up and dragging the mouse around. Because that's all very expensive to implement.

AUDIENCE: --mouse motion events with a passion.

HALL: Yeah, it sounds like that--

AUDIENCE: [INAUDIBLE]

HALL: Okay, that's fine. That's--

AUDIENCE: [INAUDIBLE]

HALL: Right, that's fine. I would say that that's more of a question for the underlying X. What we are trying to do is not revise X. We're trying to use X and put things in there that will make user interface easier on top of that.

AUDIENCE: I guess I was thinking that that can be done at that level. You just have to [INAUDIBLE]

HALL: Okay, and one final comment to that, have we been looking at other machines? We ran our regression suite. By the way, we have a regression suite of tests that we've done for X-ray. And we validated our port to the microVAX in Sun that was very easy.

It was a little harder to get to Sun. Because they're more of a Berkeley system than in System 5. But we were able to get it over and run our verification tests on that before we contributed to MIT. We have not compiled it up on a Cray.

AUDIENCE: [INAUDIBLE]

HALL: But okay, you've got the source. So you can do that. Thank you.

[APPLAUSE]

HALL: They're pretty weird.

Let's Let's get you set up here.

BREWSTER: My name is Jon Brewster. And I'm from Hewlett-Packard also in the Corvallis division. And Frank and I work together as a team. And we've divided the job up between the application user interface and system user interface.

And I've been working on the system user interface problem for about eight years. And I've gotten tired of implementing window managers. So we were pretty delighted when we ran across DECs. And it met our current specifications, met our current needs. And so we started doing ports.

One of the things we discovered, though, is we have the same old problem that we're spending a lot of our time after doing these ports just trying to decide whether or not we can afford to release the product. The bug rates, are they acceptable? How do we know we've tested all the functionality? All of that.

HP has a variety of displays, CPUs, input devices, operating systems. When you multiply all of that together, you have a real testing problem. One of the things I'd like to point out is I changed the title of the talk a little bit. I removed validation and put in verification.

My purpose was really to describe the quality problem and the release problem, the knowing that you have a product which you can sell to your customers and that they get what they want. The validation problem may be defined to be the standardization issue. And that's really a different thought process, and one that I wasn't prepared to talk about.

Speaking of which, however, I just came from the press conference down the street. And I'm pleased to announce to you folks that this is no longer an emerging standard. It is an official windowing standard.

[APPLAUSE]

Very impressive bunch of folks down there.

[INTERPOSING VOICES]

So now that we have a standard, now that we have this standard, you find you've got a lot of interesting problems. These are the problems that we ran up against as implementers of X. We ported the server.

And we were done. And we ran X Demo. And we liked it. But--

[LAUGHTER]

How do we know it's all really there? We also ported the client code. And we said, oh, yeah, this stuff will just run. And it seemed to. But how do we know that that was good stuff?

And as we got through it, in particular at the final stages when 10.4 was available, like one week before product introduction, and we integrated the fixes into there, wow, we were pretty nervous. So these are the problems we run up against constantly and the ones that which we are most interested in solving. The way we've solved it was by applying some information that we've gathered over the last several years.

We've done some test suites on some different window managers. We had learned a few things. We learned that you've got to have a hand-off test regression that you'd walk away from, come back the next day, and find out whether or not your system is ready to release.

You cannot afford to do monkey testing, hours and hours of people just banging on the keyboard. It's not going to cut it. You have to be able to capture a session where the guy has put together a test plan, this is what he's going to test, blah, blah, blah, and away it goes. Now it'll regress.

And it's got to be a fairly nice capture technology. You should not be forced into doing something very artificial to capture the session. You should be able to just use the system.

Otherwise, your testers will rebel. Many times the testing of a user interface has to be done by naive users to really pick up on certain details. You can't always come in here with author test. Sometimes it's non-author tests.

Sometimes you don't have the resources to come up with author test. So your people are likely to be manual writers or other folks that are maybe users of the system but not sophisticated in the test technologies. So a simple session capture has to be possible.

You also want to be able to develop, for those of you that are process oriented, a test plan, and then some test specs, and then actually evolve that into a test script very mechanically, and then just stand back and watch it happen. You don't necessarily want to capture sessions. And you have to modify these things. The comment that was made earlier on the user interface data files is relevant to test scripts.

These things, an eight hour long test script, you do not want to start from scratch to create that hummer. You have got to be able to get in there and diddle it. Okay, so what do you have to do to put together such a system?

You have to have resource locking. You've got to be able to have the test monitor client, as it were, go out and lock the resource to do some things. There are some operations that have to be atomic, collections of operations.

You have to be able to capture images so that you can test to find out what's going on. This is a graphical user interface. And you've got to be able to get back that graphics. And you've got to be able to capture the input for session capture and to be able to synthesize that input.

And we've mentioned at a very low level, the point I wanted to make there is that we want to test all of the server, as much of the server as we possibly can. We've got to get that input down at a very low level so that it can filter back up. And then we can get the most testing.

We found that X did these two already, at least the version 10 version, did the first two things immediately. So we didn't have a problem. But we had to do some modification. It was a fairly small machete. So it wasn't too drastic.

The basic strategy then is that we have a tool that we call tmon for test monitor. And you typically have either a client that you're going to be testing or some strange little test program that's attacking the server. And then you've got the basic server itself.

We chose to go with a server that has the tmon modifications for input synthesis in it all the time, even in the end product, for a couple of reasons. It's good for demos. But we also didn't want to incur the open loop cycle of going in, modifying our server after we'd run our tests, and then shipping that. So the server that we actually ship will be capable of running these regressions.

AUDIENCE: Did you document the tmon interface to be up and working?

BREWSTER: Well--

[LAUGHTER]

Yes and no, our original project plan stated that we would have that interface understood at first release and maybe some tools. We're pleased that we've got actually the tools up and running in the beginning of a suite. And we're now ready to, in fact, this is one of the things we're doing here, is finding out who's interested, what bodies should be involved. And so yeah, we're willing to document it, give it away, extend it into V 11, whatever we want to do.

Basic usage pattern for a user who needs to use the system then, he's got two ways to go about doing his business. Assuming he's got a test plan to work from, whether or not he's got automated tools, he may decide to sit down at the user session, one of the modes of tmon to capture the input into a file. So he'll use the system.

He'll break into a special command mode to do certain image captures, capture the top window, capture the mouse window, capture the whole screen, whatever. There are other extensions possible into the future that we've done on previous systems. You can capture a rectangular area on the screen of any size to cut down on your dataspace size for these tests

Okay, you can also develop a test specification using any random editor that then runs through another utility and is turned into a special kind of test group that has no data. So it's another way to run tmon with the input being a test script, which is the output of this guy. And what happens then is the test script is run and the data is captured.

So now you turn it from this kind of test script file with no data into a test script file with data. Now you've got your database. And you can run test monitor a second time with a different option, which will run that data, database, run that test script, generating a file. I did this, I did that, I did the other thing.

And a failure file, and the failure filed consists of the image that it was supposed to get and the image that it did get. And I think that's about all it does for right now, though there are ways to get out more information like, where is the difference? If you've got a 1K by 1K, couple of 1K images, it's sometimes hard to figure out what went wrong. But there's some extra tools there, could be done.

Another thing that we can do is take the test script, which is the output of the first pass of tmon, and a couple of more utilities to turn it back into a human readable format so tha tthis thing can be edited. So this test script, after it's run through this tran out utility, is then a series of keystrokes, a series of timing information, a series of data sections. And you can go in and extract pieces, merge test scripts, change the timing, do those sorts of things. And then run it through tran in, back into a test script now, that can then be forced back into tmon.

We're not all that pleased with the usability of all of this. But at least it's a start. It's some proof that some tools can exist and they can do some interesting things.

Okay, so in summary of the actual tools then, test monitor is the primary tool that we set out to do. It does image capture. It does the test regression. And it does printing.

These test files are kind of bizarre. They're all compacted, strange sorts of things. Tmon knows how to take those things and print them out to you so they're in nice formats for archival purposes or whatever.

We did some data compression on the images. Because those images get very big, sometimes into the megabytes. And a test suite can take 20, 30, 40 megabytes, depending on what it's testing. We've done some simple run length encoding, which is fine for window tests. But for full screen tests with a lot of gray space, you take it in the teeth.

Okay, the other interesting information here is that we generated a new type of event on the image synthesis. We found that we wanted the client to run in the same environment as he would normally run, the client under test. And we found that if we were capturing every single diddly little event, all mouse movements, all keystrokes, everything, that some of our systems, some of our lower cost systems were very bogged down.

So we went ahead and compressed many of the mouse events into a new event type. And that helped our system throughput quite a bit. There's timing information in there. The timing information can be manipulated generically also.

When you run test monitor, you can say, run it twice as fast as it was originally captured. Or add five seconds to every keystroke or several other kinds of options. So that if you are not compute bound, in other words, you're not trying to get in sync, you can run things faster and get quite a benefit about that.

There are several things we'd like to do with it. We'd like to extend our knowledge of X 11 into this test monitor technology, see what can be done there. Perhaps use some of the X 11 enhancements, extension codes to implement some of the things.

We're not yet sure whether the X 11 needs to be extended, whether the input synthesis that's already there is sufficient, or whether we need to come out at a lower level. Again, the problem there is that we want to test all of the server, not just the client to client path. So we're a little bit rassling with that. And I think we're to wait until we get the X 11 bits from MIT to make that final decision.

Okay, we are willing to make these things available. But we need to really find out whether it's worth it to anybody. It was certainly worth it to us to do it. But we could certainly submit tmon and the other technology back to MIT and place that in the public domain as we did with X-ray.

And we'd be willing to work with the rest of the community to come up with a good suite, a good test suite. It's a massive job. There's a lot of test that could be done. We've just barely scratched the surface. We have maybe 20% of our total tests automated this way at the moment.

It's good. It's great. I mean, it gives you a nice, good feeling to come back the next morning and know that at least your system is 20% there.

[LAUGHTER]

Because a lot of times when you're just monkey testing, you just execute the same old paths through the code time after time. Okay, what we have done to help get this ball rolling, is we've identified an engineer. And his boss agrees that he can do this role. So phone numbers and mail addresses, because I'm difficult to get a hold of through the mail. So any comments, if you want to be involved in generating a test suite, we're going to start, we're going to organize that through George.

Our summary areas that we're very pleased with the tools that we've got. We've found them to be very useful. When we did our MR meeting, when we discussed the output of these tests, which is the thing that we do at HP. We go through an MR meeting and discuss our quality.

We found that not only was it interesting for us as X implementers, but that application vendors ought to be interested in this sort of technology also. Let's say Alice gets their port running. And they, some new whiz bang server comes on the scene. How do we know that Alice can really support that server as a display server?

Well, if they've got a test suite then for themselves, they can just hook up the new server, run the test suite, and they know that they can support that server. So it's a good deal for application vendors also. Great demos, you don't have to have anybody sitting there punching the wrong key or anything like that. You can just stand back.

You've got your option of whether or not to lock the keyboard out while the test is going. So you could have a locked keyboard. Just stand back and watch the menus come and go and the windows slide around.

And in summary, we'd like to keep working on this technology. And we'd love to get some help. Questions? Yes.

AUDIENCE: Two questions. In the demo capability, can you switch between giving control to the terminal and then resuming control back to the test script?

BREWSTER: Question is, when you're doing demos, can you get in and out of this demo mode easily? No, not right now at the moment. It could be done. It's all under control of test tmon. But at the moment, it just runs to completion.

AUDIENCE: The second is if you have any tools that make it easier for you to look at the output of the test run, particularly to debug what was the differences between the expected output and the actual output.

BREWSTER: So are there any tools? And how do you debug your test run? How do you know, how can you deal with that?

One of the things we've got in the human readable formats is we've got comments, which appear in the log file. So I got here, and this happened. I got there, and this happened.

And matching that and the bad images file give you a pretty good feel for where you are and what went wrong. I think there's more work to be done there. But at least we're not totally in the dark going, hey, it just failed.

AUDIENCE: The reason I ask is I consult with a company that built a very similar tool for a form system. And they ended up implementing an editor, which the tester could actually run over through the script, back up through the script. And by the end if they look at that sort of stuff, it turns out, until they built that, they had exactly the same problem. The testers were [INAUDIBLE] because it was so hard to do the [INAUDIBLE].

BREWSTER: So perhaps an interactive tool to allow you to create the thing interactively, not just simple capture and not just simple post-processing, maybe something more in between. That's a good idea. Yes.

AUDIENCE: When you're going from [INAUDIBLE] hardware play [INAUDIBLE] that's going to throw off your data capture.

BREWSTER: Sure, yeah, the problem here is different resolutions and different color depths, it turns out, will throw off your image capture. A lot of times what you have to do on a different screen is run tmon with another mode that captures an original set of images and actually go through them by eye and make sure they're right. Sometimes if the screens are that far off, you have to do that.

The other problem you run into, when we're talking about different platform architectures, is the CPU intensive applications will take more or less time to execute. And you can outrun your test monitor. The test monitor could say, capture the screen. And the application hasn't come up with an answer yet.

Failure, boom, you're dead. You can rerun those kinds of things and diddle those also. So sometimes you can't just stand back and assume it's going to work. You have to go in and diddle it.

We haven't ever found a way out of that. It's not really a tmon problem as much as it is a-- yeah.

AUDIENCE: You had mentioned switching [INAUDIBLE] machines--

BREWSTER: There are some things you can do. There are some things you can do to help it out. For instance, you never read the whole screen. Don't do it, bad idea.

You also run your tests from a dedicated directory with all the default files, font specifications, and everything under your control so that you do window captures with fonts that you have controlled. You do those sorts of things. And you've got a very high, a better chance of success. If you're just doing full screen image captures, then when you go from hardware to hardware, you will have a problem. Yes.

AUDIENCE: You had talked about timing information about 20 push keystrokes and so on. But they keep the more detailed timing of whether the display finished, like how fast [INAUDIBLE] was or whatever. So that if some programmers had a good idea for speeding up the [INAUDIBLE], you can see whether, not only would it work, but whether it made a 10% improvement or not.

BREWSTER: No real performance measuring, we measure the capture sequence. Or you can specify the time in order to do the match. But all it really does is after generating some keystrokes, it will wait a certain amount of time, capture the image.

So it won't find out how fast it took the program to generate that image. There's no real performance monitoring tools. That'd be an interesting study also.

One extra detail on that, if an image match fails, we do assume that there may be a computation problem, wait a few seconds, try again. I think we do that maybe five times so that, to try to help that situation. But, yes.

AUDIENCE: You said that in order to get the test script going, you [INAUDIBLE]. What type of [INAUDIBLE]

BREWSTER: No, we used the standard event header stuff, specified a new event type, and then just used the data specific stuff to cram a bunch of--

AUDIENCE: [INAUDIBLE] is that a notion that's useful in [INAUDIBLE]

BREWSTER: Could well be.

AUDIENCE: And then when you had to take X lib [INAUDIBLE]

BREWSTER: No, actually we didn't do anything with X lib. We generated our own little library that we link in before X lib.

AUDIENCE: Is there a problem with the application that [INAUDIBLE]

BREWSTER: No, let's go back to this diagram. The client under test which may be the application doesn't see compound events. It's only the test monitor that sees compound events. Yes.

AUDIENCE: To extend your problem of timing, information [INAUDIBLE] the complex software systems typically have multiple clients interacting with the screen at once [INAUDIBLE]. Are you anywhere at all in thinking about that? Have you thought about extensions that would allow you to block the application until the application does next ending or next--

BREWSTER: Yeah, we've experimented with that on different windows systems. And there's some work that can be done there. Synchronizing this guy with one application is bad enough. Synchronizing him with a whole lot of applications can get really sticky.

At the moment, what you have to do is you specify a long enough wait time, the application is going to be done. The other thing we have done, we have found is that we run our tests from scripts, which then sleep a jillion in between themselves. And the capture sequence that runs this thing is that it runs this test, the test goes to sleep.

Now tmon comes in, captures the image. After that point, it does something like a control-C or whatever to kill off one test and go into the next. So you can do some cleverness by killing your application from your tmon suite or controlling it via keystrokes from your tmon suite and yet still have these applications unaware of what's going on. So there's some funny business you can do. But we don't have any formalism yet.

AUDIENCE: Did I understand, that was a cooperative application?

BREWSTER: No, it was a cooperative test creation state where the user said, I'm going to set up things with some scripts and some other synchronization. Because most applications have got a pause or a wait or stick around. Or there's a lot of different kinds of things you can take advantage of. And you can synchronize those by encoding into your test suite some of those commands to these applications. Yes.