GSoC final report - UI logger

Overview

The idea of this GSoC project was to implement a feature which logs the interaction of users with the UI elements in an interpretable Domain Specific Language (DSL), and an interpreter which converts the log into a file into something which will help in replaying the user actions. The project involved working with the UI elements and the UI testing framework of LibreOffice.

Achievements

• The logger -
  
  The logging for the following is currently supported -
  1. The UI elements of Visual Component Library like Buttons, Checkboxes, Radio-buttons, Listboxes, Comboboxes, Spinbuttons, etc.
  2. Keys pressed in different UI elements
  3. UNO commands sent with parameters
  4. Selection of Objects
  5. Opening of Sidebar/Deck
  
  Some application-specific events can also be logged -
  1. Writer - Goto Page, Set Zoom, Text Selection
  2. Calc - Autofilter Launch, Selection of cells/range of cells, Switching of Sheet
  3. Math - Selection of elements from the element selector panel
  4. Impress - Set Zoom
  
  Additionally, the opening and closing of some dialogs is also logged according to the DSL.


• The UI Testing Framework -
  
  We already had a framework for UI testing before the start of the project. Efforts were made to ensure that we can utilize this for the replay of actions to the greatest extent possible. The wrapper classes for a couple of VCL classes were completed. A function to execute UNO commands with parameters was also implemented.


• The log interpreter -
  
  The logger logs events in the following grammar -
  
  

  <Keyword> <EventDescription> [<Parameters>]
  

  
  where EventDescription tells the ID, parent, etc. of the UIObject and Parameters are the parameters of the action user performs in a format directly readable by mkPropertyValues.
  The interpreter starts reading the log file line-by-line and tries to convert the statements into a UI test case. The code for the interpreter can be found in core/uitest/loginterpreter.py

Merged Patches

1. uitest logger: support for buttons, dialog, command and key logging
2. uitest logger: ListboxSelect and ListboxFocus logging
3. WIP: uitest: complete combobox, tabcontrol wrapper
4. uitest: interpreter for log files
5. uitest logger: print parent when logging keys
6. uitest logger: some changes in log statements
7. uitest interpreter: add more features
8. uitest interpreter: let's use argparse to parse arguments
9. uitest interpreter: add support for starting apps
10. uitest interpreter: handle when parent is empty string
11. uitest interpreter: support opening existing document
12. uitest logger: log the events calling CallEventListeners()
13. uitest: complete wrapper for SpinField and SpinButton
14. uitest logger: log key according to DSL inside dialogs too
15. uitest: support parameters when sending UNO commands
16. uitest logger: log more events

The list of all my merged patches can be found with this Gerrit search.

Using the logger

As written in the UI Test wiki, the following needs to be done to log the UI actions -

1. Launch LibreOffice like

LO_COLLECT_UIINFO="test.log" instdir/program/soffice

2. Simulate what you want to do with the mouse
3. Close LibreOffice
4. Open the resulting file in instdir/uitest/test.log

The following command can be used to convert to log to UI Test Case:

python3 <path_to_log_interpreter> <path_to_log_file> <path_to_a_new_python_file>

where <path_to_log_interpreter> should be replaced with something like SourceDirectory/core/uitest/loginterpreter.py
and <path_to_log_file> should be replaced with something like SourceDirectory/core/instdir/uitest/test.log
and <path_to_a_new_python_file> can be a location of your choice where you would like to see the generated code.

Remaining Tasks

1. Not all UI elements and dialogs are supported by the interpreter/logger/UI testing framework. Eg. - Color selection palette in color Calc -> Format cell.
2. Enhancement suggestion - After this commit, it will be helpful for the interpreter to locate a UI Object if the logger prints the entire parent hierarchy of the UI object instead of just printing the top window. This would help in resolving UI ID conflicts, as well as in other cases when the interpreter might not locate the object properly otherwise.

Thank You!

Thank you for giving me the opportunity to work on this project. I learned a lot from this project. I will definitely like to spend more time with LO community in future whenever I get time.

PS: The links given to the patches in the post contain a commit message to explain what the patch does. An additional effort to document the UI Test logger can be found at An overview of how UI logging works in LibreOffice.