A while back we looked at the automated testing of ⎕WC based GUI applications and some of the gnarly challenges involved. Automated testing of Abacus applications and the HTMLRenderer presents similar challenges. Some of the issues were addressed in a previous post on the Abacus threading model. Now that we have a sample Abacus application it's time to see what actual tests look like.

It is imperative that an HTMLRenderer testing framework can work in one APL process, all in APL, that the tests can be executed automatically, and that they can be traced. (With asynchronous events, it's easy to get into a situation where things work when running a function, but not when tracing, and vice versa). We must be able to sit in the APL session, write tests, write code, run tests, and write more tests and code seamlessly.

Testing in Abacus builds on the Provanto testing framework, and Abacus tests and test suites should be structured in the same way they are for Provanto. The Abacus RunTests function is a cover for the Provanto Run function:

RunTests←{
     ⍝ ⍺ ←→ ¯1 ←→ Setup only
     ⍝ ⍺ ←→ [Stop 0|1 [Quiet 0|1|2]]
     ⍝ ⍵ ←→ Test space, [Code space]
     ⍝ ← ←→ Result space
     ⍺←0
     s←⊃⍵
     d←s.Start 0
     _←s InitTestSuite d
     _←WaitForApp d.Client
     ⍺≡¯1:0
     ⍺ #.Provanto.Main.Run&⍵
 }

This function starts the Abacus application, intitializes it along with the test namespace for testing purposes, and waits for the application to complete its start up. This last is critical as before the tests can run, the HTMLRenderer OnWebSocketUpgrade event must fire and its callback must complete. Finally the test namespace is handed off to Provanto to execute in a new thread. Abacus tests must run in a thread other than 0.

The Abacus testing framework enhances the Provanto testing framework, injecting additional functions into the test namespace including Click, Press, Enter and GetElementById.

While Provanto allows for a user supplied Startup and Teardown function to run before and after a test suite, the Abacus framework requires an additional function Start, which should build the application, create the HTMLRenderer, and return the application document. The Startup and Teardown functions may still be used for additional functionality, but note they will run in the test thread, while Start executes in thread 0.

Note that a left argument of ¯1 will start the application and initialize the test suite, but not run the tests. This is handy when you want to work on and run one test at a time. When running an individual test it must be threaded, or else it will hang due to thread lock. You can avoid accidentally doing this by including this line at the start of your test function:

      0=⎕TID:∇&0

When testing an Abacus application, we want to dispatch events on the browser under program control in APL, then examine the APLDOM and the state of the application, as well as the the state of the browser, and make assertions. Let's look now at a sample test:

TestScrollingAroundBattingFile←{
     Click'File':
     Click'Open':
     Enter GetFileName'Batting.csv':
     Click'OK':
     g←GetElementById'CSVdata'
     Assert 22=≢g.Values:
     Assert 107429=≢⊃g.Values:
     Assert g.DataCell≡0 0:
     Press'ArrowRight':
     Assert g.DataCell=0 1:
     Press'ArrowLeft':
     Assert g.DataCell=0 0:
     Press'End':
     Assert g.DataCell=0 21:
     Press'Home':
     Assert g.DataCell=0 0:
     Press'Ctrl' 'End':
     Assert g.DataCell=107428 21:
     Press'Ctrl' 'Home':
     Assert g.DataCell=0 0:
     0
 }

Note that the naked guard technique is used not just for assertions, but for the Abacus-specific functions Click, Press, and Enter, which have no use for a result. Note also that there is no reference to the application document. The test framework establishes a global reference in the test namespace named Document.

The Click function takes an element id as its right argument, with the global Document as an implicit argument.

The Enter function takes a string to enter into an <input> or <text-area> element as its right argument. The left argument defaults to the input element of the PromptBox component if one is open.

The GetElementById function returns an element, and in addition establishes a global reference named Element. Subsequent calls to functions that require an element reference that is not explicitly provided will use this value. The element returned is an element in the APLDOM, not the browser. In this case it is instance of the Abacus DataGrid component, and thus has a Values and DataCell property that we can inspect.

The Press function fires a keystroke on the element provided as a left argument, or the last element spcified by GetElementById.

The net result of having the globals Document and Element is a very clean test function with a minimum of fuss. Any framework function that requires a DOM element to run will use the global Element if an element is not explicitly provided.

The above test function fires events and inspects the APL DOM, or the state of the application in APL. It is useful to also be able to make assertions about the state of the browser as well. This can be done using the GetElementFromBrowser function which takes an element id as its right argument, and returns the element from the browser. For example:

 TestOpenBattingFile←{
     Click'File':
     Click'Open':
     Enter GetFileName'Batting.csv':
     Click'OK':
     sg←GetElementById'CSVdata'
     Assert 22=≢sg.Values:
     Assert 107429=≢0⊃sg.Values:
     cg←GetElementFromBrowser'CSVdata'
     v←A.BodyValues cg
     Assert'abercda01' '1871'≡v[0;1 2]:
     Press'ArrowDown':
     Press'ArrowDown':
     Press'End':
     Assert sg.DataCell≡2 21:
     cg←GetElementFromBrowser'CSVdata'
     v←A.BodyValues cg
     Assert(2⊃⊢/v)≡,'1':
     0
 }

Here in addition to inspecting the APL DOM element CSVdata (in var sg), we look at the corresponding browser-side element (cg). Note that sg is a live reference - it only needs to be extracted once and we can test the same instance as the application state changes. On the other hand, cg is static as it comes over the websocket as text, and is then deserialized into an object. It must be repeatedly retrieved to see changes in the browser.

Note also that in addition to the few functions injected into the test namespace for convenience like Click and Press, the entire Abacus library is available via the reference A.