Device Test Harness

[gfwilliams]

I've been after doing actual on-device testing for some time - it was the first Patreon goal, but we haven't actually hit that yet. There's another request here: #1207 (comment)

Realistically to avoid false positives you'd want to reset the device after running each test, so my idea is to have a system for Raspberry Pi that anyone could plug a board into and run. It would:

• Scans for new firmware builds on http://www.espruino.com/binaries/travis/
• Downloads a new build and flashes it to the device (obviously devices are all quite different - we'd probably want to use OpenOCD for most of them to avoid the bootloader)
• Runs through a set of tests - resetting the device, uploading the JS, then:
  • Measuring execution time (via an IO output?)
  • Recording any output to the console
  • Recording the status of the test
• We'd have tests for Bluetooth LE and WiFi too I guess (with some other known device set up that could be communicated with)
• Store the results online somewhere (probably a GitHub repo or something each user owns)

Other potential options:

• Allow multiple different boards to be plugged into one Raspberry Pi
• Ability to test the bootloaders from each release
• Test the hardware directly (we could of course connect pins together on the test board - eg. to test Serial RX and TX)
• Test things like the amount of execution stack or JS Vars used during execution.

Then someone also needs to make something to aggregate all this data, graph it, and make it usable.

[MaBecker]

Tried a quick and dirty approache to run some tests for a ESP8266_4MB board.

First used cu -l /dev/cu.usbserial -s 115200 interactive just to try

Connected.
reset();
=undefined

 ____                 _ 
|  __|___ ___ ___ _ _|_|___ ___ 
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 1v99.4083 (c) 2018 G.Williams

Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:1024/1024, manuf 0xe0 chip 0x4016

>~>tests/test_E_getSizeOf.js
1 2 3 4 5 6 7 8 
[file transfer complete]

[connected]
=[ 
  [ 1, 2, 3 ], 
  [ 1, 2, 3 ], 
  [ 1, 2, 3 ], 
   ... 
 ]
>
>var sb = E.getSizeOf(b);
=4
=9
=true
>~.
Disconnected.

Than with scripting and a for loop

1. use socat and tee to redirect output to file
2. setup a for loop with test files
   2.1 send reset();
   2.2. send frame test files
3. analyse output

Some comments:

• lots of data lost during transfer (cat file > serialport) so changed to 9600 baud
• some file are to big to handle like test_262 or test_language.js
• some files contain extra modules that are not included in build
• testfile "test_modifying prototype_after_instantiation.js" contains a blank in name
• some test return true or false and some numbers
• suggest to create a file containing a list of tests that can run on any board and one board specific.

Snippets used to run tests:

• run_tests_v3.sh.txt
• frame.js.txt
• socat /dev/cu.usbserial,raw,echonl - | tee -a result.log
• result.log.txt

Question:

• Is it possible to extend Espruino Command-line Tool to run the test and capture the output?

[wilberforce]

You can specify an input file with the tool - and then just pipe the output to a file.

You can also specify a TCP/IP port instead of using serial - connect to the wifi address via telnet - that way it will be full speed with out any dropping of chars.

[gfwilliams]

It would definitely be possible to tweak the command-line tool - when using -w you get a REPL, but I think when using the normal file upload you don't get much of the output shown at all?

Actually figuring out when to stop listening could be tricky though - maybe there could be a JSON format comment in each test that said how long to wait for, and also if the test was suitable for embedded.

Anyway, using the CLI would be a huge bonus as then serial/telnet/bluetooth/etc could be used depending on the board being tested.

[MaBecker]

Is it possible to define a key word as stop watch indicator?

[gfwilliams]

not sure I understand?

[MaBecker]

console.log( ".stop.watch."); or an escape sequence to tell command-line-tool to stop watching

[MaBecker]

or just add a number to -w <num of sec> to tell how long to watch

[MaBecker]

Got a nice mix of framing tests.js files and use command-line-tool to send them

• branch experimental_compact_vars
• board ESP8266_4MB
• switched build to console speed 9600
• capsulate test.js files and add console.log to the end
• use `espruino -q -p $COMPORT -b 9600 fame.js
• create a file with test that can run on your device
• start your test and lean back
• grep lines with "I name=" and check the results

A sample frame.js

// some test files have result and error and some only result
var error, result;          
var name = "test_immediate_function.js";

// test for defining and executing a function
/*function foo(x) { 
 var arr = ["zero","one","two","three"];
 return arr[x];
}*/
var foo = function() {      
 var arr = ["zero","one","two","three"];
 return function (x) { return arr[x]; }  
}(); 
result = foo(1)=="one";

console.log("I name="+name,"result="+result,"error="+error);

run

for i in $(cat esp8266_test.lst )
do 
        sh run_tests_v4.sh $i
        espruino -q -p $COMPORT -b 9600 frame.js
done

esp8266_test.lst.txt
run_tests_v4.sh.txt
testlog_20190307.lst.txt

Number of tests: 139
SyntaxError: 6 testlog_20190307.lst.SyntaxError.txt
Result false: 2 testlog_20190307.lst.false.txt
Result reue: 131 testlog_20190307.lst.true.txt

[gfwilliams]

Ahh, ok. The issue is that when the tests are running in the Linux build, it knows when everything is finished by checking if any timers are active and can finish automatically.

Adding something like:

var t=setTimeout(function cb() {
  clearTimeout(t);
  print(global["\xff"].timers);
  if (!global["\xff"].timers.length)
   console.log("DONE");
  else t=setTimeout(cb,50);
}, 50);

at the end of the code you upload should fix it - although honestly a timeout of 1s is probably good enough? I actually added a --sleep command to the CLI a few days ago (it's not published on NPM yet.
but is on https://github.com/espruino/EspruinoTools#command-line) but actually the minimum wait there is 1 second.

It's great that you got the tests going though.

[MaBecker]

test_digitalwrite_words.js only works for board that the have those pin names

Have you thought about adding some device specific tests

• create folder for boards specific stuff
  or
• add board name / all to testfile
  or
• add file with general test and for each board a file with specific tests
  or
  ......

[MaBecker]

Have you thought about running on board tests via make?

[gfwilliams]

Yes, device specific tests would be good. However first I'd be happy with a system for running general tests on the device and sending back the result :) In an ideal world we'd have a setup of maybe an I2C, SPI and OneWire RAM device on every single board, and would then have code that'd test they all worked correctly.

With testing - sure, it could be a make test command when it's done, but I imagine all that's actually going to end up doing is just calling out to some Python/Node.js code that actually does the test itself?

[MaBecker]

However first I'd be happy with a system for running general tests on the device and sending back the result :)

This is what I have done now with shell scripting just added a new finding to #1631

The whole output is redirected to a result.log file to analyse afer a run.

Eg. Output for running testfile test_abstract_comparison.js:

Explicit board JSON supplied: "/Espruino/repos/Espruino/boards/ESP8266_4MB.json"

 ____                 _ 
|  __|___ ___ ___ _ _|_|___ ___ 
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 2v01.65 (c) 2018 G.Williams

Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:1024/1024, manuf 0x1c chip 0x3016

>
Upload Complete
I name=test_abstract_comparison.js result=true error=undefined

And then grep SyntaxError ReferenceError and flavors of result=

[MaBecker]

Number of test files 300 /Espruino/repos/testing/esp8266_test_v2.lst
Number of syntax errors 6
Number of reference errors 21
Number of result = true 208
Number of result = false 6
Number of result = 1 68
Number of result = 0 5

[MaBecker]

a setup of maybe an I2C, SPI and OneWire RAM device on every single board, and would then have code that'd test they all worked correctly.

Like to add some neopixels too.

[MaBecker]

Ok, fine with bash scripting stuff for now.

I guess extending espruino command line tool would be the most efficient way.

• take over testing by passing a filen containing the test cases to run
• adding information to test to identify the test name and the result (eg. frame.js.txt)
• passing the plain output to a result file.

[MaBecker]

Got grafana and influxdb running localy and want to import a test_run_resultfile.txt

What do you think about this table columns to store test results?

EspruinoVersion BoardName NameOfTestfile Result

Result: error, true, false

eg

2v04  PICO  test_262.js error 

Not sure about a timestamp or a RunCounter or what is helpful to be added as column.

[gfwilliams]

When I was doing it I stored a log - but that could just be the contents of error. Also, runtime could be handy - it might provide a useful way of benchmarking between commits

[MaBecker]

Runtime is a good point.

insert one sample row for measurement dth in database Espruino

> use "Espruino"
Using database Espruino

> insert dth,git=0b24da150,ver=2v05,board=PICO,case=test_E_getSizeOf.js rt=500i,rc=false,se=false,s=false

> show measurements
name: measurements
name
----
dth

> show series
key
---
dth,board=PICO,case=test_E_getSizeOf.js,git=0b24da150,ver=2v05

> select * from dth
name: dth
time                board case                git       rc    rt  s     se    ver
----                ----- ----                ---       --    --  -     --    ---
1584740294163060000 PICO  test_E_getSizeOf.js 0b24da150 false 500 false false 2v05

git: short HEAD as string
case: test case file name as string
rt: runtime in ms as integer
rc: return code as boolean
se: syntax error as boolean
s: skipped, because not working for this board as boolean

[MaBecker]

So this is going to happen - great!

Just some question about the files in folder tests:

• Are you still planning to run subset of test cases for different board?
• What about modules?
• What about
  • files names like test_generic.lst and test `test_<board>.lst
  • or comments in test files to separate between generic, <boards> or what ever

I am planing to update the ESP8266 builds to SDK version 3 and will try to run scripts build three years ago.

[MaBecker]

Hmm, after checking the documents there is no big win having version 3.

[gfwilliams]

The tests folder is really just for the linux builds. I think initially there'll be a very small subset executed on devices - realistically if those tests work on desktop there's a super high likelihood they'll work fine on the device. I wasn't planning on adding tests for modules, as most of them require external hardware to be added to work.

Really I'd be looking at adding new tests - initially probably just benchmarking (and maybe checking memory/stack usage). I think if those work we could be reasonably happy with the interpreter.

Obviously hardware testing needs a bit of thought too though - for example Wifi on ESP8266 would need to be able to connect to something else, or Bluetooth on nRF52/ESP32 really needs some kind of companion test that would run on the Raspberry Pi.

[MaBecker]

Ok, understand, first check the linux build and then second the other devices.

Not sure if you remember that here are some test for the ESP8266 here we could use for the wifi testing.