Includes Pentarctagon's improvements:
* Adds the new `bt` (batch timeout) option, to limit the maximum time batched
unit tests can take. Otherwise, if for whatever reason they take longer than
10 minutes, travis will error the job due to not receiving any output for too
long.
* Writes the test output to files rather than trying to get it from the stderr,
and then output's the log from the failed test. Currently if a test
unexpectedly times out, there is nothing printed indicating the error or
which test encountered the problem (in case of batched tests).
Cherry picked from commits
* eb7f3674dc
* 38a49ad068
* 7345ca7b0f
Updates the script for the new statuses cherry-picked from
* 7359fddecb
The check to see whether the timeout command is capable of sending
SIGKILL was broken. This fixes the check, which in turn will ensure
that tests are always terminated. This will resolve travis deciding
that the entire build has timed out.
This explains (in the log) why a WML conditional has failed (or succeeded). Currently it is fairly basic, looping through and/or/not and printing out the specific conditional tag that failed as well as, for [variable], the current content of the variable.
It's used in the WML unit tests system, but could also be useful for debugging.
This commit also fixes an issue with passing multiple -a arguments to run_wml_tests.
- If timeout is missing, it disables timeout and skips any tests that expect it
- Pass -t0 to get the above behaviour even if timeout is present
- Detect if timeout doesn't support --kill-after
- Mac users: Pass -pxcode to automatically find the Wesnoth binary in XCode's DerivedProducts folder
- In non-verbose mode, give progress report by printing a dot before each test
This only happens in non-verbose mode because this method of reporting progress makes most sense
if nothing else is printed.
This was buggy and based on deprecated SDL features that will not
continue to be available. aquileia wrote us a very nice windows
port of the run_wml_tests bash script, so for windows mac and
linux we should now be able to run the unit tests as desired.
Maybe someday built-in timeouts can be supported using boost
process library, if that ever materializes.
- travis runs tests not in verbose mode,
*but* if there is a failure,
then it runs in very verbose mode, with strict mode off, to get
errors reported in the travis log.
- run_wml_tests points out that if you don't run in strict mode,
some tests may not fail as expected, so the count at the end
may differ
Prev version could sometimes give an error if test timed out and
error.log was not created, so add a check for if error.log exists.
Also refactor so the error log handling is in its own function.
Only start using --validcache with wesnoth after at least one test
has passed.
Use --kill-after flag of unix timeout, so that if TERM fails to
stop wesnoth, a KILL will be issued.
Fixup comments.
Adds an option to the new wml test script, to use the unix
timeout command instead of wesnoth's built-in one. This is because
the built-in one is very slightly unreliable... there's a reason
killthread was deprecated in SDL unfortunately. Will leave it for
UMC but on travis will replace with unix timeout.
The unix timeout one seems consistent enough that I will enable
the empty_test timeout check for travis.
Also, adds a help listing to the wml test script.
Also, redirect some output from cout to cerr in the game...
this somewhat breaks the old script.
The old script is renamed .old
Travis points to the new script.
