Hi all,
Roughly 4,5 years after the first coverage test run (12 September 2014) and
little over 3 years after the first BDD test runs with coverage testing, on
April 12th, we have finally reached the 40% coverage milestone
demonstrating the effectiveness of the BDD tests to increase coverage.
To get to this point, we contributed to or even created various other Perl
project:
* Test::BDD::Cucumber (bugfixes, plugin-capability, "prove" integration)
* Weasel (Web-app and Single Page App testing)
* Pherkin::Extension::Weasel (Weasel integration with Test::BDD::Cucumber)
The most recent addition here - added in March - was the ability to
generate an HTML log with screenshots of the web session; for an example of
such logs, including screenshots of the application after execution of the
various test steps, see
https://132-18471236-gh.circle-artifacts.com/0/tmp/artifact/logs/index.html
All of these contributions helped to accelerate the release schedule of
LedgerSMB with fewer reported new issues on each release.
With the size of our current test suite, we were running into limitations
of Travis CI, with our test suite now approaching maximum run time for a
test job. Over the past months, we've therefore been working to move part
of our test suite to CircleCI. As part of this migration, we've been
confronted with many timing issues which made our test suite produce more
than acceptable false-negatives (but never false positives, fortunately).
We've addressed all known instability issues now and moved testing of our
master branch to run both on TravisCI and CircleCI. Our test coverage
measurements (the slowest part of our tests) is produced on CircleCI and
published on Coveralls.io (
https://coveralls.io/github/ledgersmb/LedgerSMB?branch=master).
With the improved testing in place, I've been able to clear my backlog in
old and pending changes. In the immediate future, new releases of
Pherkin::Extension::Weasel and various Weasel::* modules are to be expected
-- improvements to further help developer efficiency.
Apart from work on expanding the test suite itself, Yves and I are likely
to start moving our test suite from 'prove' to 'yath' (
https://metacpan.org/pod/distribution/Test2-Harness/scripts/yath), because
it deals with UTF-8 from the start, where we're currently seeing warnings
from 'prove' relating to UTF-8.
Regards,
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.