Hi,
Last year, I started experimenting with Dancer2 (one of the options on the
table for our target framework when we rewrite much of our code).
Today, I've been updating that code to the HEAD of the current master
branch. During the work I've done so far (last year and today), I've come
across several obstacles which now lead me to bring up the following for
discussion.
So, let me start by stating which problem I think we're looking to solve:
we need a means to do request parameter handling, session management,
pluggable authentication, routing, management, template handling and
abstraction of a templating library.
During my experiments with Dancer2, I found that indeed it provides a
framework with the desired properties. However, as with other frameworks,
Dancer2 imposes its "framework structure" on applications. Normally this
helps simplify the task of application development. Yet with the
peculiarities in our code base, I'm finding that to implement required
functionalities, I'm already working against the framework rather than with
it:
1. Session management (cookies)
As part of the move from our current design to the Dancer2 framework,
I'm aiming for setup.pl and login.pl to become completely separate
applications, meaning that they require separate authentication cookies (to
allow separate, concurrent logins into both apps from a single browser)
While Dancer2 does support multiple apps running on a single instance
of its framework (such as required to run login.pl and setup.pl from the
same backend), it doesn't support - in its configuration file (config.yml)
- that these apps use disjoint session management cookies. In order to
support this use-case, I've had to add request hooks, changing the name of
the session cookie on a per-request basis.
2.Pluggable authentication
As LedgerSMB uses a rather unusual authentication scheme where the
user+password+database name are required for login, our authentication
scheme doesn't fit the standard set by the framework
(Dancer2::Plugin::Auth::Extensible::*), I've ended up hacking "deep" into
the internals of auth::extensible: our authentication couldn't be achieved
without replacing the login templates and finding ways to make more
parameters available than supported out of the box.
3. Templating library abstraction
While abstraction from templating libraries sounds nice, in practice
it's cumbersome to switch templating libraries: on the Perl side the
abstraction works fine to swap from one to another, but templating
libraries usually use different templating "languages" (i.e.
TemplateToolkit uses different directives and paradigms than XSLate). So,
it seems that managing instantiation of the templating library is more
important than abstracting it.
4. Route definition & dispatch
With the ability to define routes, we'll be able to detach URL
structure from our internal code structure. Dancer2 has a nice DSL to
support the definition of routes for non-webservice entry points. There are
plugins available to extend the DSL with convenience keywords for
definition of webservice entry points.
However, the syntactic sugar provided by the endorsed plugin
(Dancer2::Plugin::REST) provides 1 new DSL keyword which is nothing more
than a way to attach 4 functions to a resource (create, get, update,
delete) providing a generated route for each; I see very little benefit to
this plugin. In addition, this plugin doesn't support HTTP's content
negotiation; instead, it expects the requested resource to have an
"extension" to indicate the desired response encoding (e.g.: /users/1.json
returns JSON, /users/1.xml returns XML, etc). This isn't "http-native"
enough for me and thereby unacceptable for our use. Then there's the (not
endorsed) Dancer2::Plugin::WebService; however, this too encodes the
desired response encoding in the URL...
5. Framework configuration
Dancer2 supports running multiple "apps" on a single instance. We want
to run multiple apps: our setup.pl and login.pl. Even: we want these
multiple apps to run on different authentication domains (i.e. when a user
is logged in in /login.pl, we don't automatically want the user to be
logged in into /setup.pl too). During my experiments I found that this
isn't something that Dancer2 supports in its configuration: it can be
"tricked" to support it by installing the correct pre- and post-request
hooks in the framework, but out-of-the-box, it's not there. I'm not sure
what other areas this will affect in the future, but from the structure of
the configuration, I'm imagining it'll be an issue in many places.
So what is our way forward then? Well, while we were working to merge the
Multi Currency branch and stabilizing the 1.7 release (this process is
still on-going), I've been looking around for solutions to these issues and
I think the Perl library ecosystem has very nice options for almost all of
the problems mentioned above, with the remark that there will obviously not
be such a nice DSL as there will be with Dancer2. The following
alternatives are available:
1. For session management using encrypted cookies, there's
Session::Storage::Secure which works and is agnostic of underlying
technology
2. For pluggable authentication, as we need to write our own authenticator
anyway (and we actually already *have* that as a Plack middleware), I
there's no effort if we can reuse that
3. For templating engine abstraction, well, I guess that with my line of
reasoning above, there's little reason to have abstraction for templating
engines at all (I mean, there *is* for a framework which needs to support
many code bases, but there isn't for our specific use-case: our code base)
4. Route definition and dispatch (including extraction of route parameters)
is available in a technology agnostic library Router::XS; while this
library doesn't have the solid DSL that Dancer2 has, I'm expecting us to
move more and more to Webservice entrypoints (either JSONRPC, GraphQL or
REST) meaning that the DSL Dancer2 provides will be far less useful as
stated above
5. Framework configuration could be handled through one of the regular YAML
libraries, but even better - especially with inversion of control /
dependency injection - we could turn to Beam::Wire and be able to deal with
the most complex configurations by deferring configuration handling to
sysadmins
Obviously, Dancer2 will be able to do request parameter handling. That's
something that's supported today too: we already have Plack::Request to
handle parameter parsing (both URL and body parameters).
So, after this long story, I'm wondering if moving to Dancer2 is worth it,
or that we can simply stay on Plack with its middlewares and put the
aforementioned modules to good use. Lets have your opinions and a good
discussion on the matter so we lay a strong foundation for next steps for
our project.
Regards,
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.
Hi all,
Roughly 4,5 years after the first coverage test run (12 September 2014) and
little over 3 years after the first BDD test runs with coverage testing, on
April 12th, we have finally reached the 40% coverage milestone
demonstrating the effectiveness of the BDD tests to increase coverage.
To get to this point, we contributed to or even created various other Perl
project:
* Test::BDD::Cucumber (bugfixes, plugin-capability, "prove" integration)
* Weasel (Web-app and Single Page App testing)
* Pherkin::Extension::Weasel (Weasel integration with Test::BDD::Cucumber)
The most recent addition here - added in March - was the ability to
generate an HTML log with screenshots of the web session; for an example of
such logs, including screenshots of the application after execution of the
various test steps, see
https://132-18471236-gh.circle-artifacts.com/0/tmp/artifact/logs/index.html
All of these contributions helped to accelerate the release schedule of
LedgerSMB with fewer reported new issues on each release.
With the size of our current test suite, we were running into limitations
of Travis CI, with our test suite now approaching maximum run time for a
test job. Over the past months, we've therefore been working to move part
of our test suite to CircleCI. As part of this migration, we've been
confronted with many timing issues which made our test suite produce more
than acceptable false-negatives (but never false positives, fortunately).
We've addressed all known instability issues now and moved testing of our
master branch to run both on TravisCI and CircleCI. Our test coverage
measurements (the slowest part of our tests) is produced on CircleCI and
published on Coveralls.io (
https://coveralls.io/github/ledgersmb/LedgerSMB?branch=master).
With the improved testing in place, I've been able to clear my backlog in
old and pending changes. In the immediate future, new releases of
Pherkin::Extension::Weasel and various Weasel::* modules are to be expected
-- improvements to further help developer efficiency.
Apart from work on expanding the test suite itself, Yves and I are likely
to start moving our test suite from 'prove' to 'yath' (
https://metacpan.org/pod/distribution/Test2-Harness/scripts/yath), because
it deals with UTF-8 from the start, where we're currently seeing warnings
from 'prove' relating to UTF-8.
Regards,
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.
Hi Folks,
I'm a LedgerSMB noobie. I have installed LedgerSMB and Postgres in
Docker successfully (using docker-compose with modified yml file) with
port 5432 open to the db so I can address the Db directly. The install
works as expected (just had to set users passwords twice).
What I would like help with is either some code to migrate from Sage to
LedgerSMB or just some code or point me to some docs to get me started,
please - I'm fluent in Java, some C++ and its been about 4 years since I
seriously coded in Perl.
--
*Regards,*
*Andre Steenkamp*
Primal Factors Consulting
MBA International Business
Bsc. Computer Science / Engineering
SA Mobile: +27828781465
USA Mobile: +12627325285
Skype: andre.steenkamp
Hi,
After the release of 1.7.0-alpha1 we have been working towards the 1.7.0
release as described in our community update. We feel that the 1.7.0 branch
is feature complete and while we're developing tests to find and fix
possible regressions and older bugs, no new features or major changes are
expected to be implemented before 1.7.0.
This means that the current code base (modulo any and all regressions we
can find and fix) will become 1.7.0 when it's ready. To help you find the
bugs we need to fix before the release, we have published 1.7.0-beta1. It's
available as a Docker image from Docker Hub under the tag
"ledgersmb/ledgersmb:1.7.0-beta1". See
https://hub.docker.com/r/ledgersmb/ledgersmb/builds/
Additionally, the release can be downloaded as a tarball from the download
site:
https://download.ledgersmb.org/f/Beta%20Releases/1.7.0-beta1/
Please report any issues you find in reply to this e-mail (that is, to the
mailing list) or create an issue on GitHub with your finding:
https://github.com/ledgersmb/LedgerSMB/issues/new?template=ISSUE_TEMPLATE.md
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.
Hi,
It was just before New Year when I sent out the last community update, so
it's time for a follow up.
This half year, we only released 1 maintenance release for 1.6.10. With no
specific new bug reports coming in, the development team was able to focus
its efforts on 1.7.0.
On that front, a lot of development effort has gone into getting the
Multi-Currency branch ready to be merged to master. On January 20th the
long-awaited multi-currency branch merge happened, followed by the release
of 1.7.0-alpha1.
Since then, we've been developing more tests simulating user interaction
with LedgerSMB to ensure that 1.7.0 will be as good as 1.6.0 when it is
released. In the process, we have been fixing a few regressions. With the
additional tests - which include tests for AR & AP invoices and
transactions - we have increased test coverage from 29% (current on 1.6) to
40% (on master) and we're working to increase the number to well above 50%
before release.
The speed at which we're able to implement new tests is severely impacted
by the fact that we hit a continuous integration limit: Travis CI allows us
to run tests for a maximum of 50 minutes. Our test suite runs between 40
and 45 minutes in code-coverage testing mode.
A lot of effort has gone into working on this problem. We're working to
off-load the code-coverage test jobs to another Continuous Integration
service (CircleCI) which *does* allow jobs to run longer than 50 minutes.
Another direction of research was to reduce execution time of our test
jobs. And while that did turn up some bugs in our Selenium test driver, it
mainly helped the area where we didn't experience any problems (the
non-code-coverage tests were reduced from 29-30 minutes of execution to
20-21 minutes of execution).
Migrating some of our test infrastructure to CircleCI turned out to have
the additional benefit of being able to start running our tests against
Chrome and Firefox. Currently we only run against PhantomJS. While
PhantomJS implements enough of JavaScript to support our testing, it's
unmaintained and getting behind on recent standards. Being able to test
with Chrome and Firefox will help to take advantage of more modern browser
development techniques.
As part of our optimization work, we helped create new releases of Weasel,
Pherkin::Extension::Weasel and Test::BDD::Cucumber, where significant new
features were added to Pherkin::Extension::Weasel to support increased
developer efficiency.
Looking to the immediate future, we're expecting to release 1.7.0-beta1
this week: we think 1.7.0 is feature complete and we'll work simply to
stabilize it from here on. This means that 1.7.0 isn't production ready,
but we're looking for people to test-drive it and value any and all bug
reports we can get!
Regards,
Erik.
--
Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP.
Robust and Flexible. No vendor lock-in.
