Ok so I was thinking about this last night and it occurred to me that our problem, really, isn't a "broken module" problem but something else.

Erik's proposed policy has my full support as a stop-gap measure.  But I have some serious long-term concerns if we don't have the deeper issue resolved.

On Wed, Aug 30, 2017 at 11:08 PM, Erik Huelsmann <ehuels@gmail.com> wrote:

As a couple notes, I would add the following expansions to this.

"Allow the widest possible version range on any module, disallowing individual malfunctioning versions to further extend the range"
I think we are better off documenting bugs in misbehaving versions of modules and perhaps offering a module check utility rather than disallowing certain versions of modules.
How do you envisage the disallowing taking place? I was actually not thinking of actively disallowing specific versions, but rather document the fact that the module doesn't work with LedgerSMB by putting a version-exclusion in our cpanfile (the canonical location to declare version dependency).

Right and my point is that this is under inclusive since it is merely an install-time check and moreover lacks any forward knowledge of later possible breakage.  Also I can see possibilities where a package distributed under Debian or Gentoo might have a back ported fix and therefore it might also be over inclusive since there might be patched versions with the same version number which lack the misbehaviour.
 If a patched version retains exactly the same version number, then I'd consider that to be a bug in the patch.
There should be at least a suffix indicating it's patched, unless the patch is simply a change to conform with distro policy, such as a different path, or change in a library name etc.
As Erik says in his response, I don't think we need to concern ourselves too much with dependencies on packaged installs, that's up to the packager (with our support as needed during packaging)

Two points: 
1.  Please correct me if I am wrong but I don't think Debian packages change the version $VERSION when they back port fixes.
That's very well possible, yes.
 
2.  This is a specific problem when installing LedgerSMB from source and packages from the distribution.  I assume we want to support that.
Well, yes and no. Ideally, we don't need to support this combination, because we provide packages (at least for the most popular of distributions). I see installing from source with packages from the dist as a workaround for packages not being available or sufficiently up to date.

But with Debian's support cycle policies I don't see an easy way out of that.  You will always see someone who wants a few latest-generation apps on Debian stable and back ports may or may not be current enough.  I think, long-run, deciding our dependencies based on what versions are in particular distributions is going to be a mess.  Short-run though it may be our best shot.


 
 
Now assuming I am right on point 1, an important question is whether we want to take on the responsibility in the cpanfile for knowing what every distro we support has done in that area.
The answer to this point - from me at least - is a firm No. People can install most packages from their distro, but if some have been blocked by us due to version mismatch, they'll have to accept that if they don't install manually for the remaining packages, `cpanm` is going to upgrade the packages which have mismatched versions. By consequence they'll need something like "local::lib".

The obvious question then is why we accept older versions of PGObject while blacklisting some but not all version numbers of PGObject::Simple.  Obviously we want working software.  It seems the answer is "that's the best we can do right now."  However, such a line makes me uncomfortable because it seems like we are saying that we require that it works with the version on CPAN, but will bend over backwards just a little for the sake of a few popular distributions (I assume we would not do the same if we were talking about a more current version in Debian and an older version in, say, OpenSuSE).

So let's accept that this is the best we can do for now but I want to say I see the situation as posing some problems that I think should be on our radar.  So I would like to see what we can do to sanely remove the blacklists.
 
 
  I would ague that maybe we don't and therefore maybe the cpanfile should presume that patched versions of broken dependencies exist and we should just say "not our problem" aside from user support.
Ah. But the "not our problem" can be taken two directions: over-inclusive at install-time (your solution, as I understand it), which results in non-working installations or under-inclusive at install-time (what I advocate) which results in unnecessary upgrades but is much more likely to result in working installs. My interpretation of "not our problem" is: it's not our problem that packagers don't change their version numbers (be it for good reasons or not) -- which means that we have to reject packages which *might* have been acceptable.

I don't think it is the job of dependency management (In an ideal world) to ensure bug-free dependencies.  I think we should strive for "these are intended to work together" from the cpanfile and leave it at that and put the responsibility for "our software works" somewhere else.

It occurred to me that this has been solved in different ways in different languages.  In C you tend to have pre-compile checks and compile-time checks, something we are I think kind of trying to emulate here.

But in perl, this problem has been solved in Perl by pre-installation integration tests (something problematic for us at the time).  But if it were this easy, I suppose we would probably prefer this instead of blacklisting by version.

So in this particular case I am noticing: I am assuming something more central (like Math::BigFloat) we would assume could be patched in place, that we can test for misbehaviour we can reproduce, and we might not blacklist on version but all our current blacklists are in the PGObject namespace, right?  If so I read the problem not as "we want to blacklist versions" but "this is a workaround because we don't have an easy way to test for these problems."

So maybe blacklisting is a workaround to that problem and something we should do sparingly.  If I could suggest it would be easier to address long-term if we comment in the cpanfile when we do this why we are blacklisting a particular version so maybe later this could be moved to a test case.
 
 
  How the user and the user's source of the package goes about fixing the bug then becomes something we don't have to know how it works or worry about.
Right. We agree there completely. But if we can't distinguish the originally broken version from the patched working version, then only packagers can solve that in the package-creation process -- which brings me to my first point: installing a mix of packages-from-dist and ledgersmb-from-source is a workaround which we should treat as such.

Right.  But I wonder what the workaround is here.  Blacklisting is a workaround for being unable, currently, to easily test for malfunctioning behaviour in a user-friendly way (I.e. not requiring a database connection), right?


Agreed. However, for a packager who starts from scratch, that doesn't matter much: that packager must do all the dependencies as well as the LedgerSMB software itself. So, it doesn't reduce the work he has to do. Moreover, if things work the way they do with Debian, you have more processing time: a package needs to transition into a specific stage before you can build packages that depend on it. Which in this case means you need to build more levels of dependent packages and thus have more moments where you have to wait for the package to transition.

I don't know.  The amount of work to build a first draft from a CPAN distribution at least in Gentoo is minimal.  There are a lot of toolchains to get you started from CPAN to deb or rpm and while they don't get you all the way, they make a difference (and g-cpan for Gentoo takes you 99% of the way there 90% of the time).  I may eventually try to get things in Portage, but gentoo overlays are actually really cool (there are even Travis-CI-based testing frameworks for *repositories*) to the point where I am not sure it is needed to get them on Portage.  And there is an official status I will be pushing for which gets our repository listed in default tools (meaning installing LedgerSMB will eventually be two commands away even without being in Portage).

 
  Pieces of software should have small, well defined responsibilities in my view.  The thing is, indirect dependencies are not our responsibility so pushing more things to indirect dependencies makes our responsibility narrower, better defined, and ultimately easier to keep.

The idea of spinning off the templates is actually an important case in point.

While I agree with the general point of small dependency trees, I like to add a subtle addition to it:

Pieces of software should have small dependency trees *compared to the functionalities they offer*.

The reason I'm saying that is because I think LedgerSMB offers a huge deal of functionalities. By comparison, we can probably slightly reduce the dependency tree, but can't get it much smaller. The tree won't get smaller by separating out; it might not get much bigger either.

This is true.  My sense though is that a narrow/deep development tree provides easier management of problems (because the tree decomposes into smaller trees nicely).  In other words what you have to check at each level is a bit smaller.

As long as we have to maintain these new separated modules ourselves too (as well as our packagers), then there seems to be little benefit to dependency management as a whole. Using modules like Template Toolkit makes a great deal of sense: we outsource maintenance of the software *and* dependency tree to other developers in the community, sharing the development and test efforts. Also, Template Toolkit probably was packaged before Jame ever started his work. So, there too is a benifit.
As long as we're the only user (in Debian) who use our factored out libraries, the burden will fall on Jame, though.

Yeah, which is why I would like to suggest a very slow, careful approach to this so that the burden is minimised. 
 
The specific modules I would suggest we look at in the near future are the templates and the reporting engine (and actually I have immediate uses for these outside of LedgerSMB but inside the LedgerSMB ecosystem).
 
 
Now if we were to have the template system outside, then we have two required modules (the successors to LedgerSMB::Template, probably, the successor to LedgerSMB::Template::HTML) which are now required.   Assuming that this successor has a good plugin framework and we can auto-detect any other template library we put in, we can effectively that the LedgerSMB application no longer has any responsibility to the PDF, OpenOffice, CSV, etc. generation.  If someone wants to generate spreadsheets, they do:

cpan App::LedgerSMB::Template::Plugin::OpenOffice;

But we already do (for a limited set of plugins):


where the syntax of the command to enable PDF support is slightly different.
 
And away they go.  We no longer worry about what dependencies this has.

$ cpanm --with-feature=latex-pdf --installdeps .

does exactly the same. we only specify which modules we use to make it happen. the same modules which would otherwise have been in LedgerSMB::Template::Plugin::PDF. Both would be maintained by us, right?

Right, but there's something I don't like about this.  This works because we know about a module, try to load it, and see if it fails.  There are two problems:

1.  We get a problem report that PDF is not enabled and we don't know if it is because the module wasn't found or that it died on load somehow.  This can make triaging a support issue unnecessarily problematic.

2.  There is no way for someone to add their own formats without modifying the internals of our modules.

Something more like the type handler system in PGObject might be a better choice.  So you load the modules and they register themselves.  These can then be added to a preloads file or something.
 
  It is not LedgerSMB's (the software) responsibility at all.  The template engine would have to have a plugin structure that probably couldn't change much, and an interface for interacting with plugins that could not change.  But this is not actually a big difficulty in that case.
 
For pre packaged modules that's only a problem for the packager, but for git or tgz installs it's a problem for the installer.
The installer of a src version then also needs to download and install many individual modules.
Development installs will become harder, especially if different versions of a module are needed for different branches being worked on.

Finally, it will add additional load to our packagers.
Right now Jame has quite a large number of perl packages that he deals with simply to support LedgerSMB being packaged.
That can lead to delays in packaging a new LedgerSMB release if there are also changes to it's dependencies .
Having even more separate dependencies will only make that worse.

I think this is actually a very valid concern.  There are two very important ways we can help with that:

1.  we don't release a lot of new spin-offs until the old ones are really stable and we are guaranteeing compatibility for a while. If Debian has an old version of PGObject, as long as we support that interface...
2.  we  make sure we have stable interfaces before we spin things off.

I think the only way to spin off is when things are well-designed and stable indeed. Probably by having the to-be-separated-out code in exactly that state in the main code base.

I would actually suggest the only way we will know that is if we spin it off first and synchronise the changes for a while.  It then is not really a bundled version and cannot be used as one.  But it means we get experience testing our assumptions before we say it is ready for packagers to have to worry about because of this decision on our part. 

<snip>
 

--
Best Wishes,
Chris Travers

Efficito:  Hosted Accounting and ERP.  Robust and Flexible.  No vendor lock-in.
http://www.efficito.com/learn_more