[SOLVED!] €500 bounty: autoloaders from different packages being ignored

Development
1

In the past month or two I’ve repeatedly seen issues where MODX extras don’t work. They throw a fatal error trying to load a class that is supposed to be available through their autoloader, which exists and is readable.

Having tried to debug several instances of them, it seems to be happening only when multiple composer autoloaders are all loaded simultaneously (i.e. different extras providing their own autoloaders, which may define dependencies and/or PSR0/PSR4 namespace roots). Somewhere along the line, it seems to break the chain and no longer reaches the namespace declaration and just stops processing.

(Wether it’s a good idea to have different extras define their own autoloader… probably not. But we don’t currently have a better way and this has worked fine for years.)

I just don’t have the faintest idea why this is happening. It’s not a dependency conflict as far as I can tell; these are widely different extras with different dependencies and the problem appears trying to load unique classes in their own namespace.

The earliest report I’ve seen is FormIt, from 2018, but I’ve now also seen it in sites with Commerce (and various extensions), Redactor, and MoreGallery. The specific class it fails on is typically the first one being used in a service class constructor, or the service class itself.

It may be a coincidence there’s a bunch of modmore extras in this list, simply because we do use a lot of dependencies and namespaces in modmore extras and modmore users know they can reach out for help with those extras, or perhaps there’s something we’re doing that’s triggering this problem. I do know the issue with FormIt was reported in 2018 but I started seeing this a lot more often in the last two months.

I’ve found that optimizing the autoloader can help. E.g., when creating the package we use the command composer install --prefer-dist --no-dev --no-progress --optimize-autoloader, or running that in the core/components/<package>/ folder of an extra that’s breaking fixes it for that extra. But it doesn’t avoid other extras breaking that do not have an optimized autoloader, so it seems that this is a workaround and not the actual root problem/solution.

I’m out of ideas on where to look and really want to know what’s going on here.

So, if anyone has any other ideas on how to tackle this, I’m offering a €300 bounty for the person to identify the problem or find a fix. If someone doesn’t identify the problem or find the fix, but still provide the hint/clue that leads into the rabbit hole where the problem/fix is located, half that amount is theirs.

–

Happy to provide more information anyone thinks may help, but I do want to note it’s been happening with different combinations of extras on different servers running different versions of MODX and PHP (modmore extras require at least PHP 7.1).

Composer version used to build modmore extras is 1.10.10 on PHP 7.4. These composer/php versions have been recently updated but if memory serves me right (often it doesn’t) that was after the first time someone reported an issue like this, and I’m also not yet convinced this is modmore-specific as it first happened with formit… but can’t rule it out either.

–

Updated Sept 28: I’m going to try and keep a list of sites that are affected and their PHP version and installed extras, to hopefully figure out the combination or eliminate specific extras. Only one site per business that has an affected site as they’re likely to use similar setups for different sites.

Support ticket numbers (for my reference) in brackets.

  • PHP 7.4.4, MODX 2.7.3: Ace 1.8.0-pl, Big Brother 1.4.1-pl, ClientConfig 2.3.0-pl, Commerce 1.2.0-rc2, controlErrorLog 1.4.0-beta, Formit 4.2.5-pl, getCache 1.1.0-pl, pdoTools 2.12.7-pl, pThumb 2.3.3-pl, Quickstart Buttons 1.3.0-pl, Redactor 3.0.1-pl, Resizer 1.0.2-beta, SiteDash Client 1.3.3-pl, statcache 1.4.0pl, ToggleTVSet 1.2.6-pl, UpgradeMODX 2.1.2-pl, VersionX 2.3.2-pl [S24387]
  • PHP 7.4.10, MODX 2.7.3: Ace 1.8.0-pl, Alpacka 1.0.0-pl, Big Brother 1.4.1-pl, ClientConfig 2.3.0-pl, Collections 3.7.1-pl, Commerce 1.2.0-rc2, CommerceTheme Red 1.0.2-pl, Formula Shipping 1.1.0-pl, MyParcel dev3, PackingSlip 1.1.2-pl, TableRates 1.1.2-pl, CSRF Helper 1.0.0-pl, fastField 1.4.0, FormIt 4.2.5-pl, getPage 1.2.4-pl, getRelated 1.2.0-pl, getResources 1.6.1-pl, If 1.1.1-pl, Login 1.9.9-pl, MIGX 2.12.0-pl, MoreGallery 1.11.1-pl, pdoTools 2.12.7-pl, pThumb 2.3.3-pl, ReCaptchaV2 3.2.1-beta1, Redactor 3.0.1-pl, Resizer: 1.0.2-beta, SEO Pro 1.3.1-pl, SiteDash Client 1.3.3-pl (not installed), SEO Tab 2.2.0-pl, SuperBoxSelect 2.4.1-pl, Tagger 1.11.0-pl, VersionX 2.3.2-pl [S24344]

In common: PHP 7.4, MODX 2.7.3, Ace, Big Brother, ClientConfig, Commerce, FormIt, pdoTools, pThumb, Redactor, SiteDash Client, VersionX

2 
if (!class_exists(\MyNamespace\MyServiceClass::class)) {
    require_once '/path/to/vendor/autoload.php';
}
3

All autoloaders are properly loaded by all extras that have one. Dumping and inspecting the autoloaders proves that they’re all there. All extras work fine on their own until something somewhere causes them not to.

4

Maybe we should ask extra authors/maintainers to package the “optimized” autoloader to avoid this issue.

5

Sure, that does seem to work around it effectively as long as every extra has that. It doesn’t explain why this happens though, both an optimized and standard autoloader should be able to load valid classes from valid locations. There must be a root cause somewhere.

I just tried creating a blank site for reproducing the problem (installing FormIt 4.2.5, Redactor 3.0.1 and Commerce 1.1.4) but that alone does not appear to be sufficient to trigger it. Perhaps it requires another extra. I’ll continue to try and figure out reliable ways to reproduce it…

6

Does it break during install/update of the extra or does installing/updating work fine but it breaks afterwards?

7

The MODX internal scripts (phpmailer) could point to a reason for this. The phpmailer autoloading is quite old and I faced lately an incompatibility of this code in PHPUnit.

Somehow the PHP_VERSION constant is not detected right inside of PHPUnit in some circumstances. This is reproducible by running the unit tests ot the Login package. The check in https://github.com/modxcms/revolution/blob/2.x/core/model/modx/mail/phpmailer/PHPMailerAutoload.php#L33 does not work right and the traditional __autoload is executed. PHP 7.2, PHPUnit 6 and MODX 2.8.0-dev

8

Possible causes.
Case sensitive vs non-case sensitive filesystems. This is false here, all extras work fine by themselves.

Different versions of same library. If two extras load the same library, and there was some method renames (or other class collision happened), this will eventually cause failure. This might be true, but will not always cause a 500 error instantly. In this scenario re-running composer, even without optimized autoloader, will fix the problem by updating the library.

Different versions of composer were used to build autoloaders. Most likely this is the scenario. Some older versions had issues with path difference when directly copying all the files into a different host/path or deploying from localhost to web. If older version of autoloader loads before newer it leads to a class conflict. The described fix with rebuilding autoloader works obviously, because it replaces the outdated autoloader with new one.

This issue does not seem to have any oneshot solutions for all cases, except of, maybe, MODX having its own updated and fresh autoloader shipped and initialized before any other comes into play. Might be one more reason to push MODX 3 to a composer-based install, if this solutions works.

Another solution comes from package maintainers, who should update their packages and dependancies.

1 Like
9

This sounds like an interesting lead, and certainly different versions of composer sounds like an almost guaranteed scenario among different developers. Do you have any more info about this old version you’re referring to?

1 Like
10

The biggest jump in autoloader was here, I suppose:

2.0.0-alpha1 2020-06-03

There is even a waring here:

Composer 2.0 will stop autoloading these classes so make sure you fix your autoload configs.

1.10.0-RC 2020-02-14

Moreover, some autoloader versions had troubles discovering classes in files, that contained different style of php open/closing tags and/or HEREDOC syntax. This leaves even more space for yet undiscovered conflicts.

11

And some more digging. As composer offers 5 (!) different links to download a phar:
Latest Stable, Latest Preview (alpha/beta/RC), Latest Snapshot, Latest 1.x, Latest 2.x.

This also means a developer can download any of those, if it was not installed on their machine. Linux repositories always contain one of those versions (depends on local package maintainer). And, of course, one can use their php-based installer to get the latest version. This results in versions fluctuating everywhere.

I tried to check the autoloader code, produced by some of them, using only one, same package - box/spout (a spreadsheet reading library). And they really differ. Tested with 1.10.13 (considered latest stable release, usually ends up in linux packages), 2.0.0-RC1 (latest 2.x) and 1.8.0, 1.8.3 (these two come from one of my servers, yes, I ended up unintentionally caught with different composers too, as anyone else).
1.8.0 and 1.8.3 produced identical autoloader code, as expected, accodring to changelogs.
But difference between 1.10.13/2.0.0-RC1 or 1.8.3/2.0.0-RC1 was bigger. require vs require_once (not a big deal), array_values call dropped, no install relative path specified (now, this looks a little bad) and even no classMap loader at all.
1.8.3-2.0.0-RC1.txt
1.10.13-2.0.0-RC1.txt

12

The specific class it fails on is typically the first one being used in a service class constructor, or the service class itself.

That’s why I have a bootstrap.php that is loaded on OnMODXInit.
My bootstrap checks if the service class exists, if not it loads the autoloader.
I also make sure to load my Service Class in the bootstrap.php. This way I’m always sure my service class is loaded. Unless some other service is also loaded OnMODXInit and triggers the load of my service class…

13

Maybe it’s only reproducible by updating the extra’s.
I installed Commerce, FormIt and Redactor. Then I Uninstall Commerce (using the uninstall option). When reinstalling Commerce two errors are logged:

(ERROR @ /path/to/modx-2x/core/xpdo/xpdo.class.php : 644) Could not load class: Commerce from commerce.
(ERROR @ /path/to/modx-2x/core/xpdo/xpdo.class.php : 1247) Problem getting service commerce, instance of class Commerce, from path /path/to/modx-2x/core/components/commerce/model/commerce/

14

These errors are different. When classes are used without a check for file existence, e. g.:

if( is_file( $filename = $core_path . ‘vendor/autoload.php’ ) ) require_once( $filename );

Uninstall uses a cached instance (with class load), when files exists. So there is still an error thrown, before plugin cache is destroyed (at that moment package files do not exist).

15

I know, but it might be a good pointer to figure out what might cause this issue, maybe it’s cache related. Or something goes wrong during installing/updating the extra(s).

16

And it always happens after updating one of the extra’s?

17

It does seem to happen after adding an extra. In the cases I see, it’s often installing Commerce that then causes FormIt to break. But there are also cases without Commerce, so I’m not convinced it’s that. Commerce being very autoloading heavy might be a tipping point of sorts.

It does fairly often seem to happen on MODX Cloud, but that may also be a reporting bias - there’s quite a few people using Cloud also using my extras. Could perhaps point to some form of optimizations/opcaching maybe?

I’m upping the bounty to €450…

18

At what point does FormIt break (when loading the manager, running the snippet etc)?

19

I’m having a similar issue and I might be on something. It might be related to Commerce and the global auto loader.

20

In a significant number cases I’ve heard of, the site seems to work fine until Commerce is installed. At that point the Commerce dashboard throws a 500 error (which leads to people sending in a ticket; see earlier note about causality/bias in my data) with the error class Sterc\FormIt does not exist. Commerce doesn’t have any integration with FormIt, so that seems to come from its plugin or service being loaded.

When that happens I point those people to this thread and give them instructions to optimize the formit autoloader. In all but one case that then sorts them out.

There have also been people who ran into this issue who did not have Commerce installed so I’m skeptical, but until it’s clear what’s really happening I can’t rule it out either. If you have a lead you want to pursue and need some licenses for that, let me know.